title: Learning the preferences of physicians for the organization of result lists of medical evidence articles.
abstract: BACKGROUND
  Online medical knowledge repositories such as MEDLINE and The Cochrane Library are increasingly used by physicians to retrieve articles to aid with clinical decision making. The prevailing approach for organizing retrieved articles is in the form of a rank-ordered list, with the assumption that the higher an article is presented on a list, the more relevant it is.
  
  
  OBJECTIVE
  Despite this common list-based organization, it is seldom studied how physicians perceive the association between the relevance of articles and the order in which articles are presented. In this paper we describe a case study that captured physician preferences for 3-element lists of medical articles in order to learn how to organize medical knowledge for decision-making.
  
  
  METHODS
  Comprehensive relevance evaluations were developed to represent 3-element lists of hypothetical articles that may be retrieved from an online medical knowledge source such as MEDLINE or The Cochrane Library. Comprehensive relevance evaluations asses not only an article's relevance for a query, but also whether it has been placed on the correct list position. In other words an article may be relevant and correctly placed on a result list (e.g. the most relevant article appears first in the result list), an article may be relevant for a query but placed on an incorrect list position (e.g. the most relevant article appears second in a result list), or an article may be irrelevant for a query yet still appear in the result list. The relevance evaluations were presented to six senior physicians who were asked to express their preferences for an article's relevance and its position on a list by pairwise comparisons representing different combinations of 3-element lists. The elicited preferences were assessed using a novel GRIP (Generalized Regression with Intensities of Preference) method and represented as an additive value function. Value functions were derived for individual physicians as well as the group of physicians.
  
  
  RESULTS
  The results show that physicians assign significant value to the 1st position on a list and they expect that the most relevant article is presented first. Whilst physicians still prefer obtaining a correctly placed article on position 2, they are also quite satisfied with misplaced relevant article. Low consideration of the 3rd position was uniformly confirmed.
  
  
  CONCLUSIONS
  Our findings confirm the importance of placing the most relevant article on the 1st position on a list and the importance paid to position on a list significantly diminishes after the 2nd position. The derived value functions may be used by developers of clinical decision support applications to decide how best to organize medical knowledge for decision making and to create personalized evaluation measures that can augment typical measures used to evaluate information retrieval systems.
authors.affiliation: Dympna O'Sullivan, School of Informatics, City University London, Northampton Square, London EC1V 0HB, United Kingdom, E-mail: Dympna.O'Sullivan.1@city.ac.uk.
authors.forename: D
authors.initials: D
authors.lastname: O'Sullivan
authors.forename: S
authors.initials: S
authors.lastname: Wilk
authors.forename: W
authors.initials: W
authors.lastname: Michalowski
authors.forename: R
authors.initials: R
authors.lastname: Slowinski
authors.forename: R
authors.initials: R
authors.lastname: Thomas
authors.forename: M
authors.initials: M
authors.lastname: Kadzinski
authors.forename: K
authors.initials: K
authors.lastname: Farion
mesh_terms: D019317:Evidence-Based Medicine; D007722:Health Knowledge, Attitudes, Practice; D006801:Humans; D016247:Information Storage and Retrieval / Q000145:classification*; D010820:Physicians / Q000523:psychology*; D011643:Publishing; D011795:Surveys and Questionnaires
keywords: Organization of medical evidence; clinical; decision support systems; evidence-based medicine; information retrieval; physician preferences; rank-ordered lists