title: Multipoint rank-order study methodology: observer issues.
abstract: OBJECTIVE
  We performed a multipoint rank-order experiment to evaluate variability in observers' sensitivity to small differences in image presentation and to assess observers' performance as a function of the type and number of tasks included.
  
  
  METHODS
  Five experienced observers were presented with four sets of chest images that had been compressed at five different levels. Each set contained six images ranging from noncompressed to approximately 60:1-compressed images. Observers were asked to review all images of each case side by side and rank-order the "quality" of each to enable determination of the presence or absence of interstitial disease and/or pneumothoraces.
  
  
  RESULTS
  Observers varied significantly in their ability to detect very small differences among the images (P < 0.001). Those who performed well did so regardless of whether they ranked a specific abnormality in a multidisease or a single-disease setting.
  
  
  CONCLUSIONS
  Selected observers can reliably detect very small differences among similar images. These readers could be used to confirm or rule out the need for objective observer-performance-type studies.
authors.affiliation: Department of Radiology, University of Pittsburgh, Pennsylvania 15261-0001, USA.
authors.forename: J D
authors.initials: JD
authors.lastname: Towers
authors.forename: J M
authors.initials: JM
authors.lastname: Holbert
authors.forename: C A
authors.initials: CA
authors.lastname: Britton
authors.forename: P
authors.initials: P
authors.lastname: Costello
authors.forename: R
authors.initials: R
authors.lastname: Sciulli
authors.forename: D
authors.initials: D
authors.lastname: Gur
mesh_terms: D006801:Humans; D007091:Image Processing, Computer-Assisted; D015588:Observer Variation; D013902:Radiography, Thoracic / Q000706:statistics & numerical data