title: Agreement of the order of overall performance levels under different reading paradigms.
abstract: OBJECTIVE
  To investigate consistency of the orders of performance levels when interpreting mammograms under three different reading paradigms.
  
  
  METHODS
  We performed a retrospective observer study in which nine experienced radiologists rated an enriched set of mammography examinations that they personally had read in the clinic ("individualized") mixed with a set that none of them had read in the clinic ("common set"). Examinations were interpreted under three different reading paradigms: binary using screening Breast Imaging Reporting and Data System (BI-RADS), receiver-operating characteristic (ROC), and free-response ROC (FROC). The performance in discriminating between cancer and noncancer findings under each of the paradigms was summarized using Youden's index/2+0.5 (Binary), nonparameteric area under the ROC curve (AUC), and an overall FROC index (JAFROC-2). Pearson correlation coefficients were then computed to assess consistency in the ordering of observers' performance levels. Statistical significance of the computed correlation coefficients was assessed using bootstrap confidence intervals obtained by resampling sets of examination-specific observations.
  
  
  RESULTS
  All but one of the computed pair-wise correlation coefficients were larger than 0.66 and were significantly different from zero. The correlation between the overall performance measures under the Binary and ROC paradigms was the lowest (0.43) and was not significantly different from zero (95% confidence interval -0.078 to 0.733).
  
  
  CONCLUSIONS
  The use of different evaluation paradigms in the laboratory tends to lead to consistent ordering of the overall performance levels of observers. However, one should recognize that conceptually similar performance indexes resulting from different paradigms often measure different performance characteristics and thus disagreements are not only possible but frequently quite natural.
authors.affiliation: Department of Radiology, University of Pittsburgh, 3362 Fifth Avenue, Pittsburgh, PA 15213-3180, USA. gurd@upmc.edu
authors.forename: David
authors.initials: D
authors.lastname: Gur
authors.forename: Andriy I
authors.initials: AI
authors.lastname: Bandos
authors.forename: Amy H
authors.initials: AH
authors.lastname: Klym
authors.forename: Cathy S
authors.initials: CS
authors.lastname: Cohen
authors.forename: Christiane M
authors.initials: CM
authors.lastname: Hakim
authors.forename: Lara A
authors.initials: LA
authors.lastname: Hardesty
authors.forename: Marie A
authors.initials: MA
authors.lastname: Ganott
authors.forename: Ronald L
authors.initials: RL
authors.lastname: Perrin
authors.forename: William R
authors.initials: WR
authors.lastname: Poller
authors.forename: Ratan
authors.initials: R
authors.lastname: Shah
authors.forename: Jules H
authors.initials: JH
authors.lastname: Sumkin
authors.forename: Luisa P
authors.initials: LP
authors.lastname: Wallace
authors.forename: Howard E
authors.initials: HE
authors.lastname: Rockette
mesh_terms: D001943:Breast Neoplasms / Q000000981:diagnostic imaging*; D003627:Data Interpretation, Statistical; D005260:Female; D006801:Humans; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D008327:Mammography / Q000379:methods*; D015588:Observer Variation; D011361:Professional Competence; D012372:ROC Curve; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D013647:Task Performance and Analysis