Is an ROC-type response truly always better than a binary response in observer performance studies?
OBJECTIVE
The aim of this study was to assess similarities and differences between methods of performance comparisons under binary (yes or no) and receiver-operating characteristic (ROC)-type pseudocontinuous (0-100) rating data ascertained during an observer performance study of interpretation of full-field digital mammography (FFDM) versus FFDM plus digital breast tomosynthesis.


METHODS
Rating data consisted of ROC-type pseudocontinuous and binary ratings generated by eight radiologists evaluating 77 digital mammographic examinations. Overall performance levels were summarized with a conventionally used probability of correct discrimination or, equivalently, the area under the ROC curve (AUC), which under a binary scale is related to Youden's index. Magnitudes of differences in the reader-averaged empirical AUCs between FFDM alone and FFDM plus digital breast tomosynthesis were compared in the context of fixed-reader and random-reader variability of the estimates.


RESULTS
The absolute differences between modes using the empirical AUCs were larger on average for the binary scale (0.12 vs 0.07) and for the majority of individual readers (six of eight). Standardized differences were consistent with this finding (2.32 vs 1.63 on average). Reader-averaged differences in AUCs standardized by fixed-reader and random-reader variances were also smaller under the binary rating paradigm. The discrepancy between AUC differences depended on the location of the reader-specific binary operating points.


CONCLUSIONS
The human observer's operating point should be a primary consideration in designing an observer performance study. Although in general, the ROC-type rating paradigm provides more detailed information on the characteristics of different modes, it does not reflect the actual operating point adopted by human observers. There are application-driven scenarios in which analysis based on binary responses may provide statistical advantages.
University of Pittsburgh, Department of Radiology, Radiology Imaging Research, Pittsburgh, PA 15213, USA. gurd@upmc.edu
David
D
Gur
Andriy I
AI
Bandos
Howard E
HE
Rockette
Margarita L
ML
Zuley
Christiane M
CM
Hakim
Denise M
DM
Chough
Marie A
MA
Ganott
Jules H
JH
Sumkin
D000465:Algorithms; D001943:Breast Neoplasms / Q000000981:diagnostic imaging*; D005260:Female; D006801:Humans; D008327:Mammography / Q000379:methods*; D015588:Observer Variation; D012372:ROC Curve; D011856:Radiographic Image Enhancement / Q000379:methods*; D011857:Radiographic Image Interpretation, Computer-Assisted / Q000379:methods*; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity