"Binary" and "non-binary" detection tasks: are current performance measures optimal?
We have observed that a very large fraction of responses for several detection tasks during the performance of observer studies are in the extreme ranges of lower than 11% or higher than 89% regardless of the actual presence or absence of the abnormality in question or its subjectively rated "subtleness." This observation raises questions regarding the validity and appropriateness of using multicategory rating scales for such detection tasks. Monte Carlo simulation of binary and multicategory ratings for these tasks demonstrate that the use of the former (binary) often results in a less biased and more precise summary index and hence may lead to a higher statistical power for determining differences between modalities.
Gur
David
D
Department of Radiology, School of Medicine, University of Pittsburgh, Pittsburgh, PA 15213, USA. gurd@upmc.edu
Rockette
Howard E
HE
Bandos
Andriy I
AI
D019540:Area Under Curve; D002983:Clinical Competence; D003936:Diagnosis, Computer-Assisted; D003952:Diagnostic Imaging; D006801:Humans; D008171:Lung Diseases / Q000175:diagnosis*; D009010:Monte Carlo Method; D015588:Observer Variation; D012372:ROC Curve; D015203:Reproducibility of Results