Multireader, multicase receiver operating characteristic analysis: an empirical comparison of five methods.
OBJECTIVE
Several statistical methods have been developed for analyzing multireader, multicase (MRMC) receiver operating characteristic (ROC) studies. The objective of this article is to increase awareness of these methods and determine if their results are concordant for published datasets.


METHODS
Data from three previously published studies were reanalyzed using five MRMC methods. For each method the 95% confidence intervals (CIs) for the mean of the readers' ROC areas for each diagnostic test, the P value for the comparison of the diagnostic tests' mean accuracies, and the 95% CIs for the mean difference in ROC areas of the diagnostic tests were reported.


RESULTS
Important differences in P values and CIs were seen when using parametric versus nonparametric estimates of accuracy, and there were the expected differences for random-reader versus fixed-reader models. Controlling for these differences, the Dorfman-Berbaum-Metz (DBM), Obuchowski-Rockette, Beiden-Wagner-Campbell, and Song's multivariate Wilcoxon-Mann-Whitney (WMW) methods gave almost identical results for the fixed-reader model. For the random-reader model, the DBM, Obuchowski-Rockette, and Beiden-Wagner-Campbell methods yielded approximately the same inferences, but the CIs for the Beiden-Wagner-Campbell method tend to be broader. Ishwaran's hierarchical ROC sometimes yielded significance not found with other methods. Song's modification of DBM's jack-knifing algorithm sometimes led to different conclusions than the original DBM algorithm.


CONCLUSIONS
In choosing and applying MRMC methods, it is important to recognize: (1) the distinction between random-reader and fixed-reader models, the uncertainties accounted for by each, and thus the level of generalizeability expected from each; (2) assumptions made by the various MRMC methods; and (3) limitations of a five- or six-reader study when the reader variability is great.
Obuchowski
Nancy A
NA
Departments of Biostatistics and Epidemiology, Cleveland Clinic Foundation, Cleveland, OH 44195, USA. nobuchow@bio.ri.ccf.org
Beiden
Sergey V
SV
Berbaum
Kevin S
KS
Hillis
Stephen L
SL
Ishwaran
Hemant
H
Song
Hae Hiang
HH
Wagner
Robert F
RF
D000704:Analysis of Variance; D000784:Aortic Dissection / Q000000981:diagnostic imaging; D001014:Aortic Aneurysm / Q000000981:diagnostic imaging; D001943:Breast Neoplasms / Q000000981:diagnostic imaging; D005189:False Positive Reactions; D005260:Female; D006801:Humans; D017563:Lung Diseases, Interstitial / Q000000981:diagnostic imaging; D008327:Mammography; D015233:Models, Statistical; D015999:Multivariate Analysis; D015588:Observer Variation; D012372:ROC Curve; D011859:Radiography; D013902:Radiography, Thoracic; D012815:Signal Processing, Computer-Assisted; D018709:Statistics, Nonparametric; D014962:X-Ray Film