Prediction accuracy of a sample-size estimation method for ROC studies.
OBJECTIVE
Sample-size estimation is an important consideration when planning a receiver operating characteristic (ROC) study. The aim of this work was to assess the prediction accuracy of a sample-size estimation method using the Monte Carlo simulation method.


METHODS
Two ROC ratings simulators characterized by low reader and high case variabilities (LH) and high reader and low case variabilities (HL) were used to generate pilot data sets in two modalities. Dorfman-Berbaum-Metz multiple-reader multiple-case (DBM-MRMC) analysis of the ratings yielded estimates of the modality-reader, modality-case, and error variances. These were input to the Hillis-Berbaum (HB) sample-size estimation method, which predicted the number of cases needed to achieve 80% power for 10 readers and an effect size of 0.06 in the pivotal study. Predictions that generalized to readers and cases (random-all), to cases only (random-cases), and to readers only (random-readers) were generated. A prediction-accuracy index defined as the probability that any single prediction yields true power in the 75%-90% range was used to assess the HB method.


RESULTS
For random-case generalization, the HB-method prediction-accuracy was reasonable, approximately 50% for five readers and 100 cases in the pilot study. Prediction-accuracy was generally higher under LH conditions than under HL conditions. Under ideal conditions (many readers in the pilot study) the DBM-MRMC-based HB method overestimated the number of cases. The overestimates could be explained by the larger modality-reader variance estimates when reader variability was large (HL). The largest benefit of increasing the number of readers in the pilot study was realized for LH, where 15 readers were enough to yield prediction accuracy >50% under all generalization conditions, but the benefit was lesser for HL where prediction accuracy was approximately 36% for 15 readers under random-all and random-reader conditions.


CONCLUSIONS
The HB method tends to overestimate the number of cases. Random-case generalization had reasonable prediction accuracy. Provided about 15 readers were used in the pilot study the method performed reasonably under all conditions for LH. When reader variability was large, the prediction-accuracy for random-all and random-reader generalizations was compromised. Study designers may wish to compare the HB predictions to those of other methods and to sample-sizes used in previous similar studies.
Chakraborty
Dev P
DP
Department of Radiology, University of Pittsburgh, Pittsburgh, PA 15213, USA. dpc10@pitt.edu
D000465:Algorithms; D003627:Data Interpretation, Statistical; D007089:Image Enhancement / Q000379:methods*; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D015588:Observer Variation; D010363:Pattern Recognition, Automated / Q000379:methods*; D012372:ROC Curve; D015203:Reproducibility of Results; D018401:Sample Size; D012680:Sensitivity and Specificity
Acad Radiol. 1997 Apr;4(4):298-303
9110028
Radiology. 1983 Sep;148(3):839-43
6878708
Acad Radiol. 2009 Feb;16(2):209-17
19124107
Acad Radiol. 1995 Mar;2 Suppl 1:S22-9; discussion S57-64, S70-1 pas
9419702
Acad Radiol. 2004 Nov;11(11):1260-73
15561573
J ICRU. 2008 Apr;8(1):37-40
24174486
Acad Radiol. 2003 Nov;10(11):1327-8
14626309
Acad Radiol. 2008 May;15(5):647-61
18423323
Med Phys. 1995 May;22(5):643-55
7643805
AJR Am J Roentgenol. 2000 Sep;175(3):603-8
10954438
AJR Am J Roentgenol. 2004 Apr;182(4):867-9
15039154
Invest Radiol. 1994 Feb;29(2):238-43
8169102
Acad Radiol. 1998 Sep;5(9):591-602
9750888
Arch Gen Psychiatry. 2006 May;63(5):484-9
16651505
Acad Radiol. 2000 May;7(5):341-9
10803614
Invest Radiol. 1986 Sep;21(9):720-33
3095258
Stat Med. 1997 Jul 15;16(13):1529-42
9249923
Acad Radiol. 2005 Dec;12(12):1534-41
16321742
Invest Radiol. 1992 Sep;27(9):723-31
1399456
Stat Methods Med Res. 1998 Dec;7(4):371-92
9871953
Acad Radiol. 1997 Aug;4(8):587-600
9261459