title: Prediction accuracy of a sample-size estimation method for ROC studies.
abstract: OBJECTIVE
  Sample-size estimation is an important consideration when planning a receiver operating characteristic (ROC) study. The aim of this work was to assess the prediction accuracy of a sample-size estimation method using the Monte Carlo simulation method.
  
  
  METHODS
  Two ROC ratings simulators characterized by low reader and high case variabilities (LH) and high reader and low case variabilities (HL) were used to generate pilot data sets in two modalities. Dorfman-Berbaum-Metz multiple-reader multiple-case (DBM-MRMC) analysis of the ratings yielded estimates of the modality-reader, modality-case, and error variances. These were input to the Hillis-Berbaum (HB) sample-size estimation method, which predicted the number of cases needed to achieve 80% power for 10 readers and an effect size of 0.06 in the pivotal study. Predictions that generalized to readers and cases (random-all), to cases only (random-cases), and to readers only (random-readers) were generated. A prediction-accuracy index defined as the probability that any single prediction yields true power in the 75%-90% range was used to assess the HB method.
  
  
  RESULTS
  For random-case generalization, the HB-method prediction-accuracy was reasonable, approximately 50% for five readers and 100 cases in the pilot study. Prediction-accuracy was generally higher under LH conditions than under HL conditions. Under ideal conditions (many readers in the pilot study) the DBM-MRMC-based HB method overestimated the number of cases. The overestimates could be explained by the larger modality-reader variance estimates when reader variability was large (HL). The largest benefit of increasing the number of readers in the pilot study was realized for LH, where 15 readers were enough to yield prediction accuracy >50% under all generalization conditions, but the benefit was lesser for HL where prediction accuracy was approximately 36% for 15 readers under random-all and random-reader conditions.
  
  
  CONCLUSIONS
  The HB method tends to overestimate the number of cases. Random-case generalization had reasonable prediction accuracy. Provided about 15 readers were used in the pilot study the method performed reasonably under all conditions for LH. When reader variability was large, the prediction-accuracy for random-all and random-reader generalizations was compromised. Study designers may wish to compare the HB predictions to those of other methods and to sample-sizes used in previous similar studies.
authors.affiliation: Department of Radiology, University of Pittsburgh, Pittsburgh, PA 15213, USA. dpc10@pitt.edu
authors.forename: Dev P
authors.initials: DP
authors.lastname: Chakraborty
mesh_terms: D000465:Algorithms; D003627:Data Interpretation, Statistical; D007089:Image Enhancement / Q000379:methods*; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D015588:Observer Variation; D010363:Pattern Recognition, Automated / Q000379:methods*; D012372:ROC Curve; D015203:Reproducibility of Results; D018401:Sample Size; D012680:Sensitivity and Specificity
references.citation: Acad Radiol. 1997 Apr;4(4):298-303
references.pmid: 9110028
references.citation: Radiology. 1983 Sep;148(3):839-43
references.pmid: 6878708
references.citation: Acad Radiol. 2009 Feb;16(2):209-17
references.pmid: 19124107
references.citation: Acad Radiol. 1995 Mar;2 Suppl 1:S22-9; discussion S57-64, S70-1 pas
references.pmid: 9419702
references.citation: Acad Radiol. 2004 Nov;11(11):1260-73
references.pmid: 15561573
references.citation: J ICRU. 2008 Apr;8(1):37-40
references.pmid: 24174486
references.citation: Acad Radiol. 2003 Nov;10(11):1327-8
references.pmid: 14626309
references.citation: Acad Radiol. 2008 May;15(5):647-61
references.pmid: 18423323
references.citation: Med Phys. 1995 May;22(5):643-55
references.pmid: 7643805
references.citation: AJR Am J Roentgenol. 2000 Sep;175(3):603-8
references.pmid: 10954438
references.citation: AJR Am J Roentgenol. 2004 Apr;182(4):867-9
references.pmid: 15039154
references.citation: Invest Radiol. 1994 Feb;29(2):238-43
references.pmid: 8169102
references.citation: Acad Radiol. 1998 Sep;5(9):591-602
references.pmid: 9750888
references.citation: Arch Gen Psychiatry. 2006 May;63(5):484-9
references.pmid: 16651505
references.citation: Acad Radiol. 2000 May;7(5):341-9
references.pmid: 10803614
references.citation: Invest Radiol. 1986 Sep;21(9):720-33
references.pmid: 3095258
references.citation: Stat Med. 1997 Jul 15;16(13):1529-42
references.pmid: 9249923
references.citation: Acad Radiol. 2005 Dec;12(12):1534-41
references.pmid: 16321742
references.citation: Invest Radiol. 1992 Sep;27(9):723-31
references.pmid: 1399456
references.citation: Stat Methods Med Res. 1998 Dec;7(4):371-92
references.pmid: 9871953
references.citation: Acad Radiol. 1997 Aug;4(8):587-600
references.pmid: 9261459