title: Multireader, multicase receiver operating characteristic methodology: a bootstrap analysis.
abstract: OBJECTIVE
  We evaluated by bootstrapping the conclusions obtained by the Dorfman-Berbaum-Metz (DBM) receiver operating characteristic (ROC) method and by the Toledano-Gatsonis (TG) method on a well-known data set.
  
  
  METHODS
  We bootstrapped in two ways, resampled cases while holding readers fixed and resampled both cases and readers.
  
  
  RESULTS
  When an analysis of variance of pseudovalues implies that reader variance and all random interactions with treatment are essentially zero, then case-resampling bootstrap and the DBM and TG methods should give the same results. Case-resampling bootstrap and the DBM and TG methods did give highly similar results for both individual readers and the averages over all readers. Both the case-resampling bootstrap and the reader-case resampling bootstrap gave smaller standard errors for group than for individual reader means, thereby providing evidence for a trade-off of readers and cases with regard to precision and power in this data set.
  
  
  CONCLUSIONS
  Case-resampling bootstrap provides some justification for the DBM and TG methods.
authors.affiliation: Department of Psychology, University of Iowa, Iowa City 52242, USA.
authors.forename: D D
authors.initials: DD
authors.lastname: Dofrman
authors.forename: K S
authors.initials: KS
authors.lastname: Berbaum
authors.forename: R V
authors.initials: RV
authors.lastname: Lenth
mesh_terms: D000704:Analysis of Variance; D016001:Confidence Intervals; D006801:Humans; D007231:Infant, Newborn; D016013:Likelihood Functions; D015233:Models, Statistical; D011601:Psychophysics / Q000379:methods; D012372:ROC Curve; D011859:Radiography / Q000706:statistics & numerical data*; D015203:Reproducibility of Results