title: Components-of-variance models and multiple-bootstrap experiments: an alternative method for random-effects, receiver operating characteristic analysis.
abstract: OBJECTIVE
  The purpose of this study was to develop an alternative approach to random-effects, receiver operating characteristic analysis inspired by a general formulation of components-of-variance models. The alternative approach is a higher-order generalization of the Dorfman, Berbaum, and Metz (DBM) approach that yields additional information on the variance structure of the problem.
  
  
  METHODS
  Six population experiments were designed to determine the six variance components in the DBM model. For practical problems, in which only a finite set of readers and patients are available, six analogous bootstrap experiments may be substituted for the population experiments to estimate the variance components. Monte Carlo simulations were performed on the population experiments, and those results were compared with the corresponding multiple-bootstrap estimates and those obtained with the DBM approach. Confidence intervals on the difference of ROC parameters for competing diagnostic modalities were estimated, and corresponding comparisons were made.
  
  
  RESULTS
  For mean values, the agreement of present estimates of variance structures with population results was excellent and, when suitably weighted and mixed, similar to or closer than that with the DBM method. For many variance structures, the confidence intervals in this study for the difference in ROC area between modalities were comparable to those with the DBM method. When reader variability was large, however, mean confidence intervals from this study were tighter than those with the DBM method and closer to population results.
  
  
  CONCLUSIONS
  The jackknife approach of DBM provides a linear approximation to receiver-operating-characteristic statistics that are intrinsically nonlinear. The multiple-bootstrap technique of this study, however, provides a more general, nonparametric, maximum-likelihood approach. It also yields estimates of the variance structure previously unavailable.
authors.affiliation: Office of Science and Technology, Center for Devices and Radiological Health, U.S. Food and Drug Administration, Rockville, MD 20857, USA.
authors.forename: S V
authors.initials: SV
authors.lastname: Beiden
authors.forename: R F
authors.initials: RF
authors.lastname: Wagner
authors.forename: G
authors.initials: G
authors.lastname: Campbell
mesh_terms: D016001:Confidence Intervals; D003951:Diagnostic Errors / Q000706:statistics & numerical data; D006801:Humans; D008962:Models, Theoretical; D012372:ROC Curve; D011871:Radiology / Q000706:statistics & numerical data*; D015203:Reproducibility of Results