title: Reducing the number of reader interpretations in MRMC studies.
abstract: OBJECTIVE
  Multireader, multicase (MRMC) receiver-operating characteristic studies often require large numbers of patients, readers, and reader interpretations. The objective of this work is to evaluate a new "mixed" MRMC study design that reduces the number of reader interpretations.
  
  
  METHODS
  As compared to the traditional MRMC design, the number of reader interpretations and the number of cases that must be truth-verified for the new mixed design was evaluated theoretically and empirically for various correlation values and sample sizes.
  
  
  RESULTS
  For large MRMC studies, the new mixed design offers a substantial savings in the number of reader interpretations if the magnitude of the difference in between-reader correlations is not zero. For example, compared to a traditional design with 20 readers, 200 total cases, and a difference in between-reader correlations of 0.05, the newly proposed mixed design requires each reader to interpret only 132 cases, but at a cost of truth-verifying an additional 64 cases.
  
  
  CONCLUSIONS
  The mixed design can reduce the number of cases that readers need to interpret and the overall duration of a study, but at a cost in terms of the number of cases that must be truth-verified. The mixed design is particularly useful for studies where the condition being detected is not rare and patients routinely undergo the gold standard assessment.
authors.affiliation: Department of Quantitative Health Sciences/Wb4, and the Imaging Institute, The Cleveland Clinic, Cleveland, OH 44195, USA. obuchon@ccf.org
authors.forename: Nancy A
authors.initials: NA
authors.lastname: Obuchowski
mesh_terms: D000465:Algorithms; D003627:Data Interpretation, Statistical; D007089:Image Enhancement / Q000379:methods; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D015588:Observer Variation; D012372:ROC Curve; D015203:Reproducibility of Results; D018401:Sample Size; D012680:Sensitivity and Specificity