One-shot estimate of MRMC variance: AUC.
OBJECTIVE
One popular study design for estimating the area under the receiver operating characteristic curve (AUC) is the one in which a set of readers reads a set of cases: a fully crossed design in which every reader reads every case. The variability of the subsequent reader-averaged AUC has two sources: the multiple readers and the multiple cases (MRMC). In this article, we present a nonparametric estimate for the variance of the reader-averaged AUC that is unbiased and does not use resampling tools.


METHODS
The one-shot estimate is based on the MRMC variance derived by the mechanistic approach of Barrett et al. (2005), as well as the nonparametric variance of a single-reader AUC derived in the literature on U statistics. We investigate the bias and variance properties of the one-shot estimate through a set of Monte Carlo simulations with simulated model observers and images. The different simulation configurations vary numbers of readers and cases, amounts of image noise and internal noise, as well as how the readers are constructed. We compare the one-shot estimate to a method that uses the jackknife resampling technique with an analysis of variance model at its foundation (Dorfman et al. 1992). The name one-shot highlights that resampling is not used.


RESULTS
The one-shot and jackknife estimators behave similarly, with the one-shot being marginally more efficient when the number of cases is small.


CONCLUSIONS
We have derived a one-shot estimate of the MRMC variance of AUC that is based on a probabilistic foundation with limited assumptions, is unbiased, and compares favorably to an established estimate.
NIBIB/CDRH Laboratory for the Assessment of Medical Imaging Systems, US FDA/CDRH, Bldg 1, HFZ-140, 12720 Twinbrook Parkway (Rm 158), Rockville MD 20852-1720, USA. brandon.gallas@fda.hhs.gov
Brandon D
BD
Gallas
D019540:Area Under Curve; D003198:Computer Simulation; D003952:Diagnostic Imaging; D009010:Monte Carlo Method