Computationally efficient confidence intervals for cross-validated area under the ROC curve estimates.
In binary classification problems, the area under the ROC curve (AUC) is commonly used to evaluate the performance of a prediction model. Often, it is combined with cross-validation in order to assess how the results will generalize to an independent data set. In order to evaluate the quality of an estimate for cross-validated AUC, we obtain an estimate of its variance. For massive data sets, the process of generating a single performance estimate can be computationally expensive. Additionally, when using a complex prediction method, the process of cross-validating a predictive model on even a relatively small data set can still require a large amount of computation time. Thus, in many practical settings, the bootstrap is a computationally intractable approach to variance estimation. As an alternative to the bootstrap, we demonstrate a computationally efficient influence curve based approach to obtaining a variance estimate for cross-validated AUC.
Division of Biostatistics, University of California, Berkeley, Berkeley, CA 94720, USA.
Erin
E
LeDell
Division of Biostatistics, University of California, Berkeley, Berkeley, CA 94720, USA.
Maya
M
Petersen
Division of Biostatistics, University of California, Berkeley, Berkeley, CA 94720, USA.
Mark
M
van der Laan
AUC; ROC; binary classification; confidence intervals; cross-validation; influence curve; influence function; machine learning; model selection; variance estimation