title: Corrected ROC analysis for misclassified binary outcomes.
abstract: Creating accurate risk prediction models from Big Data resources such as Electronic Health Records (EHRs) is a critical step toward achieving precision medicine. A major challenge in developing these tools is accounting for imperfect aspects of EHR data, particularly the potential for misclassified outcomes. Misclassification, the swapping of case and control outcome labels, is well known to bias effect size estimates for regression prediction models. In this paper, we study the effect of misclassification on accuracy assessment for risk prediction models and find that it leads to bias in the area under the curve (AUC) metric from standard ROC analysis. The extent of the bias is determined by the false positive and false negative misclassification rates as well as disease prevalence. Notably, we show that simply correcting for misclassification while building the prediction model is not sufficient to remove the bias in AUC. We therefore introduce an intuitive misclassification-adjusted ROC procedure that accounts for uncertainty in observed outcomes and produces bias-corrected estimates of the true AUC. The method requires that misclassification rates are either known or can be estimated, quantities typically required for the modeling step. The computational simplicity of our method is a key advantage, making it ideal for efficiently comparing multiple prediction models on very large datasets. Finally, we apply the correction method to a hospitalization prediction model from a cohort of over 1 million patients from the Veterans Health Administrations EHR. Implementations of the ROC correction are provided for Stata and R. Published 2017. This article is a U.S. Government work and is in the public domain in the USA.
authors.affiliation: Veterans Affairs Center for Clinical Management Research, Ann Arbor, 48105, MI, U.S.A.|Department of Biostatistics, University of Michigan, Ann Arbor, 48109, MI, U.S.A.
authors.forename: Matthew
authors.identifier: 0000-0002-3005-083X
authors.initials: M
authors.lastname: Zawistowski
authors.affiliation: Veterans Affairs Center for Clinical Management Research, Ann Arbor, 48105, MI, U.S.A.|Department of Internal Medicine, University of Michigan Medical School, Ann Arbor, 48109, MI, U.S.A.
authors.forename: Jeremy B
authors.initials: JB
authors.lastname: Sussman
authors.affiliation: Veterans Affairs Center for Clinical Management Research, Ann Arbor, 48105, MI, U.S.A.|Department of Internal Medicine, University of Michigan Medical School, Ann Arbor, 48109, MI, U.S.A.
authors.forename: Timothy P
authors.initials: TP
authors.lastname: Hofer
authors.affiliation: Veterans Affairs Center for Clinical Management Research, Ann Arbor, 48105, MI, U.S.A.
authors.forename: Douglas
authors.initials: D
authors.lastname: Bentley
authors.affiliation: Veterans Affairs Center for Clinical Management Research, Ann Arbor, 48105, MI, U.S.A.|Department of Internal Medicine, University of Michigan Medical School, Ann Arbor, 48109, MI, U.S.A.
authors.forename: Rodney A
authors.initials: RA
authors.lastname: Hayward
authors.affiliation: Veterans Affairs Center for Clinical Management Research, Ann Arbor, 48105, MI, U.S.A.
authors.forename: Wyndy L
authors.initials: WL
authors.lastname: Wiitala
mesh_terms: D019540:Area Under Curve; D015982:Bias; D057286:Electronic Health Records; D006760:Hospitalization / Q000706:statistics & numerical data; D006801:Humans; D015233:Models, Statistical; D012372:ROC Curve; D018570:Risk Assessment / Q000379:methods; D014481:United States; D014493:United States Department of Veterans Affairs / Q000706:statistics & numerical data
keywords: ROC analysis; electronic health records; misclassification; precision medicine; risk prediction modeling