title: Assessment of predictive performance in incomplete data by combining internal validation and multiple imputation.
abstract: Missing values are a frequent issue in human studies. In many situations, multiple imputation (MI) is an appropriate missing data handling strategy, whereby missing values are imputed multiple times, the analysis is performed in every imputed data set, and the obtained estimates are pooled. If the aim is to estimate (added) predictive performance measures, such as (change in) the area under the receiver-operating characteristic curve (AUC), internal validation strategies become desirable in order to correct for optimism. It is not fully understood how internal validation should be combined with multiple imputation.
  
  
  
  In a comprehensive simulation study and in a real data set based on blood markers as predictors for mortality, we compare three combination strategies: Val-MI, internal validation followed by MI on the training and test parts separately, MI-Val, MI on the full data set followed by internal validation, and MI(-y)-Val, MI on the full data set omitting the outcome followed by internal validation. Different validation strategies, including bootstrap und cross-validation, different (added) performance measures, and various data characteristics are considered, and the strategies are evaluated with regard to bias and mean squared error of the obtained performance estimates. In addition, we elaborate on the number of resamples and imputations to be used, and adopt a strategy for confidence interval construction to incomplete data.
  
  
  
  Internal validation is essential in order to avoid optimism, with the bootstrap 0.632+ estimate representing a reliable method to correct for optimism. While estimates obtained by MI-Val are optimistically biased, those obtained by MI(-y)-Val tend to be pessimistic in the presence of a true underlying effect. Val-MI provides largely unbiased estimates, with a slight pessimistic bias with increasing true effect size, number of covariates and decreasing sample size. In Val-MI, accuracy of the estimate is more strongly improved by increasing the number of bootstrap draws rather than the number of imputations. With a simple integrated approach, valid confidence intervals for performance estimates can be obtained.
  
  
  
  When prognostic models are developed on incomplete data, Val-MI represents a valid strategy to obtain estimates of predictive performance measures.
authors: Research Unit of Molecular Epidemiology, Helmholtz Zentrum München - German Research Center for Environmental Health, Ingolstädter Landstrasse, Neuherberg, 1, 85764, Germany. simwahl@googlemail.com.|Institute of Epidemiology II, Helmholtz Zentrum München - German Research Center for Environmental Health, Ingolstädter Landstrasse, Neuherberg, 1, 85764, Germany. simwahl@googlemail.com.|German Center for Diabetes Research (DZD e.V.), Ingolstädter Landstrasse, Neuherberg, 1, 85764, Germany. simwahl@googlemail.com.
authors: Simone
authors: S
authors: Wahl
authors: Department of Medical Informatics, Biometry and Epidemiology, Ludwig-Maximilians-Universität München, Marchioninistrasse, Munich, 15, 81377, Germany.
authors: Anne-Laure
authors: AL
authors: Boulesteix
authors: Institute of Epidemiology II, Helmholtz Zentrum München - German Research Center for Environmental Health, Ingolstädter Landstrasse, Neuherberg, 1, 85764, Germany.
authors: Astrid
authors: A
authors: Zierer
authors: Institute of Epidemiology II, Helmholtz Zentrum München - German Research Center for Environmental Health, Ingolstädter Landstrasse, Neuherberg, 1, 85764, Germany.|German Center for Diabetes Research (DZD e.V.), Ingolstädter Landstrasse, Neuherberg, 1, 85764, Germany.
authors: Barbara
authors: B
authors: Thorand
authors: Department of Epidemiology and Biostatistics, VU University Medical Center, PO Box 7057, Amsterdam, 1007, MB, The Netherlands.|Department of Mathematics, VU University, De Boelelaan 1081a, Amsterdam, 1081, HV, The Netherlands.
authors: Mark A
authors: MA
authors: van de Wiel
mesh_terms: D000328:Adult; D000368:Aged; D000465:Algorithms; D019540:Area Under Curve; D015415:Biomarkers / Q000097:blood; D002318:Cardiovascular Diseases / Q000097:blood / Q000401:mortality; D003627:Data Interpretation, Statistical; D003924:Diabetes Mellitus, Type 2 / Q000097:blood / Q000401:mortality; D006801:Humans; D016015:Logistic Models; D008875:Middle Aged; D015999:Multivariate Analysis; D016011:Normal Distribution; D011237:Predictive Value of Tests; D012372:ROC Curve; D012307:Risk Factors; D018401:Sample Size; D054928:Validation Studies as Topic
chemical_list: D015415:Biomarkers
keywords: Bootstrap; Cross-validation; Incomplete data; Internal validation; MICE; Missing values; Multiple imputation; Prediction model; Predictive performance; Resampling