title: The estimation and use of predictions for the assessment of model performance using large samples with multiply imputed data.
abstract: Multiple imputation can be used as a tool in the process of constructing prediction models in medical and epidemiological studies with missing covariate values. Such models can be used to make predictions for model performance assessment, but the task is made more complicated by the multiple imputation structure. We summarize various predictions constructed from covariates, including multiply imputed covariates, and either the set of imputation-specific prediction model coefficients or the pooled prediction model coefficients. We further describe approaches for using the predictions to assess model performance. We distinguish between ideal model performance and pragmatic model performance, where the former refers to the model's performance in an ideal clinical setting where all individuals have fully observed predictors and the latter refers to the model's performance in a real-world clinical setting where some individuals have missing predictors. The approaches are compared through an extensive simulation study based on the UK700 trial. We determine that measures of ideal model performance can be estimated within imputed datasets and subsequently pooled to give an overall measure of model performance. Alternative methods to evaluate pragmatic model performance are required and we propose constructing predictions either from a second set of covariate imputations which make no use of observed outcomes, or from a set of partial prediction models constructed for each potential observed pattern of covariate. Pragmatic model performance is generally lower than ideal model performance. We focus on model performance within the derivation data, but describe how to extend all the methods to a validation dataset.
authors: Department of Public Health and Primary Care, Strangeways Research Laboratory, University of Cambridge, Worts Causeway, Cambridge CB1 8RN, UK.
authors: Angela M
authors: AM
authors: Wood
authors: MRC Clinical Trials Unit at UCL, Aviation House, 125 Kingsway, London WC2B 6NH, UK.
authors: Patrick
authors: P
authors: Royston
authors: MRC Biostatistics Unit, Cambridge Institute of Public Health, Forvie Site, Robinson Way, Cambridge Biomedical Campus, Cambridge CB2 0SR, UK.
authors: Ian R
authors: IR
authors: White
mesh_terms: D000704:Analysis of Variance; D001699:Biometry / Q000379:methods*; D002986:Clinical Trials as Topic; D006801:Humans; D016015:Logistic Models; D015233:Models, Statistical
keywords: Measures of model performance; Missing data; Model validation; Multiple imputation; Prediction models; Rubin's rules
references: Heart. 2012 May;98(9):691-8
references: 22397946
references: Heart. 2012 May;98(9):683-90
references: 22397945
references: Stat Med. 2008 Jul 30;27(17):3227-46
references: 18203127
references: BMJ. 2007 Jul 21;335(7611):136
references: 17615182
references: Stat Methods Med Res. 2007 Jun;16(3):277-98
references: 17621472
references: J Clin Epidemiol. 2006 Oct;59(10):1092-101
references: 16980150
references: Lancet. 1999 Jun 26;353(9171):2185-9
references: 10392982
references: Stat Med. 1999 Mar 30;18(6):681-94
references: 10204197
references: J Clin Epidemiol. 2003 Jan;56(1):28-37
references: 12589867
references: PLoS Med. 2008 Aug 5;5(8):e165; discussion e165
references: 18684008
references: Br J Cancer. 2012 Jul 10;107(2):260-5
references: 22699822
references: Stat Med. 2011 Feb 20;30(4):377-99
references: 21225900
references: BMC Med Res Methodol. 2010;10:81
references: 20846460
references: BMJ. 2010;340:c2442
references: 20466793
references: J Clin Epidemiol. 2010 Feb;63(2):205-14
references: 19596181
references: BMC Med Res Methodol. 2009;9:57
references: 19638200
references: BMJ. 2009;339:b2584
references: 19584409
references: BMJ. 2009;338:b2393
references: 19564179
references: Eur Heart J. 2014 Aug 7;35(30):2010-20
references: 24126876
references: Am J Epidemiol. 2014 Aug 1;180(3):318-24
references: 24966219
references: Br J Cancer. 2014 Jan 7;110(1):42-8
references: 24253502