title: A closed testing procedure to select an appropriate method for updating prediction models.
abstract: Prediction models fitted with logistic regression often show poor performance when applied in populations other than the development population. Model updating may improve predictions. Previously suggested methods vary in their extensiveness of updating the model. We aim to define a strategy in selecting an appropriate update method that considers the balance between the amount of evidence for updating in the new patient sample and the danger of overfitting. We consider recalibration in the large (re-estimation of model intercept); recalibration (re-estimation of intercept and slope) and model revision (re-estimation of all coefficients) as update methods. We propose a closed testing procedure that allows the extensiveness of the updating to increase progressively from a minimum (the original model) to a maximum (a completely revised model). The procedure involves multiple testing with maintaining approximately the chosen type I error rate. We illustrate this approach with three clinical examples: patients with prostate cancer, traumatic brain injury and children presenting with fever. The need for updating the prostate cancer model was completely driven by a different model intercept in the update sample (adjustment: 2.58). Separate testing of model revision against the original model showed statistically significant results, but led to overfitting (calibration slope at internal validation = 0.86). The closed testing procedure selected recalibration in the large as update method, without overfitting. The advantage of the closed testing procedure was confirmed by the other two examples. We conclude that the proposed closed testing procedure may be useful in selecting appropriate update methods for previously developed prediction models. Copyright © 2016 John Wiley & Sons, Ltd.
authors.affiliation: Center for Medical Decision Sciences, Department of Public Health, Erasmus MC, Rotterdam, the Netherlands.
authors.forename: Yvonne
authors.initials: Y
authors.lastname: Vergouwe
authors.affiliation: Center for Medical Decision Sciences, Department of Public Health, Erasmus MC, Rotterdam, the Netherlands.
authors.forename: Daan
authors.initials: D
authors.lastname: Nieboer
authors.affiliation: Department of General Paediatrics, Erasmus MC-Sophia Children's Hospital, Rotterdam, the Netherlands.
authors.forename: Rianne
authors.initials: R
authors.lastname: Oostenbrink
authors.affiliation: Julius Center for Health Sciences and Primary Care, UMC Utrecht, Utrecht, the Netherlands.
authors.forename: Thomas P A
authors.identifier: http://orcid.org/0000-0002-1790-2719
authors.initials: TPA
authors.lastname: Debray
authors.affiliation: Centre of Population Health Sciences, University of Edinburgh, Edinburgh, UK.
authors.forename: Gordon D
authors.initials: GD
authors.lastname: Murray
authors.affiliation: Department of Quantitative Health Sciences, Cleveland Clinic, Cleveland, OH.
authors.forename: Michael W
authors.initials: MW
authors.lastname: Kattan
authors.affiliation: Julius Center for Health Sciences and Primary Care, UMC Utrecht, Utrecht, the Netherlands.
authors.forename: Hendrik
authors.initials: H
authors.lastname: Koffijberg
authors.affiliation: Julius Center for Health Sciences and Primary Care, UMC Utrecht, Utrecht, the Netherlands.
authors.forename: Karel G M
authors.initials: KGM
authors.lastname: Moons
authors.affiliation: Center for Medical Decision Sciences, Department of Public Health, Erasmus MC, Rotterdam, the Netherlands.
authors.forename: Ewout W
authors.identifier: http://orcid.org/0000-0002-7787-0122
authors.initials: EW
authors.lastname: Steyerberg
mesh_terms: D001699:Biometry / Q000379:methods*; D001930:Brain Injuries / Q000453:epidemiology; D002648:Child; D002675:Child, Preschool; D003198:Computer Simulation; D005260:Female; D005334:Fever / Q000453:epidemiology; D006801:Humans; D007223:Infant; D016015:Logistic Models; D008297:Male; D008875:Middle Aged; D011336:Probability; D011471:Prostatic Neoplasms / Q000453:epidemiology; D012044:Regression Analysis; D018570:Risk Assessment / Q000379:methods*
keywords: closed testing procedure; logistic regression; model updating; prediction model