title: A calibration hierarchy for risk models was defined: from utopia to empirical data.
abstract: Calibrated risk models are vital for valid decision support. We define four levels of calibration and describe implications for model development and external validation of predictions.
  
  
  
  We present results based on simulated data sets.
  
  
  
  A common definition of calibration is "having an event rate of R% among patients with a predicted risk of R%," which we refer to as "moderate calibration." Weaker forms of calibration only require the average predicted risk (mean calibration) or the average prediction effects (weak calibration) to be correct. "Strong calibration" requires that the event rate equals the predicted risk for every covariate pattern. This implies that the model is fully correct for the validation setting. We argue that this is unrealistic: the model type may be incorrect, the linear predictor is only asymptotically unbiased, and all nonlinear and interaction effects should be correctly modeled. In addition, we prove that moderate calibration guarantees nonharmful decision making. Finally, results indicate that a flexible assessment of calibration in small validation data sets is problematic.
  
  
  
  Strong calibration is desirable for individualized decision support but unrealistic and counter productive by stimulating the development of overly complex models. Model development and external validation should focus on moderate calibration.
authors.affiliation: KU Leuven, Department of Development and Regeneration, Herestraat 49 Box 7003, 3000 Leuven, Belgium; Department of Public Health, Erasmus MC, 's-Gravendijkwal 230, 3015 CE Rotterdam, The Netherlands. Electronic address: ben.vancalster@med.kuleuven.be.
authors.forename: Ben
authors.initials: B
authors.lastname: Van Calster
authors.affiliation: Department of Public Health, Erasmus MC, 's-Gravendijkwal 230, 3015 CE Rotterdam, The Netherlands.
authors.forename: Daan
authors.initials: D
authors.lastname: Nieboer
authors.affiliation: Department of Public Health, Erasmus MC, 's-Gravendijkwal 230, 3015 CE Rotterdam, The Netherlands.
authors.forename: Yvonne
authors.initials: Y
authors.lastname: Vergouwe
authors.affiliation: KU Leuven, Department of Development and Regeneration, Herestraat 49 Box 7003, 3000 Leuven, Belgium.
authors.forename: Bavo
authors.initials: B
authors.lastname: De Cock
authors.affiliation: Duke Clinical Research Institute, Duke University, 2400 Pratt Street, Durham, NC 27705, USA; Department of Biostatistics and Bioinformatics, Duke University, 2424 Erwin Road, Durham, NC 27719, USA.
authors.forename: Michael J
authors.initials: MJ
authors.lastname: Pencina
authors.affiliation: Department of Public Health, Erasmus MC, 's-Gravendijkwal 230, 3015 CE Rotterdam, The Netherlands.
authors.forename: Ewout W
authors.initials: EW
authors.lastname: Steyerberg
mesh_terms: D015982:Bias; D002138:Calibration; D003198:Computer Simulation; D003661:Decision Support Techniques; D006801:Humans; D015233:Models, Statistical; D015203:Reproducibility of Results; D012306:Risk; D018570:Risk Assessment
keywords: Calibration; Decision curve analysis; External validation; Loess; Overfitting; Risk prediction models