title: A new framework to enhance the interpretation of external validation studies of clinical prediction models.
abstract: OBJECTIVE
  It is widely acknowledged that the performance of diagnostic and prognostic prediction models should be assessed in external validation studies with independent data from "different but related" samples as compared with that of the development sample. We developed a framework of methodological steps and statistical methods for analyzing and enhancing the interpretation of results from external validation studies of prediction models.
  
  
  METHODS
  We propose to quantify the degree of relatedness between development and validation samples on a scale ranging from reproducibility to transportability by evaluating their corresponding case-mix differences. We subsequently assess the models' performance in the validation sample and interpret the performance in view of the case-mix differences. Finally, we may adjust the model to the validation setting.
  
  
  RESULTS
  We illustrate this three-step framework with a prediction model for diagnosing deep venous thrombosis using three validation samples with varying case mix. While one external validation sample merely assessed the model's reproducibility, two other samples rather assessed model transportability. The performance in all validation samples was adequate, and the model did not require extensive updating to correct for miscalibration or poor fit to the validation settings.
  
  
  CONCLUSIONS
  The proposed framework enhances the interpretation of findings at external validation of prediction models.
authors.affiliation: Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Str. 6.131, PO Box 85500, 3508GA Utrecht, The Netherlands. Electronic address: T.Debray@umcutrecht.nl.
authors.forename: Thomas P A
authors.initials: TP
authors.lastname: Debray
authors.affiliation: Department of Public Health, Erasmus Medical Center, Rotterdam, The Netherlands.
authors.forename: Yvonne
authors.initials: Y
authors.lastname: Vergouwe
authors.affiliation: Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Str. 6.131, PO Box 85500, 3508GA Utrecht, The Netherlands.
authors.forename: Hendrik
authors.initials: H
authors.lastname: Koffijberg
authors.affiliation: Department of Public Health, Erasmus Medical Center, Rotterdam, The Netherlands.
authors.forename: Daan
authors.initials: D
authors.lastname: Nieboer
authors.affiliation: Department of Public Health, Erasmus Medical Center, Rotterdam, The Netherlands.
authors.forename: Ewout W
authors.initials: EW
authors.lastname: Steyerberg
authors.affiliation: Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Str. 6.131, PO Box 85500, 3508GA Utrecht, The Netherlands.
authors.forename: Karel G M
authors.initials: KG
authors.lastname: Moons
mesh_terms: D003627:Data Interpretation, Statistical; D005260:Female; D006801:Humans; D008297:Male; D015233:Models, Statistical; D017063:Outcome Assessment, Health Care; D011237:Predictive Value of Tests; D015203:Reproducibility of Results; D054928:Validation Studies as Topic
keywords: Case mix; Generalizability; Prediction model; Reproducibility; Transportability; Validation