title: Assessing Discriminative Performance at External Validation of Clinical Prediction Models.
abstract: BACKGROUND
  External validation studies are essential to study the generalizability of prediction models. Recently a permutation test, focusing on discrimination as quantified by the c-statistic, was proposed to judge whether a prediction model is transportable to a new setting. We aimed to evaluate this test and compare it to previously proposed procedures to judge any changes in c-statistic from development to external validation setting.
  
  
  METHODS
  We compared the use of the permutation test to the use of benchmark values of the c-statistic following from a previously proposed framework to judge transportability of a prediction model. In a simulation study we developed a prediction model with logistic regression on a development set and validated them in the validation set. We concentrated on two scenarios: 1) the case-mix was more heterogeneous and predictor effects were weaker in the validation set compared to the development set, and 2) the case-mix was less heterogeneous in the validation set and predictor effects were identical in the validation and development set. Furthermore we illustrated the methods in a case study using 15 datasets of patients suffering from traumatic brain injury.
  
  
  RESULTS
  The permutation test indicated that the validation and development set were homogenous in scenario 1 (in almost all simulated samples) and heterogeneous in scenario 2 (in 17%-39% of simulated samples). Previously proposed benchmark values of the c-statistic and the standard deviation of the linear predictors correctly pointed at the more heterogeneous case-mix in scenario 1 and the less heterogeneous case-mix in scenario 2.
  
  
  CONCLUSIONS
  The recently proposed permutation test may provide misleading results when externally validating prediction models in the presence of case-mix differences between the development and validation population. To correctly interpret the c-statistic found at external validation it is crucial to disentangle case-mix differences from incorrect regression coefficients.
authors.affiliation: Department of Public Health, Erasmus MC-University medical center, Rotterdam, the Netherlands.
authors.forename: Daan
authors.initials: D
authors.lastname: Nieboer
authors.affiliation: Department of Science, Medical Center Alkmaar/Inholland University, Alkmaar, the Netherlands.
authors.forename: Tjeerd
authors.initials: T
authors.lastname: van der Ploeg
authors.affiliation: Department of Public Health, Erasmus MC-University medical center, Rotterdam, the Netherlands.
authors.forename: Ewout W
authors.initials: EW
authors.lastname: Steyerberg
mesh_terms: D001930:Brain Injuries / Q000503:physiopathology*; D006801:Humans; D016015:Logistic Models; D008962:Models, Theoretical; D011379:Prognosis