title: External validation of prognostic models for critically ill patients required substantial sample sizes.
abstract: OBJECTIVE
  To investigate the behavior of predictive performance measures that are commonly used in external validation of prognostic models for outcome at intensive care units (ICUs).
  
  
  METHODS
  Four prognostic models (Simplified Acute Physiology Score II, the Acute Physiology and Chronic Health Evaluation II, and the Mortality Probability Models II) were evaluated in the Dutch National Intensive Care Evaluation registry database. For each model discrimination (AUC), accuracy (Brier score), and two calibration measures were assessed on data from 41,239 ICU admissions. This validation procedure was repeated with smaller subsamples randomly drawn from the database, and the results were compared with those obtained on the entire data set.
  
  
  RESULTS
  Differences in performance between the models were small. The AUC and Brier score showed large variation with small samples. Standard errors of AUC values were accurate but the power to detect differences in performance was low. Calibration tests were extremely sensitive to sample size. Direct comparison of performance, without statistical analysis, was unreliable with either measure.
  
  
  CONCLUSIONS
  Substantial sample sizes are required for performance assessment and model comparison in external validation. Calibration statistics and significance tests should not be used in these settings. Instead, a simple customization method to repair lack-of-fit problems is recommended.
authors.affiliation: Department of Medical Informatics, Academic Medical Center--Universiteit van Amsterdam, Amsterdam, the Netherlands. n.b.peek@amc.uva.nl
authors.forename: N
authors.initials: N
authors.lastname: Peek
authors.forename: D G T
authors.initials: DG
authors.lastname: Arts
authors.forename: R J
authors.initials: RJ
authors.lastname: Bosman
authors.forename: P H J
authors.initials: PH
authors.lastname: van der Voort
authors.forename: N F
authors.initials: NF
authors.lastname: de Keizer
mesh_terms: D000368:Aged; D002138:Calibration; D016638:Critical Illness / Q000401:mortality*; D004812:Epidemiologic Methods; D005260:Female; D006801:Humans; D007362:Intensive Care Units; D008297:Male; D008875:Middle Aged; D009426:Netherlands / Q000453:epidemiology; D017063:Outcome Assessment, Health Care / Q000379:methods*; D011379:Prognosis