title: External validity of predictive models: a comparison of logistic regression, classification trees, and neural networks.
abstract: OBJECTIVE
  The utility of predictive models depends on their external validity, that is, their ability to maintain accuracy when applied to patients and settings different from those on which the models were developed. We report a simulation study that compared the external validity of standard logistic regression (LR1), logistic regression with piecewise-linear and quadratic terms (LR2), classification trees, and neural networks (NNETs).
  
  
  METHODS
  We developed predictive models on data simulated from a specified population and on data from perturbed forms of the population not representative of the original distribution. All models were tested on new data generated from the population.
  
  
  RESULTS
  The performance of LR2 was superior to that of the other model types when the models were developed on data sampled from the population (mean receiver operating characteristic [ROC] areas 0.769, 0.741, 0.724, and 0.682, for LR2, LR1, NNETs, and trees, respectively) and when they were developed on nonrepresentative data (mean ROC areas 0.734, 0.713, 0.703, and 0.667). However, when the models developed using nonrepresentative data were compared with models developed from data sampled from the population, LR2 had the greatest loss in performance.
  
  
  CONCLUSIONS
  Our results highlight the necessity of external validation to test the transportability of predictive models.
authors.affiliation: Division of Clinical Care Research, Department of Medicine, Tufts-New England Medical Center, and Tufts University School of Medicine, 750 Washington Street, Boston, MA 02111, USA. nterrin@tufts-nemc.org
authors.forename: Norma
authors.initials: N
authors.lastname: Terrin
authors.forename: Christopher H
authors.initials: CH
authors.lastname: Schmid
authors.forename: John L
authors.initials: JL
authors.lastname: Griffith
authors.forename: Ralph B
authors.initials: RB
authors.lastname: D'Agostino
authors.forename: Harry P
authors.initials: HP
authors.lastname: Selker
mesh_terms: D002965:Classification; D003198:Computer Simulation; D003933:Diagnosis; D006801:Humans; D016015:Logistic Models; D015233:Models, Statistical; D016571:Neural Networks, Computer; D011379:Prognosis; D015203:Reproducibility of Results