Development and validation of clinical prediction models: marginal differences between logistic regression, penalized maximum likelihood estimation, and genetic programming.
OBJECTIVE
Many prediction models are developed by multivariable logistic regression. However, there are several alternative methods to develop prediction models. We compared the accuracy of a model that predicts the presence of deep venous thrombosis (DVT) when developed by four different methods.


METHODS
We used the data of 2,086 primary care patients suspected of DVT, which included 21 candidate predictors. The cohort was split into a derivation set (1,668 patients, 329 with DVT) and a validation set (418 patients, 86 with DVT). Also, 100 cross-validations were conducted in the full cohort. The models were developed by logistic regression, logistic regression with shrinkage by bootstrapping techniques, logistic regression with shrinkage by penalized maximum likelihood estimation, and genetic programming. The accuracy of the models was tested by assessing discrimination and calibration.


RESULTS
There were only marginal differences in the discrimination and calibration of the models in the validation set and cross-validations.


CONCLUSIONS
The accuracy measures of the models developed by the four different methods were only slightly different, and the 95% confidence intervals were mostly overlapped. We have shown that models with good predictive accuracy are most likely developed by sensible modeling strategies rather than by complex development methods.
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands. kristel.janssen@mapigroup.com
Kristel J M
KJ
Janssen
Ivar
I
Siccama
Yvonne
Y
Vergouwe
Hendrik
H
Koffijberg
T P A
TP
Debray
Maarten
M
Keijzer
Diederick E
DE
Grobbee
Karel G M
KG
Moons
D015331:Cohort Studies; D016001:Confidence Intervals; D005260:Female; D006801:Humans; D016013:Likelihood Functions; D016015:Logistic Models; D008297:Male; D008432:Mathematical Computing; D008875:Middle Aged; D008957:Models, Genetic; D011237:Predictive Value of Tests; D011247:Pregnancy; D011250:Pregnancy Complications, Hematologic / Q000175:diagnosis* / Q000235:genetics; D012372:ROC Curve; D015203:Reproducibility of Results; D020246:Venous Thrombosis / Q000175:diagnosis* / Q000235:genetics