Improving prediction models with new markers: a comparison of updating strategies.
BACKGROUND
New markers hold the promise of improving risk prediction for individual patients. We aimed to compare the performance of different strategies to extend a previously developed prediction model with a new marker.


METHODS
Our motivating example was the extension of a risk calculator for prostate cancer with a new marker that was available in a relatively small dataset. Performance of the strategies was also investigated in simulations. Development, marker and test sets with different sample sizes originating from the same underlying population were generated. A prediction model was fitted using logistic regression in the development set, extended using the marker set and validated in the test set. Extension strategies considered were re-estimating individual regression coefficients, updating of predictions using conditional likelihood ratios (LR) and imputation of marker values in the development set and subsequently fitting a model in the combined development and marker sets. Sample sizes considered for the development and marker set were 500 and 100, 500 and 500, and 100 and 500 patients. Discriminative ability of the extended models was quantified using the concordance statistic (c-statistic) and calibration was quantified using the calibration slope.


RESULTS
All strategies led to extended models with increased discrimination (c-statistic increase from 0.75 to 0.80 in test sets). Strategies estimating a large number of parameters (re-estimation of all coefficients and updating using conditional LR) led to overfitting (calibration slope below 1). Parsimonious methods, limiting the number of coefficients to be re-estimated, or applying shrinkage after model revision, limited the amount of overfitting. Combining the development and marker set using imputation of missing marker values approach led to consistently good performing models in all scenarios. Similar results were observed in the motivating example.


CONCLUSIONS
When the sample with the new marker information is small, parsimonious methods are required to prevent overfitting of a new prediction model. Combining all data with imputation of missing marker values is an attractive option, even if a relatively large marker data set is available.
Department of Public Health, Erasmus MC-University Medical Center Rotterdam, P.O. box 2040, 3000, Rotterdam, CA, The Netherlands. d.nieboer@erasmusmc.nl.
D
D
Nieboer
Department of Public Health, Erasmus MC-University Medical Center Rotterdam, P.O. box 2040, 3000, Rotterdam, CA, The Netherlands.
Y
Y
Vergouwe
Department of Mathematics, Technical University Munich, Munich, Germany.|University of Texas Health Science Center at San Antonio, San Antonio, TX, USA.
Danna P
DP
Ankerst
Department of Urology, Erasmus MC-University Medical Center Rotterdam, Rotterdam, The Netherlands.
Monique J
MJ
Roobol
Department of Public Health, Erasmus MC-University Medical Center Rotterdam, P.O. box 2040, 3000, Rotterdam, CA, The Netherlands.
Ewout W
EW
Steyerberg
Logistic regression; Model updating; Prediction model; Prostate cancer
Eur Urol. 2012 Mar;61(3):577-83
22104592
Stat Med. 2004 Aug 30;23(16):2567-86
15287085
Stat Med. 2014 Aug 30;33(19):3415-8
25042215
Med Decis Making. 2006 Nov-Dec;26(6):565-74
17099194
JAMA. 1982 May 14;247(18):2543-6
7069920
J Clin Epidemiol. 2015 Mar;68(3):279-89
25179855
Prostate. 2008 Nov 1;68(15):1674-80
18712715
Stat Med. 2008 Jan 30;27(2):157-72; discussion 207-12
17569110
PLoS Med. 2013;10(2):e1001380
23393429
J Urol. 2007 Jan;177(1):107-12; discussion 112
17162015
Biom J. 2012 Jan;54(1):127-42
22095849
Stat Med. 1990 Nov;9(11):1303-25
2277880
Stat Med. 2014 Aug 30;33(19):3405-14
23553436
Stat Med. 2000 Dec 30;19(24):3401-15
11122504
PLoS Med. 2013;10(2):e1001381
23393430
Ann Intern Med. 2012 Mar 20;156(6):468-9
22431679
J Urol. 2011 May;185(5):1650-5
21419439
Stat Med. 2006 Dec 30;25(24):4279-92
16947139
J Natl Cancer Inst. 2009 Nov 18;101(22):1538-42
19843888
Chest. 2002 Sep;122(3):1037-57
12226051
Eur Urol. 2014 Nov;66(5):906-12
24361258