External validity of risk models: Use of benchmark values to disentangle a case-mix effect from incorrect coefficients.
Various performance measures related to calibration and discrimination are available for the assessment of risk models. When the validity of a risk model is assessed in a new population, estimates of the model's performance can be influenced in several ways. The regression coefficients can be incorrect, which indeed results in an invalid model. However, the distribution of patient characteristics (case mix) may also influence the performance of the model. Here the authors consider a number of typical situations that can be encountered in external validation studies. Theoretical relations between differences in development and validation samples and performance measures are studied by simulation. Benchmark values for the performance measures are proposed to disentangle a case-mix effect from incorrect regression coefficients, when interpreting the model's estimated performance in validation samples. The authors demonstrate the use of the benchmark values using data on traumatic brain injury obtained from the International Tirilazad Trial and the North American Tirilazad Trial (1991-1994).
Vergouwe
Yvonne
Y
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, the Netherlands. y.vergouwe@umcutrecht.nl
Moons
Karel G M
KG
Steyerberg
Ewout W
EW
D019985:Benchmarking / Q000706:statistics & numerical data*; D003198:Computer Simulation; D003953:Diagnosis-Related Groups / Q000706:statistics & numerical data*; D006801:Humans; D015233:Models, Statistical; D012307:Risk Factors
Ann Intern Med. 1999 Mar 16;130(6):515-24
10075620
J Natl Cancer Inst. 1989 Dec 20;81(24):1879-86
2593165
Stat Med. 1995 Sep 30;14(18):1999-2008
8677400
J Clin Epidemiol. 2008 Apr;61(4):331-43
18313557
PLoS Med. 2008 Aug 5;5(8):e165; discussion e165
18684008
Clin Cancer Res. 2008 Oct 1;14(19):5977-83
18829476
J Neurosurg. 2002 Sep;97(3):549-57
12296638
J Neurosurg. 1998 Oct;89(4):519-25
9761043
J Clin Epidemiol. 2001 Aug;54(8):774-81
11470385
Stat Med. 2000 Feb 29;19(4):453-73
10694730
Stat Med. 2000 Apr 30;19(8):1059-79
10790680
Stat Med. 1990 Nov;9(11):1303-25
2277880
J Clin Epidemiol. 2003 Sep;56(9):826-32
14505766
Stat Med. 1997 May 15;16(9):965-80
9160492
Stat Med. 2004 Aug 30;23(16):2567-86
15287085
Br J Cancer. 1994 Jun;69(6):979-85
8198989
Circulation. 1998 May 12;97(18):1837-47
9603539
Biometrics. 2005 Mar;61(1):92-105
15737082
JAMA. 1982 May 14;247(18):2543-6
7069920
Stat Med. 1991 Aug;10(8):1213-26
1925153
Eur J Cancer Prev. 2004 Jun;13(3):183-91
15167217
J Clin Epidemiol. 2005 May;58(5):475-83
15845334
Cerebrovasc Dis. 2001;12(3):159-70
11641579
Biom J. 2008 Aug;50(4):457-79
18663757
Radiology. 1982 Apr;143(1):29-36
7063747
Cancer Treat Rep. 1985 Oct;69(10):1071-77
4042087
JAMA. 2001 Jul 11;286(2):180-7
11448281
Epidemiology. 2010 Jan;21(1):128-38
20010215
Biometrics. 2000 Jun;56(2):337-44
10877287
Stat Med. 1996 Oct 15;15(19):1987-97
8896134