title: External validity of risk models: Use of benchmark values to disentangle a case-mix effect from incorrect coefficients.
abstract: Various performance measures related to calibration and discrimination are available for the assessment of risk models. When the validity of a risk model is assessed in a new population, estimates of the model's performance can be influenced in several ways. The regression coefficients can be incorrect, which indeed results in an invalid model. However, the distribution of patient characteristics (case mix) may also influence the performance of the model. Here the authors consider a number of typical situations that can be encountered in external validation studies. Theoretical relations between differences in development and validation samples and performance measures are studied by simulation. Benchmark values for the performance measures are proposed to disentangle a case-mix effect from incorrect regression coefficients, when interpreting the model's estimated performance in validation samples. The authors demonstrate the use of the benchmark values using data on traumatic brain injury obtained from the International Tirilazad Trial and the North American Tirilazad Trial (1991-1994).
authors.affiliation: Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, the Netherlands. y.vergouwe@umcutrecht.nl
authors.forename: Yvonne
authors.initials: Y
authors.lastname: Vergouwe
authors.forename: Karel G M
authors.initials: KG
authors.lastname: Moons
authors.forename: Ewout W
authors.initials: EW
authors.lastname: Steyerberg
mesh_terms: D019985:Benchmarking / Q000706:statistics & numerical data*; D003198:Computer Simulation; D003953:Diagnosis-Related Groups / Q000706:statistics & numerical data*; D006801:Humans; D015233:Models, Statistical; D012307:Risk Factors
references.citation: Ann Intern Med. 1999 Mar 16;130(6):515-24
references.pmid: 10075620
references.citation: J Natl Cancer Inst. 1989 Dec 20;81(24):1879-86
references.pmid: 2593165
references.citation: Stat Med. 1995 Sep 30;14(18):1999-2008
references.pmid: 8677400
references.citation: J Clin Epidemiol. 2008 Apr;61(4):331-43
references.pmid: 18313557
references.citation: PLoS Med. 2008 Aug 5;5(8):e165; discussion e165
references.pmid: 18684008
references.citation: Clin Cancer Res. 2008 Oct 1;14(19):5977-83
references.pmid: 18829476
references.citation: J Neurosurg. 2002 Sep;97(3):549-57
references.pmid: 12296638
references.citation: J Neurosurg. 1998 Oct;89(4):519-25
references.pmid: 9761043
references.citation: J Clin Epidemiol. 2001 Aug;54(8):774-81
references.pmid: 11470385
references.citation: Stat Med. 2000 Feb 29;19(4):453-73
references.pmid: 10694730
references.citation: Stat Med. 2000 Apr 30;19(8):1059-79
references.pmid: 10790680
references.citation: Stat Med. 1990 Nov;9(11):1303-25
references.pmid: 2277880
references.citation: J Clin Epidemiol. 2003 Sep;56(9):826-32
references.pmid: 14505766
references.citation: Stat Med. 1997 May 15;16(9):965-80
references.pmid: 9160492
references.citation: Stat Med. 2004 Aug 30;23(16):2567-86
references.pmid: 15287085
references.citation: Br J Cancer. 1994 Jun;69(6):979-85
references.pmid: 8198989
references.citation: Circulation. 1998 May 12;97(18):1837-47
references.pmid: 9603539
references.citation: Biometrics. 2005 Mar;61(1):92-105
references.pmid: 15737082
references.citation: JAMA. 1982 May 14;247(18):2543-6
references.pmid: 7069920
references.citation: Stat Med. 1991 Aug;10(8):1213-26
references.pmid: 1925153
references.citation: Eur J Cancer Prev. 2004 Jun;13(3):183-91
references.pmid: 15167217
references.citation: J Clin Epidemiol. 2005 May;58(5):475-83
references.pmid: 15845334
references.citation: Cerebrovasc Dis. 2001;12(3):159-70
references.pmid: 11641579
references.citation: Biom J. 2008 Aug;50(4):457-79
references.pmid: 18663757
references.citation: Radiology. 1982 Apr;143(1):29-36
references.pmid: 7063747
references.citation: Cancer Treat Rep. 1985 Oct;69(10):1071-77
references.pmid: 4042087
references.citation: JAMA. 2001 Jul 11;286(2):180-7
references.pmid: 11448281
references.citation: Epidemiology. 2010 Jan;21(1):128-38
references.pmid: 20010215
references.citation: Biometrics. 2000 Jun;56(2):337-44
references.pmid: 10877287
references.citation: Stat Med. 1996 Oct 15;15(19):1987-97
references.pmid: 8896134