title: Assessing calibration of prognostic risk scores.
abstract: Current methods used to assess calibration are limited, particularly in the assessment of prognostic models. Methods for testing and visualizing calibration (e.g. the Hosmer-Lemeshow test and calibration slope) have been well thought out in the binary regression setting. However, extension of these methods to Cox models is less well known and could be improved. We describe a model-based framework for the assessment of calibration in the binary setting that provides natural extensions to the survival data setting. We show that Poisson regression models can be used to easily assess calibration in prognostic models. In addition, we show that a calibration test suggested for use in survival data has poor performance. Finally, we apply these methods to the problem of external validation of a risk score developed for the general population when assessed in a special patient population (i.e. patients with particular comorbidities, such as rheumatoid arthritis).
authors.affiliation: Division of Biomedical Statistics and Informatics, Department of Health Sciences Research, Mayo Clinic, Mayo Clinic College of Medicine, Rochester, Minnesota, USA crowson@mayo.edu.
authors.forename: Cynthia S
authors.initials: CS
authors.lastname: Crowson
authors.affiliation: Division of Biomedical Statistics and Informatics, Department of Health Sciences Research, Mayo Clinic, Mayo Clinic College of Medicine, Rochester, Minnesota, USA.
authors.forename: Elizabeth J
authors.initials: EJ
authors.lastname: Atkinson
authors.affiliation: Division of Biomedical Statistics and Informatics, Department of Health Sciences Research, Mayo Clinic, Mayo Clinic College of Medicine, Rochester, Minnesota, USA.
authors.forename: Terry M
authors.initials: TM
authors.lastname: Therneau
mesh_terms: D000328:Adult; D001172:Arthritis, Rheumatoid; D002138:Calibration; D006801:Humans; D011379:Prognosis; D016016:Proportional Hazards Models; D015203:Reproducibility of Results; D012306:Risk
keywords: Cox model; Poisson; calibration; prognostic risk scores; standardized incidence ratio; survival
references.citation: Lifetime Data Anal. 1998;4(2):109-20
references.pmid: 9658770
references.citation: Stat Med. 1984 Jul-Sep;3(3):301-2
references.pmid: 6484380
references.citation: Biometrics. 1995 Jun;51(2):600-14
references.pmid: 7662848
references.citation: Med Decis Making. 1993 Jan-Mar;13(1):49-58
references.pmid: 8433637
references.citation: Stat Med. 1985 Jan-Mar;4(1):105-9
references.pmid: 3992069
references.citation: Am J Cardiol. 2012 Aug 1;110(3):420-4
references.pmid: 22521305
references.citation: J Rheumatol. 2008 Jun;35(6):1009-14
references.pmid: 18412312
references.citation: Biometrics. 1999 Jun;55(2):580-4
references.pmid: 11318217
references.citation: Lifetime Data Anal. 1996;2(4):315-28
references.pmid: 9384628
references.citation: J Clin Epidemiol. 2007 May;60(5):491-501
references.pmid: 17419960
references.citation: Epidemiology. 2010 Jan;21(1):128-38
references.pmid: 20010215
references.citation: Stat Med. 1997 May 15;16(9):965-80
references.pmid: 9160492
references.citation: Stat Med. 2004 Aug 30;23(16):2567-86
references.pmid: 15287085
references.citation: Biometrics. 1983 Mar;39(1):173-84
references.pmid: 6871346
references.citation: Circulation. 2008 Feb 12;117(6):743-53
references.pmid: 18212285
references.citation: Lifetime Data Anal. 2004 Sep;10(3):283-91
references.pmid: 15456108
references.citation: IARC Sci Publ. 1987;(82):1-406
references.pmid: 3329634
references.citation: Stat Med. 1996 Feb 28;15(4):361-87
references.pmid: 8668867
references.citation: Stat Med. 1983 Jul-Sep;2(3):381-6
references.pmid: 6648151
references.citation: Circulation. 2012 Apr 10;125(14):1748-56, S1-11
references.pmid: 22399535
references.citation: Can J Anaesth. 2009 Mar;56(3):194-201
references.pmid: 19247740