title: Assessing the goodness of fit of personal risk models.
abstract: We describe a flexible family of tests for evaluating the goodness of fit (calibration) of a pre-specified personal risk model to the outcomes observed in a longitudinal cohort. Such evaluation involves using the risk model to assign each subject an absolute risk of developing the outcome within a given time from cohort entry and comparing subjects' assigned risks with their observed outcomes. This comparison involves several issues. For example, subjects followed only for part of the risk period have unknown outcomes. Moreover, existing tests do not reveal the reasons for poor model fit when it occurs, which can reflect misspecification of the model's hazards for the competing risks of outcome development and death. To address these issues, we extend the model-specified hazards for outcome and death, and use score statistics to test the null hypothesis that the extensions are unnecessary. Simulated cohort data applied to risk models whose outcome and mortality hazards agreed and disagreed with those generating the data show that the tests are sensitive to poor model fit, provide insight into the reasons for poor fit, and accommodate a wide range of model misspecification. We illustrate the methods by examining the calibration of two breast cancer risk models as applied to a cohort of participants in the Breast Cancer Family Registry. The methods can be implemented using the Risk Model Assessment Program, an R package freely available at http://stanford.edu/~ggong/rmap/.
authors.affiliation: Department of Health Research and Policy, Stanford University School of Medicine, Stanford, CA 94305, U.S.A.
authors.forename: Gail
authors.initials: G
authors.lastname: Gong
authors.forename: Anne S
authors.initials: AS
authors.lastname: Quante
authors.forename: Mary Beth
authors.initials: MB
authors.lastname: Terry
authors.forename: Alice S
authors.initials: AS
authors.lastname: Whittemore
mesh_terms: D056808:Biostatistics; D001943:Breast Neoplasms / Q000453:epidemiology; D015331:Cohort Studies; D003198:Computer Simulation; D005260:Female; D006801:Humans; D008137:Longitudinal Studies; D015233:Models, Statistical; D057285:Precision Medicine / Q000706:statistics & numerical data; D011336:Probability; D016016:Proportional Hazards Models; D012044:Regression Analysis; D012306:Risk; D018570:Risk Assessment / Q000706:statistics & numerical data; D012307:Risk Factors
keywords: absolute risk; cohort data; efficient score statistics; goodness of fit; personal disease risk; standardized residuals
references.citation: J Natl Cancer Inst. 1999 Sep 15;91(18):1541-8
references.pmid: 10491430
references.citation: J Natl Cancer Inst. 1989 Dec 20;81(24):1879-86
references.pmid: 2593165
references.citation: Breast Cancer Res. 2004;6(4):R375-89
references.pmid: 15217505
references.citation: Stat Med. 2004 Apr 15;23(7):1111-30
references.pmid: 15057881
references.citation: Stat Methods Med Res. 2016 Aug;25(4):1313-29
references.pmid: 23592716
references.citation: Breast Cancer Res. 2012 Nov 05;14(6):R144
references.pmid: 23127309
references.citation: J Natl Cancer Inst. 2007 Dec 5;99(23):1782-92
references.pmid: 18042936