title: Predictive accuracy of novel risk factors and markers: A simulation study of the sensitivity of different performance measures for the Cox proportional hazards regression model.
abstract: Predicting outcomes that occur over time is important in clinical, population health, and health services research. We compared changes in different measures of performance when a novel risk factor or marker was added to an existing Cox proportional hazards regression model. We performed Monte Carlo simulations for common measures of performance: concordance indices ( c, including various extensions to survival outcomes), Royston's D index, R2-type measures, and Chambless' adaptation of the integrated discrimination improvement to survival outcomes. We found that the increase in performance due to the inclusion of a risk factor tended to decrease as the performance of the reference model increased. Moreover, the increase in performance increased as the hazard ratio or the prevalence of a binary risk factor increased. Finally, for the concordance indices and R2-type measures, the absolute increase in predictive accuracy due to the inclusion of a risk factor was greater when the observed event rate was higher (low censoring). Amongst the different concordance indices, Chambless and Diao's c-statistic exhibited the greatest increase in predictive accuracy when a novel risk factor was added to an existing model. Amongst the different R2-type measures, O'Quigley et al.'s modification of Nagelkerke's R2index and Kent and O'Quigley's [Formula: see text] displayed the greatest sensitivity to the addition of a novel risk factor or marker. These methods were then applied to a cohort of 8635 patients hospitalized with heart failure to examine the added benefit of a point-based scoring system for predicting mortality after initial adjustment with patient age alone.
authors: 1 Institute for Clinical Evaluative Sciences, Toronto, Canada.|2 Institute of Health Management, Policy and Evaluation, University of Toronto.|3 Schulich Heart Research Program, Sunnybrook Research Institute, Toronto, Canada.
authors: Peter C
authors: PC
authors: Austin
authors: 4 Duke Clinical Research Institute, Duke University, Durham.|5 Department of Biostatistics and Bioinformatics, Duke University School of Medicine.
authors: Michael J
authors: MJ
authors: Pencinca
authors: 6 Department of Public Health, Erasmus MC - University Medical Center Rotterdam, Rotterdam, The Netherlands.
authors: Ewout W
authors: EW
authors: Steyerberg
mesh_terms: D000367:Age Factors; D015415:Biomarkers; D006333:Heart Failure / Q000401:mortality; D006801:Humans; D009010:Monte Carlo Method; D016016:Proportional Hazards Models; D012307:Risk Factors
chemical_list: D015415:Biomarkers
keywords: Cox proportional hazards model; Monte Carlo simulations; Survival analysis; discrimination; model performance; predictive accuracy; predictive models; risk factors
references: Stat Methods Med Res. 2017 Apr;26(2):796-808
references: 25411322
references: Stat Med. 2012 Jul 10;31(15):1543-53
references: 22344892
references: Stat Med. 1996 Feb 28;15(4):361-87
references: 8668867
references: Stat Med. 2013 Feb 20;32(4):661-72
references: 22961910
references: JAMA. 2009 Dec 2;302(21):2330-7
references: 19923205
references: JAMA. 2009 Dec 2;302(21):2345-52
references: 19952321
references: Stat Med. 1996 Oct 15;15(19):1999-2012
references: 8896135
references: JAMA. 2003 Nov 19;290(19):2581-7
references: 14625335
references: Stat Med. 2012 Oct 15;31(23):2644-59
references: 22764064
references: Stat Med. 2005 Feb 15;24(3):479-89
references: 15532086
references: Stat Med. 2006 Oct 30;25(20):3474-86
references: 16220486
references: Stat Med. 2008 Jan 30;27(2):157-72; discussion 207-12
references: 17569110
references: Stat Med. 2012 Oct 15;31(23):2627-43
references: 21520455
references: Proc Natl Acad Sci U S A. 1994 Mar 15;91(6):2310-4
references: 8134393
references: Stat Med. 2011 May 10;30(10):1105-17
references: 21484848
references: Genet Med. 2007 Aug;9(8):528-35
references: 17700391
references: Biometrics. 2000 Mar;56(1):249-55
references: 10783803
references: J Clin Epidemiol. 2001 Aug;54(8):774-81
references: 11470385
references: BMC Med Res Methodol. 2012 Jun 20;12:82
references: 22716998
references: Stat Med. 1990 May;9(5):487-503
references: 2349402
references: Stat Med. 2010 Mar 30;29(7-8):818-29
references: 20213714
references: Stat Med. 2004 Jul 15;23(13):2109-23
references: 15211606
references: J Med Syst. 1997 Aug;21(4):229-38
references: 9442437
references: J Clin Epidemiol. 2010 Feb;63(2):142-53
references: 19716262
references: Stat Med. 2011 Jan 15;30(1):22-38
references: 20827726
references: Comput Methods Programs Biomed. 2007 Aug;87(2):132-7
references: 17601627
references: Stat Med. 2004 Mar 15;23(5):723-48
references: 14981672
references: Stat Med. 1999 Sep 15-30;18(17-18):2529-45
references: 10474158
references: Stat Med. 2005 Jun 15;24(11):1713-23
references: 15724232