title: Graphical assessment of incremental value of novel markers in prediction models: From statistical to decision analytical perspectives.
abstract: New markers may improve prediction of diagnostic and prognostic outcomes. We aimed to review options for graphical display and summary measures to assess the predictive value of markers over standard, readily available predictors. We illustrated various approaches using previously published data on 3264 participants from the Framingham Heart Study, where 183 developed coronary heart disease (10-year risk 5.6%). We considered performance measures for the incremental value of adding HDL cholesterol to a prediction model. An initial assessment may consider statistical significance (HR = 0.65, 95% confidence interval 0.53 to 0.80; likelihood ratio p < 0.001), and distributions of predicted risks (densities or box plots) with various summary measures. A range of decision thresholds is considered in predictiveness and receiver operating characteristic curves, where the area under the curve (AUC) increased from 0.762 to 0.774 by adding HDL. We can furthermore focus on reclassification of participants with and without an event in a reclassification graph, with the continuous net reclassification improvement (NRI) as a summary measure. When we focus on one particular decision threshold, the changes in sensitivity and specificity are central. We propose a net reclassification risk graph, which allows us to focus on the number of reclassified persons and their event rates. Summary measures include the binary AUC, the two-category NRI, and decision analytic variants such as the net benefit (NB). Various graphs and summary measures can be used to assess the incremental predictive value of a marker. Important insights for impact on decision making are provided by a simple graph for the net reclassification risk.
authors.affiliation: Department of Public Health, Erasmus MC: University Medical Center Rotterdam, Rotterdam, The Netherlands.
authors.forename: Ewout W
authors.initials: EW
authors.lastname: Steyerberg
authors.affiliation: Department of Public Health, Erasmus MC: University Medical Center Rotterdam, Rotterdam, The Netherlands.
authors.forename: Moniek M
authors.initials: MM
authors.lastname: Vedder
authors.affiliation: Department of Epidemiology, Erasmus MC: University Medical Center Rotterdam, Rotterdam, The Netherlands.|Department of Cardiology, Erasmus MC: University Medical Center Rotterdam, Rotterdam, The Netherlands.
authors.forename: Maarten J G
authors.initials: MJ
authors.lastname: Leening
authors.affiliation: Department of Epidemiology, University Medical Center Groningen, University of Groningen, Groningen, The Netherlands.
authors.forename: Douwe
authors.initials: D
authors.lastname: Postmus
authors.affiliation: Framingham Heart Study, Framingham, MA, USA.
authors.forename: Ralph B
authors.initials: RB
authors.lastname: D'Agostino
authors.affiliation: Department of Public Health, Erasmus MC: University Medical Center Rotterdam, Rotterdam, The Netherlands.|Department of Development and Regeneration, KU Leuven, Leuven, Belgium.
authors.forename: Ben
authors.initials: B
authors.lastname: Van Calster
authors.affiliation: Department of Biostatistics and Bioinformatics, Duke Clinical Research Institute, Duke University, Durham, NC, USA.
authors.forename: Michael J
authors.initials: MJ
authors.lastname: Pencina
mesh_terms: D000328:Adult; D000368:Aged; D019540:Area Under Curve; D015415:Biomarkers / Q000378:metabolism; D008076:Cholesterol, HDL / Q000378:metabolism; D003196:Computer Graphics; D003327:Coronary Disease / Q000453:epidemiology / Q000378:metabolism; D003661:Decision Support Techniques; D006801:Humans; D008875:Middle Aged; D015233:Models, Statistical; D012372:ROC Curve; D012306:Risk
chemical_list: D015415:Biomarkers; D008076:Cholesterol, HDL
keywords: Decision analysis; ROC curve; Reclassification; Regression analysis; Risk assessment