title: A note on the evaluation of novel biomarkers: do not rely on integrated discrimination improvement and net reclassification index.
abstract: The 'integrated discrimination improvement' (IDI) and the 'net reclassification index' (NRI) are statistics proposed as measures of the incremental prognostic impact that a new biomarker will have when added to an existing prediction model for a binary outcome. By design, both measures were meant to be intuitively appropriate, and the IDI and NRI formulae do look intuitively plausible. Both have become increasingly popular. We shall argue, however, that their use is not always safe. If IDI and NRI are used to measure gain in prediction performance, then poorly calibrated models may appear advantageous, and in a simulation study, even the model that actually generates the data (and hence is the best possible model) can be improved on without adding measured information. We illustrate these shortcomings in actual cancer data as well as by Monte Carlo simulations. In these examples, we contrast IDI and NRI with the area under ROC and the Brier score. Unlike IDI and NRI, these traditional measures have the characteristic that prognostic performance cannot be accidentally or deliberately inflated.
authors.affiliation: Department of Biostatistics, University of Copenhagen, Denmark.
authors.forename: Jørgen
authors.initials: J
authors.lastname: Hilden
authors.forename: Thomas A
authors.initials: TA
authors.lastname: Gerds
mesh_terms: D000970:Antineoplastic Agents / Q000627:therapeutic use; D015415:Biomarkers / Q000032:analysis*; D056808:Biostatistics; D001943:Breast Neoplasms / Q000188:drug therapy / Q000378:metabolism; D003198:Computer Simulation; D016002:Discriminant Analysis; D015251:Epirubicin / Q000627:therapeutic use; D005260:Female; D006801:Humans; D015233:Models, Statistical; D009010:Monte Carlo Method; D011379:Prognosis; D012372:ROC Curve; D011960:Receptors, Estrogen / Q000378:metabolism; D012044:Regression Analysis
chemical_list: D000970:Antineoplastic Agents; D015415:Biomarkers; D011960:Receptors, Estrogen; D015251:Epirubicin
keywords: IDI; NRI; biomarker; prediction; prognostic models; proper scoring rules