Net Reclassification Index and Integrated Discrimination Index Are Not Appropriate for Testing Whether a Biomarker Improves Predictive Performance.
One of the goals of the Critical Path Institute's Predictive Safety Testing Consortium (PSTC) is to promote best practices for evaluating novel markers of drug induced injury. This includes the use of sound statistical methods. For rat studies, these practices have centered around comparing the area under the receiver-operator characteristic curve for each novel injury biomarker to those for the standard markers. In addition, the PSTC has previously used the net reclassification index (NRI) and integrated discrimination index (IDI) to assess the increased certainty provided by each novel injury biomarker when added to the information already provided by the standard markers. Due to their relatively simple interpretations, NRI and IDI have generally been popular measures of predictive performance. However recent literature suggests that significance tests for NRI and IDI can have inflated false positive rates and thus, tests based on these metrics should not be relied upon. Instead, when parametric models are employed to assess the added predictive value of a new marker, following (Pepe, M. S., Kerr, K. F., Longton, G., and Wang, Z. (2013). Testing for improvement in prediction model performance. Stat. Med. 32, 1467-1482), the PSTC recommends that likelihood based methods be used for significance testing.
Pfizer Inc. Worldwide Research & Discovery, Groton, CT 06340.
Peter M
PM
Burch
Merck & Co, Inc, West, Point PA 19486.
Warren E
WE
Glaab
Merck & Co, Inc, West, Point PA 19486.
Daniel J
DJ
Holder
Boehringer Ingelheim Pharmaceuticals Inc, Ridgefield, CT 06877.
Jonathan A
JA
Phillips
Predictive Safety Testing Consortium, Critical Path Institute, Tucson, AZ 85718.
John-Michael
JM
Sauer
Predictive Safety Testing Consortium, Critical Path Institute, Tucson, AZ 85718.
Elizabeth G
EG
Walker
D000818:Animals; D015415:Biomarkers / Q000097:blood / Q000378:metabolism* / Q000652:urine; D004353:Drug Evaluation, Preclinical / Q000639:trends; D064420:Drug-Related Side Effects and Adverse Reactions / Q000097:blood / Q000175:diagnosis* / Q000378:metabolism / Q000652:urine; D015507:Drugs, Investigational / Q000009:adverse effects* / Q000145:classification; D005189:False Positive Reactions; D006801:Humans; D015233:Models, Statistical; D009135:Muscular Diseases / Q000139:chemically induced / Q000175:diagnosis / Q000378:metabolism; D009939:Organizations, Nonprofit; D011237:Predictive Value of Tests; D012372:ROC Curve; D051437:Renal Insufficiency / Q000139:chemically induced / Q000175:diagnosis / Q000378:metabolism; D018675:Toxicity Tests / Q000639:trends; D014481:United States; D015262:Xenobiotics / Q000145:classification / Q000633:toxicity*
D015415:Biomarkers; D015507:Drugs, Investigational; D015262:Xenobiotics
 NRI; IDI; biomarkers < Safety Evaluation; statistics.