title: Net risk reclassification p values: valid or misleading?
abstract: BACKGROUND
  The Net Reclassification Index (NRI) and its P value are used to make conclusions about improvements in prediction performance gained by adding a set of biomarkers to an existing risk prediction model. Although proposed only 5 years ago, the NRI has gained enormous traction in the risk prediction literature. Concerns have recently been raised about the statistical validity of the NRI.
  
  
  METHODS
  Using a population dataset of 10000 individuals with an event rate of 10.2%, in which four biomarkers have no predictive ability, we repeatedly simulated studies and calculated the chance that the NRI statistic provides a positive statistically significant result. Subjects for training data (n = 420) and test data (n = 420 or 840) were randomly selected from the population, and corresponding NRI statistics and P values were calculated. For comparison, the change in the area under the receiver operating characteristic curve and likelihood ratio statistics were calculated.
  
  
  RESULTS
  We found that rates of false-positive conclusions based on the NRI statistic were unacceptably high, being 63.0% in the training datasets and 18.8% to 34.4% in the test datasets. False-positive conclusions were rare when using the change in the area under the curve and occurred at the expected rate of approximately 5.0% with the likelihood ratio statistic.
  
  
  CONCLUSIONS
  Conclusions about biomarker performance that are based primarily on a statistically significant NRI statistic should be treated with skepticism. Use of NRI P values in scientific reporting should be halted.
authors.affiliation: Affiliations of authors: Public Health Sciences Division, Fred Hutchinson Cancer Research Center, Seattle, WA (MSP, HJ, CIL); Vaccine and Infectious Disease Division, Fred Hutchinson Cancer Research Center, Seattle, WA (HJ).
authors.forename: Margaret S
authors.initials: MS
authors.lastname: Pepe
authors.forename: Holly
authors.initials: H
authors.lastname: Janes
authors.forename: Christopher I
authors.initials: CI
authors.lastname: Li
mesh_terms: D019540:Area Under Curve; D014408:Biomarkers, Tumor; D016208:Databases, Factual; D005189:False Positive Reactions; D006801:Humans; D015233:Models, Statistical; D016017:Odds Ratio; D011237:Predictive Value of Tests; D011379:Prognosis; D012372:ROC Curve; D018570:Risk Assessment; D012307:Risk Factors
chemical_list: D014408:Biomarkers, Tumor