title: Assessing improvement in disease prediction using net reclassification improvement: impact of risk cut-offs and number of risk categories.
abstract: Net reclassification improvement (NRI) has received much attention for comparing risk prediction models, and might be preferable over the area under the receiver operating characteristics (ROC) curve to indicate changes in predictive ability. We investigated the influence of the choice of risk cut-offs and number of risk categories on the NRI. Using data of the European Prospective Investigation into Cancer and Nutrition-Potsdam study, three diabetes prediction models were compared according to ROC area and NRI with varying cut-offs for two and three risk categories and varying numbers of risk categories. When compared with a basic model, including age, anthropometry, and hypertension status, a model extension by waist circumference improved discrimination from 0.720 to 0.831 (0.111 [0.097-0.125]) while increase in ROC-AUC from 0.831 to 0.836 (0.006 [0.002-0.009]) indicated moderate improvement when additionally considering diet and physical activity. However, NRI based on these two model comparisons varied with varying cut-offs for two (range: 5.59-23.20%; -0.79 to 4.09%) and three risk categories (20.37-40.15%; 1.22-4.34%). This variation was more pronounced in the model extension showing a larger difference in ROC-AUC. NRI increased with increasing numbers of categories from minimum NRIs of 18.41 and 0.46% to approximately category-free NRIs of 79.61 and 19.22%, but not monotonically. There was a similar pattern for this increase in both model comparisons. In conclusion, the choice of risk cut-offs and number of categories has a substantial impact on NRI. A limited number of categories should only be used if categories have strong clinical importance.
authors.affiliation: Department of Molecular Epidemiology, German Institute of Human Nutrition Potsdam-Rehbruecke, Arthur-Scheunert-Allee 114-116, 14558 Nuthetal, Germany. Kristin.Muehlenbruch@dife.de
authors.forename: Kristin
authors.initials: K
authors.lastname: Mühlenbruch
authors.forename: Alexandros
authors.initials: A
authors.lastname: Heraclides
authors.forename: Ewout W
authors.initials: EW
authors.lastname: Steyerberg
authors.forename: Hans-Georg
authors.initials: HG
authors.lastname: Joost
authors.forename: Heiner
authors.initials: H
authors.lastname: Boeing
authors.forename: Matthias B
authors.initials: MB
authors.lastname: Schulze
mesh_terms: D000328:Adult; D019540:Area Under Curve; D003924:Diabetes Mellitus, Type 2 / Q000175:diagnosis* / Q000453:epidemiology; D005260:Female; D006801:Humans; D015994:Incidence; D008297:Male; D008954:Models, Biological; D011237:Predictive Value of Tests; D011446:Prospective Studies; D012372:ROC Curve; D015203:Reproducibility of Results; D018570:Risk Assessment / Q000145:classification* / Q000706:statistics & numerical data; D012307:Risk Factors; D044465:White People