title: Small improvement in the area under the receiver operating characteristic curve indicated small changes in predicted risks.
abstract: Adding risk factors to a prediction model often increases the area under the receiver operating characteristic curve (AUC) only slightly, particularly when the AUC of the model was already high. We investigated whether a risk factor that minimally improves the AUC may nevertheless improve the predictive ability of the model, assessed by integrated discrimination improvement (IDI).
  
  
  
  We simulated data sets with risk factors and event status for 100,000 hypothetical individuals and created prediction models with AUCs between 0.50 and 0.95. We added a single risk factor for which the effect was modeled as a certain odds ratio (OR 2, 4, 8) or AUC increment (ΔAUC 0.01, 0.02, 0.03).
  
  
  
  Across all AUC values of the baseline model, for a risk factor with the same OR, both ΔAUC and IDI were lower when the AUC of the baseline model was higher. When the increment in AUC was small (ΔAUC 0.01), the IDI was also small, except when the AUC of the baseline model was >0.90.
  
  
  
  When the addition of a risk factor shows minimal improvement in AUC, predicted risks generally show minimal changes too. Updating risk models with strong risk factors may be informative for a subgroup of individuals, but not at the population level. The AUC may not be as insensitive as is frequently argued.
authors.affiliation: Department of Clinical Genetics/EMGO Institute for Health and Care Research, Section Community Genetics, VU University Medical Center, PO Box 7057 (BS7 A-529), 1007 MB, Amsterdam, The Netherlands.
authors.forename: Forike K
authors.initials: FK
authors.lastname: Martens
authors.affiliation: Department of Clinical Genetics/EMGO Institute for Health and Care Research, Section Community Genetics, VU University Medical Center, PO Box 7057 (BS7 A-529), 1007 MB, Amsterdam, The Netherlands.
authors.forename: Elisa C M
authors.initials: EC
authors.lastname: Tonk
authors.affiliation: Department of Clinical Genetics/EMGO Institute for Health and Care Research, Section Community Genetics, VU University Medical Center, PO Box 7057 (BS7 A-529), 1007 MB, Amsterdam, The Netherlands.
authors.forename: Jannigje G
authors.initials: JG
authors.lastname: Kers
authors.affiliation: Department of Clinical Genetics/EMGO Institute for Health and Care Research, Section Community Genetics, VU University Medical Center, PO Box 7057 (BS7 A-529), 1007 MB, Amsterdam, The Netherlands; Department of Epidemiology, Rollins School of Public Health, Emory University, 1518 Clifton Road NE, Atlanta, GA 30322, USA. Electronic address: cecile.janssens@emory.edu.
authors.forename: A Cecile J W
authors.initials: AC
authors.lastname: Janssens
mesh_terms: D019540:Area Under Curve; D015340:Epidemiologic Research Design; D006801:Humans; D016017:Odds Ratio; D012372:ROC Curve; D018570:Risk Assessment / Q000379:methods* / Q000706:statistics & numerical data*; D012307:Risk Factors
keywords: Area under curve; Discrimination; Prediction; Risk assessment; Risk factors; Risk models