title: Calibration drift in regression and machine learning models for acute kidney injury.
abstract: OBJECTIVE
  Predictive analytics create opportunities to incorporate personalized risk estimates into clinical decision support. Models must be well calibrated to support decision-making, yet calibration deteriorates over time. This study explored the influence of modeling methods on performance drift and connected observed drift with data shifts in the patient population.
  
  
  METHODS
  Using 2003 admissions to Department of Veterans Affairs hospitals nationwide, we developed 7 parallel models for hospital-acquired acute kidney injury using common regression and machine learning methods, validating each over 9 subsequent years.
  
  
  RESULTS
  Discrimination was maintained for all models. Calibration declined as all models increasingly overpredicted risk. However, the random forest and neural network models maintained calibration across ranges of probability, capturing more admissions than did the regression models. The magnitude of overprediction increased over time for the regression models while remaining stable and small for the machine learning models. Changes in the rate of acute kidney injury were strongly linked to increasing overprediction, while changes in predictor-outcome associations corresponded with diverging patterns of calibration drift across methods.
  
  
  CONCLUSIONS
  Efficient and effective updating protocols will be essential for maintaining accuracy of, user confidence in, and safety of personalized risk predictions to support decision-making. Model updating protocols should be tailored to account for variations in calibration drift across methods and respond to periods of rapid performance drift rather than be limited to regularly scheduled annual or biannual intervals.
authors.affiliation: Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, TN, USA.
authors.forename: Sharon E
authors.initials: SE
authors.lastname: Davis
authors.affiliation: Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, TN, USA.
authors.forename: Thomas A
authors.initials: TA
authors.lastname: Lasko
authors.affiliation: Department of Biostatistics, Vanderbilt University School of Medicine.
authors.forename: Guanhua
authors.initials: G
authors.lastname: Chen
authors.affiliation: Geriatric Research Education and Clinical Care Service, VA Tennessee Valley Healthcare System, Nashville, TN, USA.|Division of Nephrology, Vanderbilt University School of Medicine, Vanderbilt Center for Kidney Disease and Integrated Program for AKI, Nashville, TN, USA.
authors.forename: Edward D
authors.initials: ED
authors.lastname: Siew
authors.affiliation: Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, TN, USA.|Department of Biostatistics, Vanderbilt University School of Medicine.|Geriatric Research Education and Clinical Care Service, VA Tennessee Valley Healthcare System, Nashville, TN, USA.|Division of General Internal Medicine, Vanderbilt University School of Medicine.
authors.forename: Michael E
authors.initials: ME
authors.lastname: Matheny
mesh_terms: D058186:Acute Kidney Injury; D000368:Aged; D001499:Bayes Theorem; D000066491:Clinical Decision-Making; D003661:Decision Support Techniques; D005260:Female; D006787:Hospitals, Veterans; D006801:Humans; D016015:Logistic Models; D000069550:Machine Learning; D008297:Male; D008875:Middle Aged; D014481:United States
keywords: acute kidney injury; calibration; clinical decision support; clinical prediction; discrimination; machine learning