title: Comparing observed and predicted mortality among ICUs using different prognostic systems: why do performance assessments differ?
abstract: OBJECTIVE
  To compare ICU performance using standardized mortality ratios generated by the Acute Physiology and Chronic Health Evaluation IVa and a National Quality Forum-endorsed methodology and examine potential reasons for model-based standardized mortality ratio differences.
  
  
  METHODS
  Retrospective analysis of day 1 hospital mortality predictions at the ICU level using Acute Physiology and Chronic Health Evaluation IVa and National Quality Forum models on the same patient cohort.
  
  
  METHODS
  Forty-seven ICUs at 36 U.S. hospitals from January 2008 to May 2013.
  
  
  METHODS
  Eighty-nine thousand three hundred fifty-three consecutive unselected ICU admissions.
  
  
  METHODS
  None.
  
  
  RESULTS
  We assessed standardized mortality ratios for each ICU using data for patients eligible for Acute Physiology and Chronic Health Evaluation IVa and National Quality Forum predictions in order to compare unit-level model performance, differences in ICU rankings, and how case-mix adjustment might explain standardized mortality ratio differences. Hospital mortality was 11.5%. Overall standardized mortality ratio was 0.89 using Acute Physiology and Chronic Health Evaluation IVa and 1.07 using National Quality Forum, the latter having a widely dispersed and multimodal standardized mortality ratio distribution. Model exclusion criteria eliminated mortality predictions for 10.6% of patients for Acute Physiology and Chronic Health Evaluation IVa and 27.9% for National Quality Forum. The two models agreed on the significance and direction of standardized mortality ratio only 45% of the time. Four ICUs had standardized mortality ratios significantly less than 1.0 using Acute Physiology and Chronic Health Evaluation IVa, but significantly greater than 1.0 using National Quality Forum. Two ICUs had standardized mortality ratios exceeding 1.75 using National Quality Forum, but nonsignificant performance using Acute Physiology and Chronic Health Evaluation IVa. Stratification by patient and institutional characteristics indicated that units caring for more severely ill patients and those with a higher percentage of patients on mechanical ventilation had the most discordant standardized mortality ratios between the two predictive models.
  
  
  CONCLUSIONS
  Acute Physiology and Chronic Health Evaluation IVa and National Quality Forum models yield different ICU performance assessments due to differences in case-mix adjustment. Given the growing role of outcomes in driving prospective payment patient referral and public reporting, performance should be assessed by models with fewer exclusions, superior accuracy, and better case-mix adjustment.
authors.affiliation: 1Cerner Corporation, Vienna, VA. 2Department of Biostatistics, Kansas University Medical Center, Kansas City, MO. 3Critical Care Division, Department of Medicine, Baystate Medical Center, Springfield, MA. 4Tufts University School of Medicine, Boston, MA. 5Department of Anesthesiology and Critical Care Medicine, George Washington University, Washington, DC.
authors.forename: Andrew A
authors.initials: AA
authors.lastname: Kramer
authors.forename: Thomas L
authors.initials: TL
authors.lastname: Higgins
authors.forename: Jack E
authors.initials: JE
authors.lastname: Zimmerman
mesh_terms: D018806:APACHE; D000368:Aged; D019985:Benchmarking; D005260:Female; D017052:Hospital Mortality; D006801:Humans; D007362:Intensive Care Units / Q000706:statistics & numerical data*; D008297:Male; D008875:Middle Aged; D017063:Outcome Assessment, Health Care / Q000379:methods*; D011379:Prognosis; D011787:Quality of Health Care / Q000706:statistics & numerical data; D012121:Respiration, Artificial / Q000706:statistics & numerical data; D012189:Retrospective Studies; D020379:Risk Adjustment