title: A comparison of the performance of a model based on administrative data and a model based on clinical data: effect of severity of illness on standardized mortality ratios of intensive care units.
abstract: OBJECTIVE
  It has been postulated that prognostic models based on administrative data can provide valid adjusted mortality rates in specific patient populations. In this study we compared the performance and robustness of a model based on administrative data (customized hospital standardized mortality ratio) and a model based on clinical data (customized Simplified Acute Physiology Score II) in the Dutch intensive care unit population.
  
  
  METHODS
  Cohort study of intensive care unit records from a national intensive care unit quality registry linked to administrative records from the Dutch National Medical Registration. The hospital standardized mortality ratio and Simplified Acute Physiology Score II models were first-level customized on the intensive care unit population.
  
  
  METHODS
  Fifty-five Dutch intensive care units.
  
  
  METHODS
  A total of 66,564 intensive care unit patients admitted from 2005 to 2008.
  
  
  METHODS
  None.
  
  
  RESULTS
  Performance expressed by measures of discrimination, accuracy, and calibration (area under the receiver operating characteristic curve, Brier score, Hosmer-Lemeshow Ĉ-statistic, and calibration plots). Additionally, the robustness of the models was assessed by simulating changes in the population's severity of illness and analyzing the effect on the intensive care units' standardized mortality ratios.The area under the receiver operating characteristic curve and Brier score of the customized Simplified Acute Physiology Score II were significantly superior to that of the customized hospital standardized mortality ratio (0.85 and 0.11 vs. 0.77 and 0.13, respectively). Calibration plots showed good agreement between observed and predicted mortality for low-risk patients in both models, with more discrepancy in the high-risk patients when using the customized hospital standardized mortality ratio. Severity of illness had influence on the intensive care units' standardized mortality ratios in both models, but the customized Simplified Acute Physiology Score II showed more robustness.
  
  
  CONCLUSIONS
  The customized Simplified Acute Physiology Score II outperforms the customized hospital standardized mortality ratio in the Dutch intensive care unit population. Comparing institutions based on standardized mortality ratios can be unfavorable for those with a more severely ill intensive care unit population, especially when using the customized hospital standardized mortality ratio.
authors: Department of Medical Informatics, Academic Medical Center, University of Amsterdam, Amsterdam, The Netherlands. s.brinkman@amc.uva.nl
authors: Sylvia
authors: S
authors: Brinkman
authors: Ameen
authors: A
authors: Abu-Hanna
authors: André
authors: A
authors: van der Veen
authors: Evert
authors: E
authors: de Jonge
authors: Nicolette F
authors: NF
authors: de Keizer
mesh_terms: D003422:Critical Care / Q000592:standards* / Q000706:statistics & numerical data; D016638:Critical Illness / Q000401:mortality* / Q000628:therapy; D016208:Databases, Factual; D005260:Female; D006739:Hospital Administration / Q000592:standards; D017052:Hospital Mortality; D006801:Humans; D007362:Intensive Care Units / Q000592:standards* / Q000706:statistics & numerical data; D016015:Logistic Models; D008297:Male; D016347:Medical Records Systems, Computerized; D017146:Models, Organizational; D015233:Models, Statistical; D009426:Netherlands; D011786:Quality Control; D012372:ROC Curve; D013647:Task Performance and Analysis