title: Effect of changes over time in the performance of a customized SAPS-II model on the quality of care assessment.
abstract: OBJECTIVE
  The aim of our study was to explore, using an innovative method, the effect of temporal changes in the mortality prediction performance of an existing model on the quality of care assessment. The prognostic model (rSAPS-II) was a recalibrated Simplified Acute Physiology Score-II model developed for very elderly Intensive Care Unit (ICU) patients.
  
  
  METHODS
  The study population comprised all 12,143 consecutive patients aged 80 years and older admitted between January 2004 and July 2009 to one of the ICUs of 21 Dutch hospitals. The prospective dataset was split into 30 equally sized consecutive subsets. Per subset, we measured the model's discrimination [area under the curve (AUC)], accuracy (Brier score), and standardized mortality ratio (SMR), both without and after repeated recalibration. All performance measures were considered to be stable if <2 consecutive points fell outside the green zone [mean ± 2 standard deviation (SD)] and none fell outside the yellow zone (mean ± 4SD) of pre-control charts. We compared proportions of hospitals with SMR>1 without and after repeated recalibration for the year 2009.
  
  
  RESULTS
  For all subsets, the AUCs were stable, but the Brier scores and SMRs were not. The SMR was downtrending, achieving levels significantly below 1. Repeated recalibration rendered it stable again. The proportions of hospitals with SMR>1 and SMR<1 changed from 15 versus 85% to 35 versus 65%.
  
  
  CONCLUSIONS
  Variability over time may markedly vary among different performance measures, and infrequent model recalibration can result in improper assessment of the quality of care in many hospitals. We stress the importance of the timely recalibration and repeated validation of prognostic models over time.
authors: Department of Medical Informatics, Academic Medical Center Amsterdam, Room J1b-124, PO Box 22660, 1100 DD Amsterdam, The Netherlands. L.Minne@amc.uva.nl
authors: Lilian
authors: L
authors: Minne
authors: Saeid
authors: S
authors: Eslami
authors: Nicolette
authors: N
authors: de Keizer
authors: Evert
authors: E
authors: de Jonge
authors: Sophia E
authors: SE
authors: de Rooij
authors: Ameen
authors: A
authors: Abu-Hanna
mesh_terms: D000369:Aged, 80 and over; D015331:Cohort Studies; D005260:Female; D005544:Forecasting; D017052:Hospital Mortality; D006801:Humans; D007362:Intensive Care Units; D008297:Male; D008962:Models, Theoretical; D009426:Netherlands; D011379:Prognosis; D011446:Prospective Studies; D011785:Quality Assurance, Health Care / Q000379:methods*; D012720:Severity of Illness Index; D013997:Time Factors