Using hierarchical modeling to measure ICU quality.
OBJECTIVE
To determine whether hierarchical modeling agrees with conventional logistic regression modeling on the identity of ICU quality outliers within a large multi-institutional database.


METHODS
Retrospective database analysis.


METHODS
Subset of the Project IMPACT database consisting of 40435 adult patients admitted to surgical, medical, and mixed surgical-medical ICUs ( n=55) between 1997 and 1999 who met inclusion criteria for SAPS II.


RESULTS
The SAPS II score was customized to this database using conventional logistic regression and using a hierarchical (random coefficients) model. Both models exhibited excellent discrimination ( Cstatistic) and calibration (Hosmer-Lemeshow statistic). The hierarchical and nonhierarchical models had C statistics of.870 and.865, and HL statistics of 3.71 ( p>.88, df=8) and 8.94 ( p>.35, df=8), respectively. Since the random effects component of the hierarchical model accounts for between-hospital variability, only the fixed-effects coefficients were used to calculate the expected mortality rate based on the hierarchical model. The ratio and 95% confidence intervals of the observed to expected mortality rate were calculated using both models for each ICU. ICUs whose observed/expected ratio was either less than 1 or greater than 1, and whose 95% confidence interval did not include 1 were labeled as either high-performance or low-performance outliers, respectively. Analysis using kappa statistic revealed almost perfect agreement between the two models (nonhierarchical vs. hierarchical) on the identity of ICU quality outliers.


CONCLUSIONS
Models obtained by customizing SAPS II using a nonhierarchical and a hierarchical approach exhibit excellent agreement on the identity of ICU quality outliers.
Glance
Laurent G
LG
University of Rochester, School of Medicine and Dentistry, 601 Elmwood Avenue, Rochester, NY, 14642, USA. Laurent_Glance@urmc.rochester.edu.
Dick
Andrew W
AW
University of Rochester, School of Medicine and Dentistry, 601 Elmwood Avenue, Rochester, NY, 14642, USA.
Osler
Turner M
TM
University of Vermont, Medical College, USA.
Mukamel
Dana
D
University of California, Department of Medicine, Irvine 100 Theory, Irvine CA 92697-5800, USA.
D018806:APACHE; D019985:Benchmarking / Q000379:methods*; D003422:Critical Care / Q000592:standards*; D005260:Female; D006801:Humans; D007362:Intensive Care Units / Q000706:statistics & numerical data*; D016015:Logistic Models; D008297:Male; D008875:Middle Aged; D011787:Quality of Health Care