An empirical comparison of key statistical attributes among potential ICU quality indicators.
OBJECTIVE
Good quality indicators should have face validity, relevance to patients, and be able to be measured reliably. Beyond these general requirements, good quality indicators should also have certain statistical properties, including sufficient variability to identify poor performers, relative insensitivity to severity adjustment, and the ability to capture what providers do rather than patients' characteristics. We assessed the performance of candidate indicators of ICU quality on these criteria. Indicators included ICU readmission, mortality, several length of stay outcomes, and the processes of venous-thromboembolism and stress ulcer prophylaxis provision.


METHODS
Retrospective cohort study.


METHODS
One hundred thirty-eight U.S. ICUs from 2001-2008 in the Project IMPACT database.


METHODS
Two hundred sixty-eight thousand eight hundred twenty-four patients discharged from U.S. ICUs.


METHODS
None.


RESULTS
We assessed indicators' (1) variability across ICU-years; (2) degree of influence by patient vs. ICU and hospital characteristics using the Omega statistic; (3) sensitivity to severity adjustment by comparing the area under the receiver operating characteristic curve (AUC) between models including vs. excluding patient variables, and (4) correlation between risk adjusted quality indicators using a Spearman correlation. Large ranges of among-ICU variability were noted for all quality indicators, particularly for prolonged length of stay (4.7-71.3%) and the proportion of patients discharged home (30.6-82.0%), and ICU and hospital characteristics outweighed patient characteristics for stress ulcer prophylaxis (ω, 0.43; 95% CI, 0.34-0.54), venous thromboembolism prophylaxis (ω, 0.57; 95% CI, 0.53-0.61), and ICU readmissions (ω, 0.69; 95% CI, 0.52-0.90). Mortality measures were the most sensitive to severity adjustment (area under the receiver operating characteristic curve % difference, 29.6%); process measures were the least sensitive (area under the receiver operating characteristic curve % differences: venous thromboembolism prophylaxis, 3.4%; stress ulcer prophylaxis, 2.1%). None of the 10 indicators was clearly and consistently correlated with a majority of the other nine indicators.


CONCLUSIONS
No indicator performed optimally across assessments. Future research should seek to define and operationalize quality in a way that is relevant to both patients and providers.
1Center for Clinical Epidemiology and Biostatistics, University of Pennsylvania Perelman School of Medicine, Philadelphia, PA. 2Division of Pulmonary, Allergy, and Critical Care Medicine, University of Pennsylvania Perelman School of Medicine, Philadelphia, PA. 3Leonard Davis Institute of Health Economics, University of Pennsylvania Perelman School of Medicine, Philadelphia, PA.
Sydney E S
SE
Brown
Sarah J
SJ
Ratcliffe
Scott D
SD
Halpern
D000328:Adult; D000368:Aged; D000369:Aged, 80 and over; D015331:Cohort Studies; D005260:Female; D017052:Hospital Mortality; D006801:Humans; D007362:Intensive Care Units / Q000592:standards* / Q000706:statistics & numerical data*; D007902:Length of Stay / Q000706:statistics & numerical data*; D008297:Male; D008875:Middle Aged; D015233:Models, Statistical; D010359:Patient Readmission / Q000706:statistics & numerical data*; D010437:Peptic Ulcer / Q000628:therapy; D019984:Quality Indicators, Health Care / Q000706:statistics & numerical data*; D012189:Retrospective Studies; D014481:United States; D054556:Venous Thromboembolism / Q000628:therapy