Accuracy of hospital standardized mortality rates: effects of model calibration.
BACKGROUND
Risk-adjusted mortality rates are commonly used in quality report cards to compare hospital performance. The risk adjustment depends on models that are assessed for goodness-of-fit using various discrimination and calibration measures. However, the relationship between model fit and the accuracy of hospital comparisons is not well characterized.


OBJECTIVE
To evaluate the impact of imperfect model calibration (miscalibration) on the accuracy of hospital comparisons.


METHODS
We constructed Monte Carlo simulations where a risk-adjustment model is used in a population with a different mortality distribution than in the original model. We estimated the power of calibration metrics to detect miscalibration. We estimated the sensitivity and specificity of a hospital comparisons method under different imperfect model calibration scenarios using an empirical method.


RESULTS
The U-statistics showed the highest power to detect intercept and slope deviations in the calibration curve, followed by the Hosmer-Lemeshow, and the calibration intercept and slope tests. The specificity decreased with increased intercept and slope deviations and with hospital size. The effect of an imperfect model fit on sensitivity is a function of the true standardized mortality ratio, the underlying mortality rate, sample size, and observed intercept and slope deviations. Poorly performing hospitals can appear as good performers and vice versa, depending on the deviation magnitude and direction.


CONCLUSIONS
Deviations from perfect model calibration have a direct impact on the accuracy of hospital comparisons. Publishing the calibration intercept and slope of risk-adjustment models would allow the users to monitor their performance against the true standard population.
*Division of Research †Kaiser Foundation Health Plan, Management, Information and Analysis ‡Hospital Operations Research, Kaiser Permanente Northern California, Oakland, CA.
Patricia
P
Kipnis
Vincent
V
Liu
Gabriel J
GJ
Escobar
D002138:Calibration / Q000592:standards; D017052:Hospital Mortality; D006761:Hospitals / Q000592:standards / Q000706:statistics & numerical data; D006801:Humans; D015233:Models, Statistical; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity