title: Reliability assessment of a hospital quality measure based on rates of adverse outcomes on nursing units.
abstract: The purpose of this study was to develop methods for assessing the reliability of scores on a widely disseminated hospital quality measure based on nursing unit fall rates. Poisson regression interactive multilevel modeling was adapted to account for clustering of units within hospitals. Three signal-noise reliability measures were computed. Squared correlations between the hospital score and true hospital fall rate averaged 0.52 ± 0.18 for total falls (0.68 ± 0.18 for injurious falls). Reliabilities on the other two measures averaged at least 0.70 but varied widely across hospitals. Parametric bootstrap data reflecting within-unit noise in falls were generated to evaluate percentile-ranked hospital scores as estimators of true hospital fall rate ranks. Spearman correlations between bootstrap hospital scores and true fall rates averaged 0.81 ± 0.01 (0.79 ± 0.01). Bias was negligible, but ranked hospital scores were imprecise, varying across bootstrap samples with average SD 11.8 (14.9) percentiles. Across bootstrap samples, hospital-measure scores fell in the same decile as the true fall rate in about 30% of cases. Findings underscore the importance of thoroughly assessing reliability of quality measurements before deciding how they will be used. Both the hospital measure and the reliability methods described can be adapted to other contexts involving clustered rates of adverse patient outcomes.
authors.affiliation: 1 Health Services and Outcomes Research, Children's Mercy Hospitals and Clinics, Kansas City, MO, USA.|2 School of Medicine, University of Missouri-Kansas City, Kansas City, MO, USA.
authors.forename: Vincent S
authors.initials: VS
authors.lastname: Staggs
mesh_terms: D000058:Accidental Falls / Q000706:statistics & numerical data; D056808:Biostatistics / Q000379:methods; D006282:Health Personnel / Q000592:standards / Q000706:statistics & numerical data; D006761:Hospitals / Q000592:standards* / Q000706:statistics & numerical data; D006801:Humans; D007297:Inpatients / Q000706:statistics & numerical data; D015233:Models, Statistical; D009732:Nursing Care / Q000592:standards / Q000706:statistics & numerical data; D061214:Patient Safety / Q000706:statistics & numerical data; D011785:Quality Assurance, Health Care / Q000706:statistics & numerical data*; D011787:Quality of Health Care / Q000706:statistics & numerical data; D015203:Reproducibility of Results; D059629:Signal-To-Noise Ratio
keywords: Health care quality; quality measurement; reliability