title: Random variation and rankability of hospitals using outcome indicators.
abstract: OBJECTIVE
  There is a growing focus on quality and safety in healthcare. Outcome indicators are increasingly used to compare hospital performance and to rank hospitals, but the reliability of ranking (rankability) is under debate. This study aims to quantify the rankability of several outcome indicators of hospital performance currently used by the Dutch government.
  
  
  METHODS
  From 52 indicators used by the Netherlands Inspectorate, the authors selected nine outcome indicators presenting a fraction and absolute numbers. Of these indicators, four were combined into two, resulting in seven indicators for analysis. The official data of 97 Dutch hospitals for the year 2007 were used. Uncertainty in the observed outcomes within the hospitals (within hospital variance, σ(2)) was estimated using fixed effect logistic regression models. Heterogeneity (between hospital variance, τ(2)) was measured with random effect logistic regression models. Subsequently, the rankability was calculated by relating heterogeneity to uncertainty within and between hospitals (τ(2)/(τ(2) +median σ(2))).
  
  
  RESULTS
  Sample sizes varied but were typically around 200 per hospital (range of median 90-277) with a median of 2-21 cases, causing a substantial uncertainty in outcomes per hospital. Although fourfold to eightfold differences between hospitals were noted, the uncertainty within hospitals caused a poor (<50%) rankability in three indicators and moderate rankability (50-75%) in the other four indicators.
  
  
  CONCLUSIONS
  The currently used Dutch outcome indicators are not suitable for ranking hospitals. When judging hospital quality the influence of random variation must be accounted for to avoid overinterpretation of the numbers in the quest for more transparency in healthcare. Adequate sample size is a prerequisite in attempting reliable ranking.
authors.affiliation: Department of Public Health, Center for Medical Decision Making, Erasmus MC, University Medical Center Rotterdam, Rotterdam, The Netherland. a.m.vandishoeck@erasmusmc.nl
authors.forename: Anne-Margreet
authors.initials: AM
authors.lastname: van Dishoeck
authors.forename: Hester F
authors.initials: HF
authors.lastname: Lingsma
authors.forename: Johan P
authors.initials: JP
authors.lastname: Mackenbach
authors.forename: Ewout W
authors.initials: EW
authors.lastname: Steyerberg
mesh_terms: D006739:Hospital Administration / Q000706:statistics & numerical data*; D006801:Humans; D016015:Logistic Models; D009426:Netherlands; D017063:Outcome Assessment, Health Care / Q000706:statistics & numerical data*; D019984:Quality Indicators, Health Care / Q000706:statistics & numerical data*; D015203:Reproducibility of Results; D020379:Risk Adjustment; D035501:Uncertainty