title: The Importance of Integrating Clinical Relevance and Statistical Significance in the Assessment of Quality of Care--Illustrated Using the Swedish Stroke Register.
abstract: BACKGROUND
  When profiling hospital performance, quality inicators are commonly evaluated through hospital-specific adjusted means with confidence intervals. When identifying deviations from a norm, large hospitals can have statistically significant results even for clinically irrelevant deviations while important deviations in small hospitals can remain undiscovered. We have used data from the Swedish Stroke Register (Riksstroke) to illustrate the properties of a benchmarking method that integrates considerations of both clinical relevance and level of statistical significance.
  
  
  METHODS
  The performance measure used was case-mix adjusted risk of death or dependency in activities of daily living within 3 months after stroke. A hospital was labeled as having outlying performance if its case-mix adjusted risk exceeded a benchmark value with a specified statistical confidence level. The benchmark was expressed relative to the population risk and should reflect the clinically relevant deviation that is to be detected. A simulation study based on Riksstroke patient data from 2008-2009 was performed to investigate the effect of the choice of the statistical confidence level and benchmark value on the diagnostic properties of the method.
  
  
  RESULTS
  Simulations were based on 18,309 patients in 76 hospitals. The widely used setting, comparing 95% confidence intervals to the national average, resulted in low sensitivity (0.252) and high specificity (0.991). There were large variations in sensitivity and specificity for different requirements of statistical confidence. Lowering statistical confidence improved sensitivity with a relatively smaller loss of specificity. Variations due to different benchmark values were smaller, especially for sensitivity. This allows the choice of a clinically relevant benchmark to be driven by clinical factors without major concerns about sufficiently reliable evidence.
  
  
  CONCLUSIONS
  The study emphasizes the importance of combining clinical relevance and level of statistical confidence when profiling hospital performance. To guide the decision process a web-based tool that gives ROC-curves for different scenarios is provided.
authors.affiliation: Department of Statistics, Umeå School of Business and Economics, Umeå University, Umeå, Sweden.
authors.forename: Anita
authors.initials: A
authors.lastname: Lindmark
authors.affiliation: Department of Statistics, Umeå School of Business and Economics, Umeå University, Umeå, Sweden.|Department of Applied Mathematics, Computer Science and Statistics, Ghent University, Ghent, Belgium.
authors.forename: Bart
authors.initials: B
authors.lastname: van Rompaye
authors.affiliation: Department of Applied Mathematics, Computer Science and Statistics, Ghent University, Ghent, Belgium.
authors.forename: Els
authors.initials: E
authors.lastname: Goetghebeur
authors.affiliation: Department of Public Health and Clinical Medicine, Umeå University, Umeå, Sweden.
authors.forename: Eva-Lotta
authors.initials: EL
authors.lastname: Glader
authors.affiliation: Department of Statistics, Umeå School of Business and Economics, Umeå University, Umeå, Sweden.
authors.forename: Marie
authors.initials: M
authors.lastname: Eriksson
mesh_terms: D000203:Activities of Daily Living; D000328:Adult; D000368:Aged; D000369:Aged, 80 and over; D019985:Benchmarking / Q000379:methods*; D005260:Female; D006801:Humans; D008297:Male; D008875:Middle Aged; D019984:Quality Indicators, Health Care; D012372:ROC Curve; D012042:Registries; D020521:Stroke / Q000175:diagnosis* / Q000401:mortality; D013548:Sweden; D055815:Young Adult