title: Concordance and robustness of quality indicator sets for hospitals: an analysis of routine data.
abstract: BACKGROUND
  Hospitals are increasingly being evaluated with respect to the quality of provided care. In this setting, several indicator sets compete with one another for the assessment of effectiveness and safety. However, there have been few comparative investigations covering different sets. The objective of this study was to answer three questions: How concordant are different indicator sets on a hospital level? What is the effect of applying different reference values? How stable are the positions of a hospital ranking?
  
  
  METHODS
  Routine data were made available to three companies offering the Patient Safety Indicators, an indicator set from the HELIOS Hospital Group, and measurements based on Disease Staging™. Ten hospitals from North Rhine-Westphalia, comprising a total of 151,960 inpatients in 2006, volunteered to participate in this study. The companies provided standard quality reports for the ten hospitals. Composite measures were defined for strengths and weaknesses. In addition to the different indicator sets, different reference values for one set allowed the construction of several comparison groups. Concordance and robustness were analyzed using the non-parametric correlation coefficient and Kendall's W.
  
  
  RESULTS
  Indicator sets differing only in the reference values of the indicators showed significant correlations in most of the pairs with respect to weaknesses (maximum r = 0.927, CI 0.714-0.983, p < 0.001). There were also significant correlations between different sets (maximum r = 0.829, CI 0.417-0.958, p = 0.003) having different indicators or when different methods for performance assessment were applied. The results were weaker measuring hospital strengths (maximum r = 0.669, CI 0.068-0.914, p = 0.034). In a hospital ranking, only two hospitals belonged consistently either to the superior or to the inferior half of the group. Even altering reference values or the supplier for the same indicator set changed the rank for nine out of ten hospitals.
  
  
  CONCLUSIONS
  Our results reveal an unsettling lack of concordance in estimates of hospital performance when different quality indicator sets are used. These findings underline the lack of consensus regarding optimal validated measures for judging hospital quality. The indicator sets shared a common definition of quality, independent of their focus on patient safety, mortality, or length of stay. However, for most of the hospitals, changing the indicator set or the reference value resulted in a shift from the superior to the inferior half of the group or vice versa. Thus, while taken together the indicator sets offer the hospitals complementary pictures of their quality, on an individual basis they do not establish a reliable ranking.
authors: Ludwig-Maximilians-Universität München, Institut für Medizinische Informationsverarbeitung, Biometrie und Epidemiologie, München, Germany. stausberg@ekmed.de
authors: Jürgen
authors: J
authors: Stausberg
authors: Axel
authors: A
authors: Halim
authors: Robert
authors: R
authors: Färber
references: Int J Qual Health Care. 2009 Apr;21(2):119-29
references: 19155288
references: CMAJ. 2008 Jul 15;179(2):153-7
references: 18625987
references: Int J Qual Health Care. 2007 Apr;19(2):90-104
references: 17277010
references: J Hosp Med. 2007 May;2(3):119-22
references: 17549774
references: J Hosp Med. 2007 May;2(3):128-34
references: 17549759
references: Med Care. 2005 Dec;43(12):1177-84
references: 16299428
references: Ann Intern Med. 2008 Jan 15;148(2):111-23
references: 18195336
references: JAMA. 1984 Feb 3;251(5):637-44
references: 6418903
references: Health Aff (Millwood). 2009 Mar-Apr;28(2):w294-304
references: 19174391
references: Gesundheitswesen. 2007 May;69(5):277-83
references: 17582544
references: Br Med J (Clin Res Ed). 1988 Apr 30;296(6631):1238-42
references: 3248128
references: Med Klin (Munich). 2007 Jul 15;102(7):507-14
references: 17634868
references: Am J Med Qual. 2009 Nov-Dec;24(6):465-73
references: 19696232