Assessing the reliability of standardized performance indicators.
OBJECTIVE
. To investigate the reliability of self-reported standardized performance indicators introduced by the Joint Commission on Accreditation of Healthcare Organizations in July 2002 and implemented in approximately 3400 accredited US hospitals. The study sought to identify the most common data quality problems and determine causes and possible strategies for resolution.


METHODS
Data were independently reabstracted from a random sample of 30 hospitals. Reabstracted data were compared with data originally abstracted, and discrepancies were adjudicated with hospital staff. Structured interviews were used to probe possible reasons for abstraction discrepancies.


RESULTS
The mean data element agreement rate for the 61 data elements evaluated was 91.9%, and the mean kappa statistic for binary data elements was 0.68. The rate of agreement for individual data elements ranged from 100 to 62.4%. The mean difference between calculated indicator rates was 4.88% (absolute value) and the range of differences was 0.0-13.3%. Symmetry of disagreement among original abstractors and reabstractors identified eight indicators whose differences in calculated rates were statistically unlikely to have occurred through random chance (P < 0.05).


CONCLUSIONS
Although improvement in the accuracy and completeness of the self-reported data is possible and desirable, the baseline level of data reliability appears to be acceptable for indicators used to assess and improve hospital performance on selected clinical topics.
Williams
Scott C
SC
Division of Research, Joint Commission on Accreditation of Healthcare Organizations, Oakbrook Terrace, IL 60181, USA. swilliams@jcaho.org
Watt
Ann
A
Schmaltz
Stephen P
SP
Koss
Richard G
RG
Loeb
Jerod M
JM
D006761:Hospitals / Q000592:standards*; D007407:Interviews as Topic; D007591:Joint Commission on Accreditation of Healthcare Organizations; D019984:Quality Indicators, Health Care / Q000592:standards*; D012650:Self Disclosure; D014481:United States