title: Navigating a ship with a broken compass: evaluating standard algorithms to measure patient safety.
abstract: Agency for Healthcare Research and Quality (AHRQ) software applies standardized algorithms to hospital administrative data to identify patient safety indicators (PSIs). The objective of this study was to assess the validity of PSI flags and report reasons for invalid flagging.
  
  
  
  At a 6-hospital academic medical center, a retrospective analysis was conducted of all PSIs flagged in fiscal year 2014. A multidisciplinary PSI Quality Team reviewed each flagged PSI based on quarterly reports. The positive predictive value (PPV, the percent of clinically validated cases) was calculated for 12 PSI categories. The documentation for each reversed case was reviewed to determine the reasons for PSI reversal.
  
  
  
  Of 657 PSI flags, 185 were reversed. Seven PSI categories had a PPV below 75%. Four broad categories of reasons for reversal were AHRQ algorithm limitations (38%), coding misinterpretations (45%), present upon admission (10%), and documentation insufficiency (7%). AHRQ algorithm limitations included 2 subcategories: an "incident" was inherent to the procedure, or highly likely (eg, vascular tumor bleed), or an "incident" was nonsignificant, easily controlled, and/or no intervention was needed.
  
  
  
  These findings support previous research highlighting administrative data problems. Additionally, AHRQ algorithm limitations was an emergent category not considered in previous research. Herein we present potential solutions to address these issues.
  
  
  
  If, despite poor validity, US policy continues to rely on PSIs for incentive and penalty programs, improvements are needed in the quality of administrative data and the standardized PSI algorithms. These solutions require national motivation, research attention, and dissemination support.
authors: Department of Family Medicine, College of Medicine, The Ohio State University, Columbus, Ohio, USA.
authors: Jennifer L
authors: JL
authors: Hefner
authors: Department of Family Medicine, College of Medicine, The Ohio State University, Columbus, Ohio, USA.|Department of Biomedical Informatics, College of Medicine, The Ohio State University, Columbus, Ohio, USA.
authors: Timothy R
authors: TR
authors: Huerta
authors: Department of Family Medicine, College of Medicine, The Ohio State University, Columbus, Ohio, USA.|Division of Health Services Management and Policy, College of Public Health, The Ohio State University, Columbus, Ohio, USA.
authors: Ann Scheck
authors: AS
authors: McAlearney
authors: Department of Family Medicine, College of Medicine, The Ohio State University, Columbus, Ohio, USA.
authors: Barbara
authors: B
authors: Barash
authors: Quality and Operations, The Ohio State University Wexner Medical Center, Columbus, Ohio, USA.
authors: Tina
authors: T
authors: Latimer
authors: Quality and Operations, The Ohio State University Wexner Medical Center, Columbus, Ohio, USA.|Department of Surgery, The Ohio State University Wexner Medical Center, Columbus, Ohio, USA.
authors: Susan D
authors: SD
authors: Moffatt-Bruce
mesh_terms: D000046:Academic Medical Centers; D000465:Algorithms; D006801:Humans; D019300:Medical Errors; D061214:Patient Safety; D019984:Quality Indicators, Health Care; D012189:Retrospective Studies; D017751:Safety Management; D012984:Software; D014481:United States; D016270:United States Agency for Healthcare Research and Quality
keywords: administrative data; patient safety indicators; quality measurement