title: Validating the patient safety indicators in the Veterans Health Administration: do they accurately identify true safety events?
abstract: BACKGROUND
  The Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicators (PSIs) use administrative data to detect potentially preventable in-hospital adverse events. However, few studies have determined how accurately the PSIs identify true safety events.
  
  
  OBJECTIVE
  We examined the criterion validity, specifically the positive predictive value (PPV), of 12 selected PSIs using clinical data abstracted from the Veterans Health Administration (VA) electronic medical record as the gold standard.
  
  
  METHODS
  We identified PSI-flagged cases from 28 representative hospitals by applying the AHRQ PSI software (v.3.1a) to VA fiscal year 2003 to 2007 administrative data. Trained nurse-abstractors used standardized abstraction tools to review a random sample of flagged medical records (112 records per PSI) for the presence of true adverse events. Interrater reliability was assessed. We evaluated PPVs and associated 95% confidence intervals of each PSI and examined false positive (FP) cases to determine why they were incorrectly flagged and gain insight into how each PSI might be improved.
  
  
  RESULTS
  PPVs ranged from 28% (95% CI, 15%-43%) for Postoperative Hip Fracture to 87% (95% CI, 79%-92%) for Postoperative Wound Dehiscence. Common reasons for FPs included conditions that were present on admission (POA), coding errors, and lack of coding specificity. PSIs with the lowest PPVs had the highest proportion of FPs owing to POA.
  
  
  CONCLUSIONS
  Overall, PPVs were moderate for most of the PSIs. Implementing POA codes and using more specific ICD-9-CM codes would improve their validity. Our results suggest that additional coding improvements are needed before the PSIs evaluated herein are used for hospital reporting or pay for performance.
authors: Center for Organization, Leadership and Management Research (COLMR), VA Boston Healthcare System, Boston, MA 02130, USA. akrosen@bu.edu or amy.rosen2@va
authors: Amy K
authors: AK
authors: Rosen
authors: Kamal M F
authors: KM
authors: Itani
authors: Marisa
authors: M
authors: Cevasco
authors: Haytham M A
authors: HM
authors: Kaafarani
authors: Amresh
authors: A
authors: Hanchate
authors: Marlena
authors: M
authors: Shin
authors: Michael
authors: M
authors: Shwartz
authors: Susan
authors: S
authors: Loveland
authors: Qi
authors: Q
authors: Chen
authors: Ann
authors: A
authors: Borzecki
mesh_terms: D003430:Cross-Sectional Studies; D006801:Humans; D015588:Observer Variation; D061214:Patient Safety / Q000706:statistics & numerical data*; D019984:Quality Indicators, Health Care / Q000706:statistics & numerical data*; D012189:Retrospective Studies; D012959:Socioeconomic Factors; D014481:United States; D016270:United States Agency for Healthcare Research and Quality; D014493:United States Department of Veterans Affairs