title: A comparative assessment of adverse event classification in the out-of-hospital setting.
abstract: OBJECTIVE
  We sought to test reliability of two approaches to classify adverse events (AEs) associated with helicopter EMS (HEMS) transport.
  
  
  METHODS
  The first approach for AE classification involved flight nurses and paramedics (RN/Medics) and mid-career emergency physicians (MC-EMPs) independently reviewing 50 randomly selected HEMS medical records. The second approach involved RN/Medics and MC-EMPs meeting as a group to openly discuss 20 additional medical records and reach consensus-based AE decision. We compared all AE decisions to a reference criterion based on the decision of three senior emergency physicians (Sr-EMPs). We designed a study to detect an improvement in agreement (reliability) from fair (kappa = 0.2) to moderate (kappa = 0.5). We calculated sensitivity, specificity, percent agreement, and positive and negative predictive values (PPV/NPV).
  
  
  RESULTS
  For the independent reviews, the Sr-EMP group identified 26 AEs while individual clinician reviewers identified between 19 and 50 AEs. Agreement on the presence/absence of an AE between Sr-EMPs and three MC-EMPs ranged from κ = 0.20 to κ = 0.25. Agreement between Sr-EMPs and three RN/Medics ranged from κ = 0.11 to κ = 0.19. For the consensus/open-discussion approach, the Sr-EMPs identified 13 AEs, the MC-EMP group identified 18 AEs, and RN/medic group identified 36 AEs. Agreement between Sr-EMPs and MC-EMP group was (κ = 0.30 95%CI -0.12, 0.72), whereas agreement between Sr-EMPs and RN/medic group was (κ = 0.40 95%CI 0.01, 0.79). Agreement between all three groups was fair (κ = 0.33, 95%CI 0.06, 0.66). Percent agreement (58-68%) and NPV (63-76%) was moderately dissimilar between clinicians, while sensitivity (25-80%), specificity (43-97%), and PPV (48-83%) varied.
  
  
  CONCLUSIONS
  We identified a higher level of agreement/reliability in AE decisions utilizing a consensus-based approach for review rather than independent reviews.
authors.forename: P Daniel
authors.initials: PD
authors.lastname: Patterson
authors.forename: Judith R
authors.initials: JR
authors.lastname: Lave
authors.forename: Matthew D
authors.initials: MD
authors.lastname: Weaver
authors.forename: Francis X
authors.initials: FX
authors.lastname: Guyette
authors.forename: Robert M
authors.initials: RM
authors.lastname: Arnold
authors.forename: Christian
authors.initials: C
authors.lastname: Martin-Gill
authors.forename: Jon C
authors.initials: JC
authors.lastname: Rittenberger
authors.forename: David
authors.initials: D
authors.lastname: Krackhardt
authors.forename: Vincent N
authors.initials: VN
authors.lastname: Mosesso
authors.forename: Ronald N
authors.initials: RN
authors.lastname: Roth
authors.forename: Richard J
authors.initials: RJ
authors.lastname: Wadas
authors.forename: Donald M
authors.initials: DM
authors.lastname: Yealy
mesh_terms: D017732:Air Ambulances / Q000706:statistics & numerical data*; D032921:Consensus; D004632:Emergency Medical Services / Q000706:statistics & numerical data*; D006801:Humans; D019300:Medical Errors / Q000706:statistics & numerical data*; D011237:Predictive Value of Tests; D019984:Quality Indicators, Health Care; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D014481:United States
keywords: adverse events; measurement; reliability; safety