title: Interrater reliability of a near-miss risk index for incident learning systems in radiation oncology.
abstract: OBJECTIVE
  Tools for assessing the severity and risk of near-miss events in radiation oncology are few and needed. Recent work has described guidelines for the use of a 5-tier near-miss risk index (NMRI) for the classification of near-miss events. The purpose of this study was to assess the reliability of the NMRI among users in a radiation oncology department.
  
  
  METHODS
  Reliability of the NMRI was assessed using an online survey distributed to members of a radiation oncology department. The survey contained 70 events extracted from the department's incident learning system (ILS). Survey participants rated each event using the NMRI guidelines, reported their attendance to weekly ILS meetings (used as a surrogate for familiarity with the ILS), and indicated their familiarity with the radiation oncology workflow. Interrater reliability was determined using Krippendorff's alpha. Use of the NMRI to rate actual events during 5 weekly ILS meetings was also assessed and interrater reliability determined.
  
  
  RESULTS
  Twenty-eight survey respondents represented a wide variety of care providers. Krippendorff's alpha was calculated for the whole respondent cohort to be 0.376, indicating fair agreement among raters. Respondents who had the most participation at ILS meetings (n = 4) had moderate agreement with an alpha of 0.501. Interestingly, there were significant differences in reliability and median NMRI scores between professions. NMRI use during weekly NMRI meetings (80 events rated), participants showed moderate reliability (alpha = 0.607).
  
  
  CONCLUSIONS
  Using the NMRI guidelines, raters from a wide variety of professions were able to assess the severity of near-miss incidents with fair agreement. Those experienced with the ILS showed better agreement, and higher agreement was seen during multidisciplinary ILS meetings. These data support the use the indices such as the NMRI for near-miss risk assessment in patient safety and prioritization of process improvements in radiation oncology.
authors.affiliation: Department of Radiation Oncology, University of Washington, Seattle, Washington. Electronic address: mullent9@uw.edu.
authors.forename: Thomas
authors.initials: T
authors.lastname: Mullen
authors.affiliation: Department of Radiation Oncology, University of Washington, Seattle, Washington.
authors.forename: Matthew
authors.initials: M
authors.lastname: Nyflot
authors.affiliation: Department of Radiation Oncology, University of Washington, Seattle, Washington.
authors.forename: Jing
authors.initials: J
authors.lastname: Zeng
authors.affiliation: Department of Radiation Oncology, University of Washington, Seattle, Washington.
authors.forename: Loucille
authors.initials: L
authors.lastname: Jordan
authors.affiliation: Department of Radiation Oncology, University of Washington, Seattle, Washington.
authors.forename: Patricia A
authors.initials: PA
authors.lastname: Sponseller
authors.affiliation: Department of Radiation Oncology, University of Washington, Seattle, Washington.
authors.forename: Joshua
authors.initials: J
authors.lastname: Carlson
authors.affiliation: Department of Radiation Oncology, University of Washington, Seattle, Washington.
authors.forename: Gabrielle
authors.initials: G
authors.lastname: Kane
authors.affiliation: Department of Radiation Oncology, University of Washington, Seattle, Washington.
authors.forename: Eric C
authors.initials: EC
authors.lastname: Ford
mesh_terms: D006801:Humans; D000067491:Near Miss, Healthcare / Q000706:statistics & numerical data*; D015588:Observer Variation; D018787:Radiation Oncology; D015203:Reproducibility of Results; D018570:Risk Assessment / Q000379:methods*; D011795:Surveys and Questionnaires; D057188:Workflow