title: Triage tool inter-rater reliability: a comparison of live versus paper case scenarios.
abstract: BACKGROUND
  Published studies of triage scale inter-rater reliability assessment have been conducted mostly using paper case scenarios.
  
  
  OBJECTIVE
  To determine if this method of inter-rater reliability assessment generated significantly different measures from those generated from live triage cases.
  
  
  METHODS
  This was a multicenter, prospective, observational cohort study of a population-based random sample of patients triaged at 2 emergency departments during a period of 4 months. All patients presenting to the emergency department within the study periods were simultaneously and independently triaged using a 5-level triage acuity scale by 2 to 3 research triage nurses blinded to each other's assessment and to the study objective. After 6 months, the same nurses were asked to assign triage scores to paper case scenarios of the same patients that they had each previously triaged.
  
  
  RESULTS
  Each of the 9 research nurses triaged approximately 90 cases. The inter-rater reliabilities as measured by an intraclass correlation coefficient were 0.9 (95% CI = 0.87 to 93) for the live triage assessments and 0.76 (95% CI = 0.73 to 0.79) for the paper case scenarios. The mean triage score assigned to the live cases was significantly less than that assigned to the paper-based cases (3.17; 95% CI = 3.08 to 3.26) (p < 0.001).
  
  
  CONCLUSIONS
  There is moderate to high agreement between live cases and paper case scenarios, and the inter-rater reliabilities, although significantly different, are acceptable in both cases. It is impossible to determine which triage setting provides a more accurate triage score but paper case scenarios generally receive lower triage scores than live cases.
authors.affiliation: Division of Emergency Medicine, McMaster University and Hamilton Health Sciences, Department of Emergency Medicine, Hamilton, Ontario, Canada.
authors.forename: Andrew
authors.initials: A
authors.lastname: Worster
authors.forename: Arlene
authors.initials: A
authors.lastname: Sardo
authors.forename: Kevin
authors.initials: K
authors.lastname: Eva
authors.forename: Christopher M B
authors.initials: CM
authors.lastname: Fernandes
authors.forename: Suneel
authors.initials: S
authors.lastname: Upadhye
mesh_terms: D004636:Emergency Service, Hospital; D006801:Humans; D015403:Nursing Evaluation Research / Q000379:methods*; D015588:Observer Variation; D009864:Ontario; D011446:Prospective Studies; D015203:Reproducibility of Results; D012107:Research Design; D016037:Single-Blind Method; D014218:Triage