Triage tool inter-rater reliability: a comparison of live versus paper case scenarios.
BACKGROUND
Published studies of triage scale inter-rater reliability assessment have been conducted mostly using paper case scenarios.


OBJECTIVE
To determine if this method of inter-rater reliability assessment generated significantly different measures from those generated from live triage cases.


METHODS
This was a multicenter, prospective, observational cohort study of a population-based random sample of patients triaged at 2 emergency departments during a period of 4 months. All patients presenting to the emergency department within the study periods were simultaneously and independently triaged using a 5-level triage acuity scale by 2 to 3 research triage nurses blinded to each other's assessment and to the study objective. After 6 months, the same nurses were asked to assign triage scores to paper case scenarios of the same patients that they had each previously triaged.


RESULTS
Each of the 9 research nurses triaged approximately 90 cases. The inter-rater reliabilities as measured by an intraclass correlation coefficient were 0.9 (95% CI = 0.87 to 93) for the live triage assessments and 0.76 (95% CI = 0.73 to 0.79) for the paper case scenarios. The mean triage score assigned to the live cases was significantly less than that assigned to the paper-based cases (3.17; 95% CI = 3.08 to 3.26) (p < 0.001).


CONCLUSIONS
There is moderate to high agreement between live cases and paper case scenarios, and the inter-rater reliabilities, although significantly different, are acceptable in both cases. It is impossible to determine which triage setting provides a more accurate triage score but paper case scenarios generally receive lower triage scores than live cases.
Worster
Andrew
A
Division of Emergency Medicine, McMaster University and Hamilton Health Sciences, Department of Emergency Medicine, Hamilton, Ontario, Canada.
Sardo
Arlene
A
Eva
Kevin
K
Fernandes
Christopher M B
CM
Upadhye
Suneel
S
D004636:Emergency Service, Hospital; D006801:Humans; D015403:Nursing Evaluation Research / Q000379:methods*; D015588:Observer Variation; D009864:Ontario; D011446:Prospective Studies; D015203:Reproducibility of Results; D012107:Research Design; D016037:Single-Blind Method; D014218:Triage