Calculating kappas from adjusted data improved the comparability of the reliability of triage systems: a comparative study.
OBJECTIVE
It is difficult to compare the reliability of triage systems with the kappa statistic. In this article, a method for comparing triage systems was developed and applied to previously conducted triage reliability studies.


METHODS
From simulations with theoretical distributions, the minimum, normal, and maximum weighted kappa for 3- to 5-level triage systems were computed. To compare the reliability of triage systems in previously conducted triage reliability studies, the normal kappa was calculated. Furthermore, the reported quadratically weighted kappas were compared with the minimum, normal, and maximum weighted kappa to characterize the degree and direction of skewness of the data.


RESULTS
The normal kappa was higher in 3-level triage systems (median: κ=0.84) compared with 4-level (median: κ=0.37) and 5-level (median: κ=0.57) systems. In 3-level triage systems, the percentages observed agreement were unequally distributed, which resulted in small quadratically weighted kappas. In 4- and 5-level systems, the percentages observed agreement were more equally distributed compared with 3-level systems, which resulted in higher quadratically weighted kappa values.


CONCLUSIONS
When comparing triage systems with different numbers of categories, one should report both the normal and quadratically weighted kappa. Calculating normal kappas from previously conducted triage reliability studies revealed substantial theoretical differences in interrater reliability of triage systems than previously reported.
van der Wulp
Ineke
I
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands. i.vanderwulp@umcutrecht.nl
van Stel
Henk F
HF
D000465:Algorithms; D003627:Data Interpretation, Statistical; D004636:Emergency Service, Hospital / Q000592:standards* / Q000706:statistics & numerical data; D006801:Humans; D015203:Reproducibility of Results; D012720:Severity of Illness Index; D014218:Triage / Q000592:standards*