Adjusting weighted kappa for severity of mistriage decreases reported reliability of emergency department triage systems: a comparative study.
OBJECTIVE
Mistriage can have serious consequences for patients with urgent complaints. We therefore reviewed the assessment of triage-reliability, and propose an alternative weighted kappa that accounts for severity of mistriage.


METHODS
A systematic literature search was performed in three online databases and a search engine. An alternative kappa weighting scheme was developed (triage-weighted kappa), and kappas of previous conducted reliability studies were recalculated.


RESULTS
Kappa is the most frequently used statistic in triage-reliability studies (n=33). More than half of the studies did not report the type of kappa that was used. Linear and quadratically weighted kappa values do not reflect the seriousness of mistriage. Several studies reported almost perfect agreement, whereas percentages of mistriage ranged between 11.1% and 43.4%. In all studies, triage-weighted kappa was lower than reported kappas with a mean difference of 0.17 (range: 0.04-0.32).


CONCLUSIONS
No existing studies on reliability of triage systems account for mistriage. Using triage-weighted kappa, which reflects severity of mistriage, shows that the reliability of triage systems is lower than reported.
van der Wulp
Ineke
I
Department of Public Health, Julius Center for Health Sciences and Primary Care, UMC Utrecht, Utrecht, The Netherlands. I.vanderwulp@umcutrecht.nl
van Stel
Henk F
HF
D000465:Algorithms; D003627:Data Interpretation, Statistical; D004636:Emergency Service, Hospital / Q000592:standards*; D006801:Humans; D015588:Observer Variation; D015203:Reproducibility of Results; D015599:Trauma Severity Indices; D014218:Triage / Q000592:standards*