title: Calculating kappas from adjusted data improved the comparability of the reliability of triage systems: a comparative study.
abstract: OBJECTIVE
  It is difficult to compare the reliability of triage systems with the kappa statistic. In this article, a method for comparing triage systems was developed and applied to previously conducted triage reliability studies.
  
  
  METHODS
  From simulations with theoretical distributions, the minimum, normal, and maximum weighted kappa for 3- to 5-level triage systems were computed. To compare the reliability of triage systems in previously conducted triage reliability studies, the normal kappa was calculated. Furthermore, the reported quadratically weighted kappas were compared with the minimum, normal, and maximum weighted kappa to characterize the degree and direction of skewness of the data.
  
  
  RESULTS
  The normal kappa was higher in 3-level triage systems (median: κ=0.84) compared with 4-level (median: κ=0.37) and 5-level (median: κ=0.57) systems. In 3-level triage systems, the percentages observed agreement were unequally distributed, which resulted in small quadratically weighted kappas. In 4- and 5-level systems, the percentages observed agreement were more equally distributed compared with 3-level systems, which resulted in higher quadratically weighted kappa values.
  
  
  CONCLUSIONS
  When comparing triage systems with different numbers of categories, one should report both the normal and quadratically weighted kappa. Calculating normal kappas from previously conducted triage reliability studies revealed substantial theoretical differences in interrater reliability of triage systems than previously reported.
authors.affiliation: Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands. i.vanderwulp@umcutrecht.nl
authors.forename: Ineke
authors.initials: I
authors.lastname: van der Wulp
authors.forename: Henk F
authors.initials: HF
authors.lastname: van Stel
mesh_terms: D000465:Algorithms; D003627:Data Interpretation, Statistical; D004636:Emergency Service, Hospital / Q000592:standards* / Q000706:statistics & numerical data; D006801:Humans; D015203:Reproducibility of Results; D012720:Severity of Illness Index; D014218:Triage / Q000592:standards*