title: Pitfalls in the use of kappa when interpreting agreement between multiple raters in reliability studies.
abstract: OBJECTIVE
  To compare different reliability coefficients (exact agreement, and variations of the kappa (generalised, Cohen's and Prevalence Adjusted and Biased Adjusted (PABAK))) for four physiotherapists conducting visual assessments of scapulae.
  
  
  METHODS
  Inter-therapist reliability study.
  
  
  METHODS
  Research laboratory.
  
  
  METHODS
  30 individuals with no history of neck or shoulder pain were recruited with no obvious significant postural abnormalities.
  
  
  METHODS
  Ratings of scapular posture were recorded in multiple biomechanical planes under four test conditions (at rest, and while under three isometric conditions) by four physiotherapists.
  
  
  RESULTS
  The magnitude of discrepancy between the two therapist pairs was 0.04 to 0.76 for Cohen's kappa, and 0.00 to 0.86 for PABAK. In comparison, the generalised kappa provided a score between the two paired kappa coefficients. The difference between mean generalised kappa coefficients and mean Cohen's kappa (0.02) and between mean generalised kappa and PABAK (0.02) were negligible, but the magnitude of difference between the generalised kappa and paired kappa within each plane and condition was substantial; 0.02 to 0.57 for Cohen's kappa and 0.02 to 0.63 for PABAK, respectively.
  
  
  CONCLUSIONS
  Calculating coefficients for therapist pairs alone may result in inconsistent findings. In contrast, the generalised kappa provided a coefficient close to the mean of the paired kappa coefficients. These findings support an assertion that generalised kappa may lead to a better representation of reliability between three or more raters and that reliability studies only calculating agreement between two raters should be interpreted with caution. However, generalised kappa may mask more extreme cases of agreement (or disagreement) that paired comparisons may reveal.
authors.affiliation: NHMRC Centre for Clinical Research Excellence in Spinal Pain, Injury and Health, University of Queensland, Brisbane, QLD 4072, Australia; Physiotherapy Department, Royal Brisbane and Women's Hospital, Queensland Health, Herston, Brisbane, QLD 4029, Australia. Electronic address: shaun_oleary@health.qld.gov.au.
authors.forename: Shaun
authors.initials: S
authors.lastname: O'Leary
authors.affiliation: NHMRC Centre for Clinical Research Excellence in Spinal Pain, Injury and Health, University of Queensland, Brisbane, QLD 4072, Australia; Norwegian Sports Medicine Clinic (NIMI), Oslo, Norway. Electronic address: marte.lund@nimi.no.
authors.forename: Marte
authors.initials: M
authors.lastname: Lund
authors.affiliation: NHMRC Centre for Clinical Research Excellence in Spinal Pain, Injury and Health, University of Queensland, Brisbane, QLD 4072, Australia; Medi 3 Clinic, Aalesund, Norway. Electronic address: toreyh@hotmail.com.
authors.forename: Tore Johan
authors.initials: TJ
authors.lastname: Ytre-Hauge
authors.affiliation: NHMRC Centre for Clinical Research Excellence in Spinal Pain, Injury and Health, University of Queensland, Brisbane, QLD 4072, Australia; University Hospital of Northern Norway, Tromsø, Norway. Electronic address: sigrid.r.holm@gmail.com.
authors.forename: Sigrid Reiersen
authors.initials: SR
authors.lastname: Holm
authors.affiliation: NHMRC Centre for Clinical Research Excellence in Spinal Pain, Injury and Health, University of Queensland, Brisbane, QLD 4072, Australia; Hans & Olaf Physiotherapy Clinic, Oslo, Norway. Electronic address: kaja.nass@hof.nhn.no.
authors.forename: Kaja
authors.initials: K
authors.lastname: Naess
authors.affiliation: NHMRC Centre for Clinical Research Excellence in Spinal Pain, Injury and Health, University of Queensland, Brisbane, QLD 4072, Australia; Eggedal Physiotherapy Clinic, Sigdal, Norway. Electronic address: larsdalland@hotmail.com.
authors.forename: Lars Nagelstad
authors.initials: LN
authors.lastname: Dalland
authors.affiliation: Centre for Functioning and Health Research, Queensland Health, Cnr of Ipswich Road and Cornwall Street, Brisbane, Australia; School of Public Health and Institute of Health and Biomedical Innovation, Queensland University of Technology, Victoria Park Road, Brisbane, Australia. Electronic address: steven_mcphail@health.qld.gov.au.
authors.forename: Steven M
authors.initials: SM
authors.lastname: McPhail
mesh_terms: D000328:Adult; D001696:Biomechanical Phenomena; D003627:Data Interpretation, Statistical; D005260:Female; D006801:Humans; D008297:Male; D015588:Observer Variation; D026761:Physical Therapy Specialty / Q000379:methods*; D011187:Posture; D015203:Reproducibility of Results; D012540:Scapula
keywords: Agreement; Inter-therapist; Kappa; Posture; Reliability; Scapular