title: Interrater agreement statistics with skewed data: evaluation of alternatives to Cohen's kappa.
abstract: OBJECTIVE
  In this study, we aimed to evaluate interrater agreement statistics (IRAS) for use in research on low base rate clinical diagnoses or observed behaviors. Establishing and reporting sufficient interrater agreement is essential in such studies. Yet the most commonly applied agreement statistic, Cohen's κ, has a well known sensitivity to base rates that results in a substantial penalization of interrater agreement when behaviors or diagnoses are very uncommon, a prevalent and frustrating concern in such studies.
  
  
  METHODS
  We performed Monte Carlo simulations to evaluate the performance of 5 of κ's alternatives (Van Eerdewegh's V, Yule's Y, Holley and Guilford's G, Scott's π, and Gwet's AC₁), alongside κ itself. The simulations investigated the robustness of these IRAS to conditions that are common in clinical research, with varying levels of behavior or diagnosis base rate, rater bias, observed interrater agreement, and sample size.
  
  
  RESULTS
  When the base rate was 0.5, each IRAS provided similar estimates, particularly with unbiased raters. G was the least sensitive of the IRAS to base rates.
  
  
  CONCLUSIONS
  The results encourage the use of the G statistic for its consistent performance across the simulation conditions. We recommend separately reporting the rates of agreement on the presence and absence of a behavior or diagnosis alongside G as an index of chance corrected overall agreement.
authors.affiliation: Family Translational Research Group.
authors.forename: Shu
authors.initials: S
authors.lastname: Xu
authors.affiliation: Family Translational Research Group.
authors.forename: Michael F
authors.initials: MF
authors.lastname: Lorber
mesh_terms: D006801:Humans; D015588:Observer Variation; D015203:Reproducibility of Results; D013223:Statistics as Topic