title: Statistical significance criteria for the rWG and average deviation interrater agreement indices.
abstract: Despite the widespread use of interrater agreement statistics for multilevel modeling and other types of research, the existing guidelines for inferring the statistical significance of interrater agreement are quite limited. They are largely relevant only under conditions that numerous researchers have argued rarely exist. Here we address this problem by generating guidelines for inferring statistical significance under a number of conditions via a computer simulation. As a set, these guidelines cover many of the conditions researchers commonly face. We discuss how researchers can use the guidelines presented to more reasonably infer the statistical significance of interrater agreement relative to using the limited guidelines available in the extant literature.
authors.affiliation: David Eccles School of Business, University of Utah.
authors.forename: Kristin
authors.initials: K
authors.lastname: Smith-Crowe
authors.affiliation: Freeman School of Business, Tulane University.
authors.forename: Michael J
authors.initials: MJ
authors.lastname: Burke
authors.affiliation: Davidson Faculty of Industrial Engineering and Management, Technion-Israel Institute of Technology.
authors.forename: Ayala
authors.initials: A
authors.lastname: Cohen
authors.affiliation: Davidson Faculty of Industrial Engineering and Management, Technion-Israel Institute of Technology.
authors.forename: Etti
authors.initials: E
authors.lastname: Doveh
mesh_terms: D003198:Computer Simulation; D003627:Data Interpretation, Statistical; D006801:Humans; D011585:Psychology, Applied / Q000379:methods*; D016008:Statistical Distributions