title: How reliable are chance-corrected measures of agreement?
abstract: Chance-corrected measures of agreement are prone to exhibit paradoxical and counter-intuitive results when used as measures of reliability. It is demonstrated that these problems arise with Cohen's kappa as well as with Aickin's alpha. They are the consequence of an analogy to Simpson's paradox in mixed populations. It is further shown that chance-corrected measures of agreement may yield misleading values for binary ratings. It is concluded that improvements in the design and the analysis of reliability studies are a prerequisite for valid and pertinent results.
authors.affiliation: Department of Biostatistics and Medical Informatics, Free University of Berlin, Germany.
authors.forename: I
authors.initials: I
authors.lastname: Guggenmoos-Holzmann
mesh_terms: D002965:Classification; D003627:Data Interpretation, Statistical; D003933:Diagnosis; D004813:Epidemiology; D006801:Humans; D016013:Likelihood Functions; D016014:Linear Models; D015233:Models, Statistical; D015588:Observer Variation; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity