title: Assessing agreement with multiple raters on correlated kappa statistics.
abstract: In clinical studies, it is often of interest to see the diagnostic agreement among clinicians on certain symptoms. Previous work has focused on the agreement between two clinicians under two different conditions or the agreement among multiple clinicians under one condition. Few have discussed the agreement study with a design where multiple clinicians examine the same group of patients under two different conditions. In this paper, we use the intraclass kappa statistic for assessing nominal scale agreement with such a design. We derive an explicit variance formula for the difference of correlated kappa statistics and conduct hypothesis testing for the equality of kappa statistics. Simulation studies show that the method performs well with realistic sample sizes and may be superior to a method that did not take into account the measurement dependence structure. The practical utility of the method is illustrated on data from an eosinophilic esophagitis (EoE) study.
authors.affiliation: Department of Statistics, University of Missouri-Columbia, Columbia, MO 65211, USA.
authors.forename: Hongyuan
authors.initials: H
authors.lastname: Cao
authors.affiliation: Department of Biostatistics, University of North Carolina-Chapel Hill, Chapel Hill, NC 27514, USA.
authors.forename: Pranab K
authors.initials: PK
authors.lastname: Sen
authors.affiliation: Department of Medicine, University of North Carolina-Chapel Hill, Chapel Hill, NC 27514, USA.
authors.forename: Anne F
authors.initials: AF
authors.lastname: Peery
authors.affiliation: Department of Medicine, University of North Carolina-Chapel Hill, Chapel Hill, NC 27514, USA.
authors.forename: Evan S
authors.initials: ES
authors.lastname: Dellon
mesh_terms: D001699:Biometry / Q000379:methods*; D003198:Computer Simulation; D019937:Diagnostic Techniques and Procedures / Q000592:standards*; D006801:Humans; D015233:Models, Statistical; D015203:Reproducibility of Results; D012107:Research Design; D018401:Sample Size
keywords: Contingency table; Dependent kappa statistics; Multinomial distribution