title: Exact one-sided confidence limits for Cohen's kappa as a measurement of agreement.
abstract: Cohen's kappa coefficient, κ, is a statistical measure of inter-rater agreement or inter-annotator agreement for qualitative items. In this paper, we focus on interval estimation of κ in the case of two raters and binary items. So far, only asymptotic and bootstrap intervals are available for κ due to its complexity. However, there is no guarantee that such intervals will capture κ with the desired nominal level 1- α. In other words, the statistical inferences based on these intervals are not reliable. We apply the Buehler method to obtain exact confidence intervals based on four widely used asymptotic intervals, three Wald-type confidence intervals and one interval constructed from a profile variance. These exact intervals are compared with regard to coverage probability and length for small to medium sample sizes. The exact intervals based on the Garner interval and the Lee and Tu interval are generally recommended for use in practice due to good performance in both coverage probability and length.
authors.affiliation: 1 Epidemiology and Biostatistics Program, Department of Environmental and Occupational Health, School of Community Health Sciences, University of Nevada Las Vegas, Las Vegas, USA.
authors.forename: Guogen
authors.initials: G
authors.lastname: Shan
authors.affiliation: 2 College of Applied Sciences, Beijing University of Technology, Beijing, PR China.|3 Department of Mathematics and Statistics, Wright State University, Dayton, USA.
authors.forename: Weizhen
authors.initials: W
authors.lastname: Wang
mesh_terms: D056808:Biostatistics / Q000379:methods*; D002986:Clinical Trials as Topic / Q000706:statistics & numerical data; D016001:Confidence Intervals; D006801:Humans; D015233:Models, Statistical; D015588:Observer Variation; D018401:Sample Size
keywords: Buehler method; coverage probability; exact confidence interval; expected length; order