title: Exact approaches for testing hypotheses based on the intra-class kappa coefficient.
abstract: Testing involving the intra-class kappa coefficient is commonly performed in order to assess agreement involving categorical ratings. A number of procedures have been proposed, which make use of the limiting null distribution as the sample size goes to infinity in order to compute the observed significance. As with many tests based on asymptotic null distributions, these tests are associated with problematic type I error control for selected sample sizes and points in the parameter space. We propose and study a collection of exact testing approaches for both the one-sample and K-sample scenarios. For the one-sample case, p-values are obtained using the exact distribution of the test statistic conditional on a sufficient statistic. In addition, unconditional approaches are considered on the basis of maximization across the nuisance parameter space. Numerical evaluation reveals advantages with the exact unconditional procedures.
authors.affiliation: Department of Biostatistics, The State University of New York at Buffalo, Buffalo, NY, USA.
authors.forename: Gregory E
authors.initials: GE
authors.lastname: Wilding
authors.forename: Joseph D
authors.initials: JD
authors.lastname: Consiglio
authors.forename: Guogen
authors.initials: G
authors.lastname: Shan
mesh_terms: D003627:Data Interpretation, Statistical; D004327:Drinking Behavior; D006801:Humans; D008297:Male; D015588:Observer Variation; D015203:Reproducibility of Results; D018401:Sample Size; D014429:Twins, Dizygotic / Q000523:psychology; D014430:Twins, Monozygotic / Q000523:psychology
keywords: exact testing; intra-class kappa; nuisance parameters; unconditional tests