title: A note on interval estimation of kappa in a series of 2 x 2 tables.
abstract: When there are confounders in reliability studies, failing to stratify data to account for these confounding effects may produce a misleading estimate of the interrater agreement. In this paper, we focus discussion on interval estimation of kappa for measuring agreement between two raters with stratified data. Using Monte Carlo simulation, we compare four asymptotic interval estimators of kappa for stratified data: estimator (1), a weighted average of the stratum-specific kappa estimates with weights equal to the inverse of the estimated asymptotic variances of these estimates; the two estimators, (2) and (3), with use of the logarithmic and the square root transformations, respectively, and a principle similar as used in estimator (1); estimator (4), a weighted average of the stratum-specific kappa estimates with weights equal to stratum sizes. We find that while the coverage probability of the first three interval estimators (1)-(3) can often be less than the desired confidence level, the fourth interval estimator (4) consistently performs well in all the situations considered here. We further find that when the underlying kappa is moderate (0.30</=kappa</=0.50), we can substantially improve the performance of the first three estimators by using a point estimator recently proposed elsewhere for kappa in estimation of weights. Because interval estimator (4) outperforms the other three estimators in a variety of situations, we recommend this estimator for general use.
authors.affiliation: Department of Mathematical and Computer Sciences, College of Sciences, San Diego State University, San Diego, CA 92182-7720, USA.
authors.forename: K J
authors.initials: KJ
authors.lastname: Lui
authors.forename: C
authors.initials: C
authors.lastname: Kelly
mesh_terms: D003198:Computer Simulation; D016001:Confidence Intervals; D006801:Humans; D009010:Monte Carlo Method; D015588:Observer Variation