Inference procedures for assessing interobserver agreement among multiple raters.
We propose a new procedure for constructing inferences about a measure of interobserver agreement in studies involving a binary outcome and multiple raters. The proposed procedure, based on a chi-square goodness-of-fit test as applied to the correlated binomial model (Bahadur, 1961, in Studies in Item Analysis and Prediction, 158-176), is an extension of the goodness-of-fit procedure developed by Donner and Eliasziw (1992, Statistics in Medicine 11, 1511-1519) for the case of two raters. The new procedure is shown to provide confidence-interval coverage levels that are close to nominal over a wide range of parameter combinations. The procedure also provides a sample-size formula that may be used to determine the required number of subjects and raters for such studies.
Altaye
M
M
Center for Pediatric Research, Children's Hospital of the King's Daughters, Eastern Virginia Medical School, Norfolk 23510, USA. altayem@chkd.com
Donner
A
A
Klar
N
N
D016010:Binomial Distribution; D001699:Biometry / Q000379:methods; D016001:Confidence Intervals; D015353:Eye Enucleation / Q000009:adverse effects; D005134:Eye Neoplasms / Q000473:pathology / Q000601:surgery; D006801:Humans; D015233:Models, Statistical; D009336:Necrosis; D015588:Observer Variation; D015203:Reproducibility of Results