title: Assessing the inter-rater agreement for ordinal data through weighted indexes.
abstract: Assessing the inter-rater agreement between observers, in the case of ordinal variables, is an important issue in both the statistical theory and biomedical applications. Typically, this problem has been dealt with the use of Cohen's weighted kappa, which is a modification of the original kappa statistic, proposed for nominal variables in the case of two observers. Fleiss (1971) put forth a generalization of kappa in the case of multiple observers, but both Cohen's and Fleiss' kappa could have a paradoxical behavior, which may lead to a difficult interpretation of their magnitude. In this paper, a modification of Fleiss' kappa, not affected by paradoxes, is proposed, and subsequently generalized to the case of ordinal variables. Monte Carlo simulations are used both to testing statistical hypotheses and to calculating percentile and bootstrap-t confidence intervals based on this statistic. The normal asymptotic distribution of the proposed statistic is demonstrated. Our results are applied to the classical Holmquist et al.'s (1967) dataset on the classification, by multiple observers, of carcinoma in situ of the uterine cervix. Finally, we generalize the use of s* to a bivariate case.
authors.affiliation: Statistical Section, Department of Economics, Management and Statistics, University of Milan-Bicocca, Milano, Italy.
authors.forename: Donata
authors.initials: D
authors.lastname: Marasini
authors.affiliation: Statistical Section, Department of Economics, Management and Statistics, University of Milan-Bicocca, Milano, Italy.
authors.forename: Piero
authors.initials: P
authors.lastname: Quatto
authors.affiliation: Statistical Section, Department of Economics, Management and Statistics, University of Milan-Bicocca, Milano, Italy enrico.ripamonti@unimib.it.
authors.forename: Enrico
authors.initials: E
authors.lastname: Ripamonti
mesh_terms: D002278:Carcinoma in Situ / Q000145:classification / Q000175:diagnosis / Q000473:pathology; D005260:Female; D006801:Humans; D009010:Monte Carlo Method; D015588:Observer Variation; D015203:Reproducibility of Results; D002583:Uterine Cervical Neoplasms / Q000145:classification / Q000175:diagnosis / Q000473:pathology
keywords: Fleiss’ kappa; inter-rater agreement; multiple observers; ordinal variables; weighted indexes