title: Measuring agreement for ordered ratings in 3 x 3 tables.
abstract: OBJECTIVE
  When two raters consider a qualitative variable ordered according to three categories, the qualitative agreement is commonly assessed with a symmetrically weighted kappa statistic. However, these statistics can present paradoxes, since they may be insensitive to variations of either complete agreements or disagreements.
  
  
  METHODS
  Agreement may be summarized by the relative amounts of complete agreements, partial and maximal disagreements beyond chance. Fixing the marginal totals and the trace, we computed symmetrically weighted kappa statistics and we developed a new statistic for qualitative agreements. Data sets from the literature were used to illustrate the methods.
  
  
  RESULTS
  We show that agreement may be better assessed with the unweighted kappa index, kappa(c), and a new statistic zeta, which assesses the excess of maximal disagreements with respect to the partial ones, and does not depend on a particular weighting system. When zeta is equal to zero, maximal and partial disagreements beyond chance are equal. With its estimated large sample variance, we compared the values of two contingency tables.
  
  
  CONCLUSIONS
  The (kappa(c), zeta) pair is sensitive to variations in agreements and/or disagreements and enables locating the difference between two qualitative agreements. The qualitative agreement is better with increasing values of kappa(c) and zeta.
authors.affiliation: Université Montpellier I, UFR Médecine, Département de l'Information Médicale, Montpellier, France. d-neveu@chu-montpellier.fr
authors.forename: D
authors.initials: D
authors.lastname: Neveu
authors.forename: P
authors.initials: P
authors.lastname: Aubas
authors.forename: F
authors.initials: F
authors.lastname: Seguret
authors.forename: A
authors.initials: A
authors.lastname: Kramar
authors.forename: P
authors.initials: P
authors.lastname: Dujols
mesh_terms: D003627:Data Interpretation, Statistical; D005602:France; D015233:Models, Statistical; D036301:Qualitative Research