title: Specific agreement on dichotomous outcomes can be calculated for more than two raters.
abstract: OBJECTIVE
  For assessing interrater agreement, the concepts of observed agreement and specific agreement have been proposed. The situation of two raters and dichotomous outcomes has been described, whereas often, multiple raters are involved. We aim to extend it for more than two raters and examine how to calculate agreement estimates and 95% confidence intervals (CIs).
  
  
  METHODS
  As an illustration, we used a reliability study that includes the scores of four plastic surgeons classifying photographs of breasts of 50 women after breast reconstruction into "satisfied" or "not satisfied." In a simulation study, we checked the hypothesized sample size for calculation of 95% CIs.
  
  
  RESULTS
  For m raters, all pairwise tables [ie, m (m - 1)/2] were summed. Then, the discordant cells were averaged before observed and specific agreements were calculated. The total number (N) in the summed table is m (m - 1)/2 times larger than the number of subjects (n), in the example, N = 300 compared to n = 50 subjects times m = 4 raters. A correction of n√(m - 1) was appropriate to find 95% CIs comparable to bootstrapped CIs.
  
  
  CONCLUSIONS
  The concept of observed agreement and specific agreement can be extended to more than two raters with a valid estimation of the 95% CIs.
authors.affiliation: Department of Epidemiology and Biostatistics, EMGO Institute for Health and Care Research, VU Medical Center, De Boelelaan 1089A, Amsterdam 1081HV, The Netherlands. Electronic address: hcw.devet@vumc.nl.
authors.forename: Henrica C W
authors.initials: HCW
authors.lastname: de Vet
authors.affiliation: Department of Epidemiology and Biostatistics, EMGO Institute for Health and Care Research, VU Medical Center, De Boelelaan 1089A, Amsterdam 1081HV, The Netherlands.
authors.forename: Rieky E
authors.initials: RE
authors.lastname: Dikmans
authors.affiliation: Department of Epidemiology and Biostatistics, EMGO Institute for Health and Care Research, VU Medical Center, De Boelelaan 1089A, Amsterdam 1081HV, The Netherlands.
authors.forename: Iris
authors.initials: I
authors.lastname: Eekhout
mesh_terms: D016001:Confidence Intervals; D003627:Data Interpretation, Statistical; D005260:Female; D006801:Humans; D016462:Mammaplasty / Q000592:standards; D015588:Observer Variation; D016896:Treatment Outcome
keywords: Confidence intervals; Continuity correction; Fleiss correction; Observed agreement; Specific agreement