title: Measuring inter-observer agreement in contour delineation of medical imaging in a dummy run using Fleiss' kappa.
abstract: BACKGROUND
  In medical imaging used for planning of radiation therapy, observers delineate contours of a treatment volume in a series of images of uniform slice thickness.
  
  
  OBJECTIVE
  To summarize agreement in contouring between an arbitrary number of observers by a single number, we generalized the kappa index proposed by Zijdenbos et al. (1994).
  
  
  METHODS
  Observers characterized voxels by allocating them to one of two categories, inside or outside the contoured region. Fleiss' kappa was used to measure association between n indistinguishable observers. Given the number Vi of voxels contoured by exactly i observers (i = 1, ..., n ), the resulting overall kappa is representable as a ratio of weighted sums of the Vi .
  
  
  RESULTS
  Overall kappa was applied to analyze inter-center variations in a multicenter trial on radiotherapy planning in patients with locally advanced lung cancer. A contouring dummy run was performed within the quality assurance program. Contouring was done twice, once before and once after a training program. Observer agreement was enhanced from 0.59 (with a 95% confidence interval (CI) of 0.51-0.67) to 0.69 (95% CI 0.59-0.78).
  
  
  CONCLUSIONS
  By contrast to average pairwise indices, overall kappa measures observer agreement for more than two observers using the full information about overlapping volumes, while not distinguishing between observers. It is particularly adequate for measuring observer agreement when identification of observers is not possible or desirable and when there is no gold standard.
authors.affiliation: Gerta Rücker, Institute of Medical Biometry and Medical Informatics, University Medical Center Freiburg, Stefan-Meier-Straße 26, 79104 Freiburg, Germany. ruecker@imbi.uni-freiburg.de
authors.forename: G
authors.initials: G
authors.lastname: Rücker
authors.forename: T
authors.initials: T
authors.lastname: Schimek-Jasch
authors.forename: U
authors.initials: U
authors.lastname: Nestle
mesh_terms: D000465:Algorithms; D005556:Form Perception; D006801:Humans; D015337:Multicenter Studies as Topic; D009369:Neoplasms / Q000532:radiotherapy; D015588:Observer Variation; D011446:Prospective Studies; D011880:Radiotherapy Planning, Computer-Assisted / Q000379:methods* / Q000706:statistics & numerical data; D016032:Randomized Controlled Trials as Topic