title: Unfolding the phenomenon of interrater agreement: a multicomponent approach for in-depth examination was proposed.
abstract: OBJECTIVE
  The overall objective was to unfold the phenomenon of interrater agreement: to identify potential sources of variation in agreement data and to explore how they can be statistically accounted for. The ultimate aim was to propose recommendations for in-depth examination of agreement to improve the reliability of assessment instruments.
  
  
  METHODS
  Using a sample where 10 rater pairs had assessed the presence/absence of 188 environmental barriers by a systematic rating form, a raters × items data set was generated (N=1,880). In addition to common agreement indices, relative shares of agreement variation were calculated. Multilevel regression analysis was carried out, using rater and item characteristics as predictors of agreement variation.
  
  
  RESULTS
  Following a conceptual decomposition, the agreement variation was statistically disentangled into relative shares. The raters accounted for 6-11%, the items for 32-33%, and the residual for 57-60% of the variation. Multilevel regression analysis showed barrier prevalence and raters' familiarity with using standardized instruments to have the strongest impact on agreement.
  
  
  CONCLUSIONS
  Supported by a conceptual analysis, we propose an approach of in-depth examination of agreement variation, as a strategy for increasing the level of interrater agreement. By identifying and limiting the most important sources of disagreement, instrument reliability can be improved ultimately.
authors: Department of Health Sciences, Faculty of Medicine, Lund University, Box 157, SE-221 00 Lund, Sweden. bjorn.slaug@med.lu.se
authors: Björn
authors: B
authors: Slaug
authors: Oliver
authors: O
authors: Schilling
authors: Tina
authors: T
authors: Helle
authors: Susanne
authors: S
authors: Iwarsson
authors: Gunilla
authors: G
authors: Carlsson
authors: Åse
authors: Å
authors: Brandt
mesh_terms: D032921:Consensus; D006801:Humans; D015588:Observer Variation; D012044:Regression Analysis; D015203:Reproducibility of Results; D012106:Research / Q000592:standards*