title: Reducing interrater variability and improving health care: a meta-analytical review.
abstract: OBJECTIVE
  In the scientific literature about reliability, the main approach to increasing reliability seems to involve increasing the number of observers and improving the instrument used. Other aspects for improving reliability - like the training of raters - seem to receive less notice. It is worth asking whether this technical approach could be complemented by training the user of the instrument. A systematic meta-analytical review of the research literature was performed to answer this question and examine the effectiveness of planned interventions for improving interrater reliability of health care professionals.
  
  
  METHODS
  The databases of PubMed (MEDLINE), Embase, Omega and PsycINFO were searched. The inclusion criteria were met by 57 studies. Details extracted from the studies included the study design, the number of observers and the number of observed cases, the intervention, the type of instrument (whether or not it was highly technical), and statistical information about the agreement before and after the intervention. Interventions were categorized into three groups: training of professionals, improving the diagnostic instrument and a combination of training and improving the instrument. A meta-analysis was performed by means of linear regression.
  
  
  RESULTS
  The interventions were arranged according to their effectiveness in improving the diagnostic instrument (mean change: β = 0.13), training combined with improving the instrument (mean change: β = 0.10) and training (mean change: β = 0.09).
  
  
  CONCLUSIONS
  On average, although all types of interventions are effective, improving the diagnostic instrument seems to be the most effective. Especially when highly technical instruments were concerned, improvement proved to be very effective (β = 0.52). Because instrumental variables constitute a major source of error, improving the instrument is an important approach. However, this review offers solid arguments that can complement the literature and practice, with a focus on training the user of the instrument.
authors: Knowledge Centre, Dutch Health Care Inspectorate, Utrecht, the Netherlands. s.tuijn@igz.nl
authors: Saskia
authors: S
authors: Tuijn
authors: Frans
authors: F
authors: Janssens
authors: Paul
authors: P
authors: Robben
authors: Huub
authors: H
authors: van den Bergh
mesh_terms: D006282:Health Personnel; D006801:Humans; D016014:Linear Models; D015588:Observer Variation; D011785:Quality Assurance, Health Care; D012106:Research / Q000592:standards*