title: Disagreements in meta-analyses using outcomes measured on continuous or rating scales: observer agreement study.
abstract: OBJECTIVE
  To study the inter-observer variation related to extraction of continuous and numerical rating scale data from trial reports for use in meta-analyses.
  
  
  METHODS
  Observer agreement study.
  
  
  METHODS
  A random sample of 10 Cochrane reviews that presented a result as a standardised mean difference (SMD), the protocols for the reviews and the trial reports (n=45) were retrieved.
  
  
  METHODS
  Five experienced methodologists and five PhD students independently extracted data from the trial reports for calculation of the first SMD result in each review. The observers did not have access to the reviews but to the protocols, where the relevant outcome was highlighted. The agreement was analysed at both trial and meta-analysis level, pairing the observers in all possible ways (45 pairs, yielding 2025 pairs of trials and 450 pairs of meta-analyses). Agreement was defined as SMDs that differed less than 0.1 in their point estimates or confidence intervals.
  
  
  RESULTS
  The agreement was 53% at trial level and 31% at meta-analysis level. Including all pairs, the median disagreement was SMD=0.22 (interquartile range 0.07-0.61). The experts agreed somewhat more than the PhD students at trial level (61% v 46%), but not at meta-analysis level. Important reasons for disagreement were differences in selection of time points, scales, control groups, and type of calculations; whether to include a trial in the meta-analysis; and data extraction errors made by the observers. In 14 out of the 100 SMDs calculated at the meta-analysis level, individual observers reached different conclusions than the originally published review.
  
  
  CONCLUSIONS
  Disagreements were common and often larger than the effect of commonly used treatments. Meta-analyses using SMDs are prone to observer variation and should be interpreted with caution. The reliability of meta-analyses might be improved by having more detailed review protocols, more than one observer, and statistical expertise.
authors.affiliation: Nordic Cochrane Centre, Rigshospitalet, Dept 3343, Blegdamsvej 9, DK-2100 Copenhagen, Denmark. bt@cochrane.dk
authors.forename: Britta
authors.initials: B
authors.lastname: Tendal
authors.forename: Julian P T
authors.initials: JP
authors.lastname: Higgins
authors.forename: Peter
authors.initials: P
authors.lastname: Jüni
authors.forename: Asbjørn
authors.initials: A
authors.lastname: Hróbjartsson
authors.forename: Sven
authors.initials: S
authors.lastname: Trelle
authors.forename: Eveline
authors.initials: E
authors.lastname: Nüesch
authors.forename: Simon
authors.initials: S
authors.lastname: Wandel
authors.forename: Anders W
authors.initials: AW
authors.lastname: Jørgensen
authors.forename: Katarina
authors.initials: K
authors.lastname: Gesser
authors.forename: Søren
authors.initials: S
authors.lastname: Ilsøe-Kristensen
authors.forename: Peter C
authors.initials: PC
authors.lastname: Gøtzsche
mesh_terms: D002986:Clinical Trials as Topic / Q000592:standards / Q000706:statistics & numerical data*; D003625:Data Collection; D015201:Meta-Analysis as Topic; D015588:Observer Variation; D012196:Review Literature as Topic
references.citation: Br J Psychiatry. 2003 Apr;182:303-11
references.pmid: 12668405
references.citation: J Clin Epidemiol. 2006 Jul;59(7):697-703
references.pmid: 16765272
references.citation: PLoS Med. 2009 Jul 21;6(7):e1000097
references.pmid: 19621072
references.citation: Cochrane Database Syst Rev. 2006 Oct 18;(4):CD004510
references.pmid: 17054207
references.citation: Cochrane Database Syst Rev. 2007 Jan 24;(1):CD005160
references.pmid: 17253541
references.citation: Cochrane Database Syst Rev. 2007 Apr 18;(2):CD005333
references.pmid: 17443583
references.citation: Cochrane Database Syst Rev. 2007 Jan 24;(1):CD004716
references.pmid: 17253519
references.citation: Cochrane Database Syst Rev. 2007 Apr 18;(2):CD004585
references.pmid: 17443550
references.citation: Cochrane Database Syst Rev. 2006 Oct 18;(4):CD005179
references.pmid: 17054243
references.citation: Cochrane Database Syst Rev. 2006 Oct 18;(4):CD004350
references.pmid: 17054201
references.citation: Br J Psychiatry. 1998 May;172:420-4
references.pmid: 9747404
references.citation: JAMA. 2004 May 26;291(20):2457-65
references.pmid: 15161896
references.citation: Cochrane Database Syst Rev. 2007 Jan 24;(1):CD005233
references.pmid: 17253549
references.citation: BMJ. 1996 Feb 10;312(7027):345-9
references.pmid: 8611831
references.citation: Br J Psychiatry. 1998 May;172:413-9
references.pmid: 9747403
references.citation: Explore (NY). 2005 Jan;1(1):40-5
references.pmid: 16791967
references.citation: JAMA. 2007 Jul 25;298(4):430-7
references.pmid: 17652297
references.citation: Cochrane Database Syst Rev. 2005 Oct 19;(4):CD003135
references.pmid: 16235315
references.citation: Behav Res Ther. 1998 Oct;36(10):959-70
references.pmid: 9714946
references.citation: Cochrane Database Syst Rev. 2006 Oct 18;(4):CD005473
references.pmid: 17054260
references.citation: Cochrane Database Syst Rev. 2006 Jul 19;(3):CD002870
references.pmid: 16855994
references.citation: J Clin Epidemiol. 2005 Jul;58(7):741-2
references.pmid: 15939227