title: The GRADE approach is reproducible in assessing the quality of evidence of quantitative evidence syntheses.
abstract: OBJECTIVE
  We evaluated the inter-rater reliability (IRR) of assessing the quality of evidence (QoE) using the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) approach.
  
  
  METHODS
  On completing two training exercises, participants worked independently as individual raters to assess the QoE of 16 outcomes. After recording their initial impression using a global rating, raters graded the QoE following the GRADE approach. Subsequently, randomly paired raters submitted a consensus rating.
  
  
  RESULTS
  The IRR without using the GRADE approach for two individual raters was 0.31 (95% confidence interval [95% CI] = 0.21-0.42) among Health Research Methodology students (n = 10) and 0.27 (95% CI = 0.19-0.37) among the GRADE working group members (n = 15). The corresponding IRR of the GRADE approach in assessing the QoE was significantly higher, that is, 0.66 (95% CI = 0.56-0.75) and 0.72 (95% CI = 0.61-0.79), respectively. The IRR further increased for three (0.80 [95% CI = 0.73-0.86] and 0.74 [95% CI = 0.65-0.81]) or four raters (0.84 [95% CI = 0.78-0.89] and 0.79 [95% CI = 0.71-0.85]). The IRR did not improve when QoE was assessed through a consensus rating.
  
  
  CONCLUSIONS
  Our findings suggest that trained individuals using the GRADE approach improves reliability in comparison to intuitive judgments about the QoE and that two individual raters can reliably assess the QoE using the GRADE system.
authors.affiliation: Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario L8S 4K1, Canada.
authors.forename: Reem A
authors.initials: RA
authors.lastname: Mustafa
authors.forename: Nancy
authors.initials: N
authors.lastname: Santesso
authors.forename: Jan
authors.initials: J
authors.lastname: Brozek
authors.forename: Elie A
authors.initials: EA
authors.lastname: Akl
authors.forename: Stephen D
authors.initials: SD
authors.lastname: Walter
authors.forename: Geoff
authors.initials: G
authors.lastname: Norman
authors.forename: Mahan
authors.initials: M
authors.lastname: Kulasegaram
authors.forename: Robin
authors.initials: R
authors.lastname: Christensen
authors.forename: Gordon H
authors.initials: GH
authors.lastname: Guyatt
authors.forename: Yngve
authors.initials: Y
authors.lastname: Falck-Ytter
authors.forename: Stephanie
authors.initials: S
authors.lastname: Chang
authors.forename: Mohammad Hassan
authors.initials: MH
authors.lastname: Murad
authors.forename: Gunn E
authors.initials: GE
authors.lastname: Vist
authors.forename: Toby
authors.initials: T
authors.lastname: Lasserson
authors.forename: Gerald
authors.initials: G
authors.lastname: Gartlehner
authors.forename: Vijay
authors.initials: V
authors.lastname: Shukla
authors.forename: Xin
authors.initials: X
authors.lastname: Sun
authors.forename: Craig
authors.initials: C
authors.lastname: Whittington
authors.forename: Piet N
authors.initials: PN
authors.lastname: Post
authors.forename: Eddy
authors.initials: E
authors.lastname: Lang
authors.forename: Kylie
authors.initials: K
authors.lastname: Thaler
authors.forename: Ilkka
authors.initials: I
authors.lastname: Kunnamo
authors.forename: Heidi
authors.initials: H
authors.lastname: Alenius
authors.forename: Joerg J
authors.initials: JJ
authors.lastname: Meerpohl
authors.forename: Ana C
authors.initials: AC
authors.lastname: Alba
authors.forename: Immaculate F
authors.initials: IF
authors.lastname: Nevis
authors.forename: Stephen
authors.initials: S
authors.lastname: Gentles
authors.forename: Marie-Chantal
authors.initials: MC
authors.lastname: Ethier
authors.forename: Alonso
authors.initials: A
authors.lastname: Carrasco-Labra
authors.forename: Rasha
authors.initials: R
authors.lastname: Khatib
authors.forename: Gihad
authors.initials: G
authors.lastname: Nesrallah
authors.forename: Jamie
authors.initials: J
authors.lastname: Kroft
authors.forename: Amanda
authors.initials: A
authors.lastname: Selk
authors.forename: Romina
authors.initials: R
authors.lastname: Brignardello-Petersen
authors.forename: Holger J
authors.initials: HJ
authors.lastname: Schünemann
mesh_terms: D002170:Canada; D019317:Evidence-Based Medicine / Q000592:standards*; D006801:Humans; D017410:Practice Guidelines as Topic / Q000592:standards*; D029424:Pulmonary Disease, Chronic Obstructive / Q000628:therapy; D015203:Reproducibility of Results; D012107:Research Design; D012648:Self Care / Q000379:methods; D011795:Surveys and Questionnaires; D054928:Validation Studies as Topic