title: Grades for quality of evidence were associated with distinct likelihoods that treatment effects will remain stable.
abstract: OBJECTIVE
  We sought to determine whether producers or users of systematic reviews using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach or a close variation give the same meanings to terms intended to convey uncertainty about treatment effects when interpreting grades for the quality or strength of evidence.
  
  
  METHODS
  Following exploratory interviews with stakeholders and user testing, we conducted an international Web-based survey among producers and users of systematic reviews. For each quality grade (high, moderate, low, very low/insufficient), we asked participants to assign a minimum likelihood that treatment effects will not change substantially as new studies emerge. Using multivariate analysis of covariance, we tested whether the estimated likelihoods differed between producers and users.
  
  
  RESULTS
  In all, 244 participants completed the survey. The associated minimum likelihoods that treatment effects will not change substantially for high, moderate, and low grades of quality of evidence (QOE) were 86% [95% confidence interval (CI): 85%, 87%], 61% (95% CI: 59%, 63%), and 34% (95% CI: 32%, 36%), respectively (very low was preset at 0%). Likelihoods for each grade were similar between producers and users of systematic reviews (P > 0.05 for all comparisons).
  
  
  CONCLUSIONS
  GRADE is, in general, a suitable method to convey uncertainties for systematic review producers to users. The wide ranges of likelihoods associated with GRADE terms suggest that current definitions of levels of QOE that rely exclusively on qualitative certainty expressions should be augmented by numerical predictions once such data are available.
authors.affiliation: Department for Clinical Epidemiology an Evidence-based Medicine, Austrian Cochrane Branch, Danube University Krems, Dr. Karl Dorrekstrasse 30, 3500 Krems, Austria; RTI-UNC Evidence-based Practice Center, Research Triangle Institute International, 3040 East Cornwallis Road, Research Triangle Park, NC 27709-2194, USA. Electronic address: gerald.gartlehner@donau-uni.ac.at.
authors.forename: Gerald
authors.initials: G
authors.lastname: Gartlehner
authors.affiliation: Department for Clinical Epidemiology an Evidence-based Medicine, Austrian Cochrane Branch, Danube University Krems, Dr. Karl Dorrekstrasse 30, 3500 Krems, Austria.
authors.forename: Isolde
authors.initials: I
authors.lastname: Sommer
authors.affiliation: RTI-UNC Evidence-based Practice Center, Research Triangle Institute International, 3040 East Cornwallis Road, Research Triangle Park, NC 27709-2194, USA.
authors.forename: Tammeka Swinson
authors.initials: TS
authors.lastname: Evans
authors.affiliation: Department for Clinical Epidemiology an Evidence-based Medicine, Austrian Cochrane Branch, Danube University Krems, Dr. Karl Dorrekstrasse 30, 3500 Krems, Austria.
authors.forename: Kylie
authors.initials: K
authors.lastname: Thaler
authors.affiliation: RTI-UNC Evidence-based Practice Center, Research Triangle Institute International, 3040 East Cornwallis Road, Research Triangle Park, NC 27709-2194, USA.
authors.forename: Kathleen N
authors.initials: KN
authors.lastname: Lohr
mesh_terms: D003625:Data Collection; D055317:Evidence-Based Practice / Q000145:classification / Q000592:standards*; D016013:Likelihood Functions; D012196:Review Literature as Topic
keywords: Certainty of treatment effects; GRADE; Quality of evidence; Stability of treatment effects; Survey; Systematic reviews