The predictive validity of quality of evidence grades for the stability of effect estimates was low: a meta-epidemiological study.
OBJECTIVE
To determine the predictive validity of the U.S. Evidence-based Practice Center (EPC) approach to GRADE (Grading of Recommendations Assessment, Development and Evaluation).


METHODS
Based on Cochrane reports with outcomes graded as high quality of evidence (QOE), we prepared 160 documents which represented different levels of QOE. Professional systematic reviewers dually graded the QOE. For each document, we determined whether estimates were concordant with high QOE estimates of the Cochrane reports. We compared the observed proportion of concordant estimates with the expected proportion from an international survey. To determine the predictive validity, we used the Hosmer-Lemeshow test to assess calibration and the C (concordance) index to assess discrimination.


RESULTS
The predictive validity of the EPC approach to GRADE was limited. Estimates graded as high QOE were less likely, estimates graded as low or insufficient QOE more likely to remain stable than expected. The EPC approach to GRADE could not reliably predict the likelihood that individual bodies of evidence remain stable as new evidence becomes available. C-indices ranged between 0.56 (95% CI, 0.47 to 0.66) and 0.58 (95% CI, 0.50 to 0.67) indicating a low discriminatory ability.


CONCLUSIONS
The limited predictive validity of the EPC approach to GRADE seems to reflect a mismatch between expected and observed changes in treatment effects as bodies of evidence advance from insufficient to high QOE.
Cochrane Austria, Danube University, Krems, Austria; RTI International, Research Triangle Park, NC, USA. Electronic address: gerald.gartlehner@donau-uni.ac.at.
Gerald
G
Gartlehner
Victor Babes University of Medicine and Pharmacy, Timisoara, Romania.
Andreea
A
Dobrescu
RTI International, Research Triangle Park, NC, USA.
Tammeka Swinson
TS
Evans
RTI International, Research Triangle Park, NC, USA.
Carla
C
Bann
Johns Hopkins University Medical Center, Baltimore, MD, USA.
Karen A
KA
Robinson
ECRI Institute, Plymouth Meeting, PA, USA.
James
J
Reston
Cochrane Austria, Danube University, Krems, Austria.
Kylie
K
Thaler
Spectrum Research, Inc., Tacoma, Washington.
Andrea
A
Skelly
Cochrane Austria, Danube University, Krems, Austria.
Anna
A
Glechner
Oregon Health & Science University, Portland, OR, USA.
Kimberly
K
Peterson
Cochrane Austria, Danube University, Krems, Austria.
Christina
C
Kien
RTI International, Research Triangle Park, NC, USA.
Kathleen N
KN
Lohr
D016021:Epidemiologic Studies; D055317:Evidence-Based Practice; D006801:Humans; D015203:Reproducibility of Results; D012107:Research Design; D054928:Validation Studies as Topic
GRADE; Methods study; Predictive validity; Quality of evidence; Stability of effects; Strength of evidence; Systematic reviews