The GRACE Checklist: A Validated Assessment Tool for High Quality Observational Studies of Comparative Effectiveness.
BACKGROUND
Recognizing the growing need for robust evidence about treatment effectiveness in real-world populations, the Good Research for Comparative Effectiveness (GRACE) guidelines have been developed for noninterventional studies of comparative effectiveness to determine which studies are sufficiently rigorous to be reliable enough for use in health technology assessments.


OBJECTIVE
To evaluate which aspects of the GRACE Checklist contribute most strongly to recognition of quality.


METHODS
We assembled 28 observational comparative effectiveness articles published from 2001 to 2010 that compared treatment effectiveness and/or safety of drugs, medical devices, and medical procedures. Twenty-two volunteers from academia, pharmaceutical companies, and government agencies applied the GRACE Checklist to those articles, providing 56 assessments. Ten senior academic and industry experts provided assessments of overall article quality for the purpose of decision support. We also rated each article based on the number of annual citations and impact factor of the journal in which the article was published. To identify checklist items that were most predictive of quality, classification and regression tree (CART) analysis, a binary, recursive, partitioning methodology, was used to create 3 decision trees, which compared the 56 article assessments with 3 external quality outcomes: (1) expert assessment of overall quality, (2) citation frequency, and (3) impact factor. A fourth tree looked at the composite outcome of all 3 quality indicators.


RESULTS
The best predictors of quality included the following: use of concurrent comparators, limiting the study to new initiators of the study drug, equivalent measurement of outcomes in study groups, collecting data on most if not all known confounders or effect modifiers, accounting for immortal time bias in the analysis, and use of sensitivity analyses to test how much effect estimates depended on various assumptions. Only sensitivity analyses appeared consistently as a predictor of quality in all 4 trees. When a composite outcome of the 3 quality measures was used, the GRACE Checklist showed high sensitivity and specificity (71.43% and 80.95%, respectively).


CONCLUSIONS
The GRACE Checklist stands out from other consensus-driven and expert guidance documents because of its extensive validation efforts. This most recent work shows that the checklist has strong sensitivity and specificity, increasing its utility as a screening tool to identify high-quality observational comparative effectiveness research worthy of in-depth review and applicability for decision support.


BACKGROUND
No outside funding supported this research. All authors are full-time employees of Quintiles, which provides research and consulting services to the biopharmaceutical industry. The authors have no other disclosures to report. Two of the 3 CART trees were presented at the International Society of Pharmacepidemiology in 2015 ("Article Citations per Year" and "Journal Impact Factor"). The original validation study was published in the March 2014 issue of the Journal of Managed Care & Specialty Pharmacy. The checklist questions and scoring were included using a table that was originally published by this journal in 2014. Study concept and design were primarily contributed by Dreyer and Velentgas, along with Bryant. Bryant took the lead in data collection and analysis, along with Dreyer and Velentgas, and data interpretation was performed by Dreyer, Velentgas, and Bryant. The manuscript was written and revised primarily by Dreyer, along with Bryant and Velentgas.
1 Quintiles Real-World & Late-Phase Research, Cambridge, Massachusetts.
Nancy A
NA
Dreyer
1 Quintiles Real-World & Late-Phase Research, Cambridge, Massachusetts.
Allison
A
Bryant
1 Quintiles Real-World & Late-Phase Research, Cambridge, Massachusetts.
Priscilla
P
Velentgas
D057189:Checklist; D057186:Comparative Effectiveness Research / Q000592:standards*; D003362:Cost-Benefit Analysis; D003663:Decision Trees; D004358:Drug Therapy / Q000706:statistics & numerical data; D004864:Equipment and Supplies / Q000706:statistics & numerical data; D019317:Evidence-Based Medicine; D006801:Humans; D055811:Journal Impact Factor; D064887:Observational Studies as Topic / Q000379:methods*; D011786:Quality Control; D015203:Reproducibility of Results; D012107:Research Design; D012680:Sensitivity and Specificity; D013673:Technology Assessment, Biomedical
Value Health. 2009 Nov-Dec;12(8):1044-52
19793072
J Clin Epidemiol. 2016 Mar;71:3-10
26548541
J Comp Eff Res. 2013 Sep;2(5):413-8
24236736
Nat Rev Rheumatol. 2015 Jul;11(7):437-41
25800216
N Engl J Med. 2015 Nov 5;373(19):1797-9
26535510
Value Health. 2014 Mar;17(2):143-56
24636373
Am J Manag Care. 2010 Jun;16(6):467-71
20560690
Value Health. 2012 Mar-Apr;15(2):217-30
22433752
J Manag Care Spec Pharm. 2014 Mar;20(3):301-8
24564810
Value Health. 2009 Nov-Dec;12(8):1062-73
19793071
J Comp Eff Res. 2016 Jan;5(1):87-98
26691269
N Engl J Med. 2015 Dec 31;373(27):2595-7
26580666
Am J Public Health. 2005;95 Suppl 1:S144-50
16030331
BMJ. 2000 Jul 29;321(7256):255-6
10915111
N Engl J Med. 2015 Nov 5;373(19):1793-7
26535509
N Engl J Med. 2015 Dec 31;373(27):2593-5
26580243
Value Health. 2009 Nov-Dec;12(8):1053-61
19744292