Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials.
BACKGROUND
Specific features of trial quality may be associated with exaggeration or shrinking of the observed treatment effect in randomized studies. Therefore, assessment of trial quality is often used in meta-analysis. However, the degree to which specific quality measures are associated with treatment effects has not been well established across a broad range of clinical areas.


OBJECTIVE
To determine if quality measures are associated with treatment effect size in randomized controlled trials (RCTs).


METHODS
Quality measures from published quality assessment scales were evaluated in RCTs included in meta-analyses from 4 medical areas (cardiovascular disease, infectious disease, pediatrics, and surgery). Included meta-analyses incorporated at least 6 RCTs, examined dichotomous outcomes, and demonstrated significant between-study heterogeneity in the odds ratio (OR) scale.


METHODS
Relative ORs comparing overall treatment effect (summary OR) of high vs low-quality studies, as determined by each quality measure, with relative ORs less than 1 indicating larger treatment effect in low-quality studies.


RESULTS
Twenty-four quality measures were analyzed for 276 RCTs from 26 meta-analyses. Relative ORs of high vs low-quality studies for these quality measures ranged from 0.83 to 1.26; none was statistically significantly associated with treatment effect. The proportion of studies fulfilling specific quality measures varied widely in the 4 medical areas. In analyses limited to specific medical areas, placebo control, multicenter studies, study country, caregiver blinding, and statistical methods were significantly associated with treatment effect on 7 occasions. These relative ORs ranged from 0.40 to 1.74. However, the directions of these associations were not consistent.


CONCLUSIONS
Individual quality measures are not reliably associated with the strength of treatment effect across studies and medical areas. Although use of specific quality measures may be appropriate in specific well-defined areas in which there is pertinent evidence, findings of associations with treatment effect cannot be generalized to all clinical areas or meta-analyses.
Balk
Ethan M
EM
Evidence-based Practice Center, Division of Clinical Care Research, Tufts University School of Medicine, New England Medical Center, 750 Washington St, No. 63, Boston, MA 02111, USA.
Bonis
Peter A L
PA
Moskowitz
Harry
H
Schmid
Christopher H
CH
Ioannidis
John P A
JP
Wang
Chenchen
C
Lau
Joseph
J
D003627:Data Interpretation, Statistical; D015201:Meta-Analysis as Topic; D011786:Quality Control; D016032:Randomized Controlled Trials as Topic / Q000592:standards* / Q000706:statistics & numerical data; D012107:Research Design; D016896:Treatment Outcome