Are potentially clinically meaningful benefits misinterpreted in cardiovascular randomized trials? A systematic examination of statistical significance, clinical significance, and authors' conclusions.
While journals and reporting guidelines recommend the presentation of confidence intervals, many authors adhere strictly to statistically significant testing. Our objective was to determine what proportions of not statistically significant (NSS) cardiovascular trials include potentially clinically meaningful effects in primary outcomes and if these are associated with authors' conclusions.



Cardiovascular studies published in six high-impact journals between 1 January 2010 and 31 December 2014 were identified via PubMed. Two independent reviewers selected trials with major adverse cardiovascular events (stroke, myocardial infarction, or cardiovascular death) as primary outcomes and extracted data on trial characteristics, quality, and primary outcome. Potentially clinically meaningful effects were defined broadly as a relative risk point estimate ≤0.94 (based on the effects of ezetimibe) and/or a lower confidence interval ≤0.75 (based on the effects of statins).



We identified 127 randomized trial comparisons from 3200 articles. The primary outcomes were statistically significant (SS) favoring treatment in 21% (27/127), NSS in 72% (92/127), and SS favoring control in 6% (8/127). In 61% of NSS trials (56/92), the point estimate and/or lower confidence interval included potentially meaningful effects. Both point estimate and confidence interval included potentially meaningful effects in 67% of trials (12/18) in which authors' concluded that treatment was superior, in 28% (16/58) with a neutral conclusion, and in 6% (1/16) in which authors' concluded that control was superior. In a sensitivity analysis, 26% of NSS trials would include potential meaningful effects with relative risk thresholds of point estimate ≤0.85 and/or a lower confidence interval ≤0.65.



Point estimates and/or confidence intervals included potentially clinically meaningful effects in up to 61% of NSS cardiovascular trials. Authors' conclusions often reflect potentially meaningful results of NSS cardiovascular trials. Given the frequency of potentially clinical meaningful effects in NSS trials, authors should be encouraged to continue to look beyond significance testing to a broader interpretation of trial results.
Evidence-Based Medicine, Department of Family Medicine - Research Program, University of Alberta, 6-10 University Terrace, Edmonton, AB, T6G 2T4, Canada. michael.allan@ualberta.ca.
G Michael
GM
Allan
Evidence-Based Medicine, Department of Family Medicine - Research Program, University of Alberta, 6-10 University Terrace, Edmonton, AB, T6G 2T4, Canada.
Caitlin R
CR
Finley
Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, British Columbia, Canada.
James
J
McCormack
Evidence-Based Medicine, Department of Family Medicine - Research Program, University of Alberta, 6-10 University Terrace, Edmonton, AB, T6G 2T4, Canada.
Vivek
V
Kumar
Evidence-Based Medicine, Department of Family Medicine - Research Program, University of Alberta, 6-10 University Terrace, Edmonton, AB, T6G 2T4, Canada.
Simon
S
Kwong
Family Medicine, McGill University, Montreal, QC, Canada.
Emelie
E
Braschi
Evidence-Based Medicine, Department of Family Medicine - Research Program, University of Alberta, 6-10 University Terrace, Edmonton, AB, T6G 2T4, Canada.
Christina
C
Korownyk
Evidence-Based Medicine, Department of Family Medicine - Research Program, University of Alberta, 6-10 University Terrace, Edmonton, AB, T6G 2T4, Canada.
Michael R
MR
Kolber
Evidence-Based Medicine, Department of Family Medicine - Research Program, University of Alberta, 6-10 University Terrace, Edmonton, AB, T6G 2T4, Canada.
Adriennne J
AJ
Lindblad
Medical Education, Department of Family Medicine, University of Alberta, Edmonton, AB, Canada.
Oksana
O
Babenko
Evidence-Based Medicine, Department of Family Medicine - Research Program, University of Alberta, 6-10 University Terrace, Edmonton, AB, T6G 2T4, Canada.
Scott
S
Garrison
D002318:Cardiovascular Diseases / Q000628:therapy*; D006801:Humans; D016032:Randomized Controlled Trials as Topic
Cardiovascular; Clinical significance; Conclusions; Confidence intervals; Randomized controlled trials; Statistical significance