The interpretation of systematic reviews with meta-analyses: an objective or subjective process?
BACKGROUND
Discrepancies between the conclusions of different meta-analyses (quantitative syntheses of systematic reviews) are often ascribed to methodological differences. The objective of this study was to determine the discordance in interpretations when meta-analysts are presented with identical data.


METHODS
We searched the literature for all randomized clinical trials (RCT) and review articles on the efficacy of intravenous magnesium in the early post-myocardial infarction period. We organized the articles chronologically and grouped them in packages. The first package included the first RCT, and a summary of the review articles published prior to first RCT. The second package contained the second and third RCT, a meta-analysis based on the data, and a summary of all review articles published prior to the third RCT. Similar packages were created for the 5th RCT, 10th RCT, 20th RCT and 23rd RCT (all articles). We presented the packages one at a time to eight different reviewers and asked them to answer three clinical questions after each package based solely on the information provided. The clinical questions included whether 1) they believed magnesium is now proven beneficial, 2) they believed magnesium will eventually be proven to be beneficial, and 3) they would recommend its use at this time.


RESULTS
There was considerable disagreement among the reviewers for each package, and for each question. The discrepancies increased when the heterogeneity of the data increased. In addition, some reviewers became more sceptical of the effectiveness of magnesium over time, and some reviewers became less sceptical.


CONCLUSIONS
The interpretation of the results of systematic reviews with meta-analyses includes a subjective component that can lead to discordant conclusions that are independent of the methodology used to obtain or analyse the data.
Shrier
Ian
I
Centre for Clinical Epidemiology and Community Studies, Lady Davis Institute for Medical Research, SMBD-Jewish General Hospital, McGill University, Montreal, Canada. ian.shrier@mcgill.ca
Boivin
Jean-François
JF
Platt
Robert W
RW
Steele
Russell J
RJ
Brophy
James M
JM
Carnevale
Franco
F
Eisenberg
Mark J
MJ
Furlan
Andrea
A
Kakuma
Ritsuko
R
Macdonald
Mary Ellen
ME
Pilote
Louise
L
Rossignol
Michel
M
D000889:Anti-Arrhythmia Agents / Q000627:therapeutic use; D003657:Decision Making; D003661:Decision Support Techniques; D006801:Humans; D008278:Magnesium Sulfate / Q000627:therapeutic use; D015201:Meta-Analysis as Topic; D009203:Myocardial Infarction / Q000188:drug therapy / Q000401:mortality; D015588:Observer Variation; D016017:Odds Ratio; D036301:Qualitative Research; D016032:Randomized Controlled Trials as Topic; D015203:Reproducibility of Results; D012196:Review Literature as Topic
D000889:Anti-Arrhythmia Agents; D008278:Magnesium Sulfate