Evaluating meta-analyses in the general surgical literature: a critical appraisal.
OBJECTIVE
To assess the methodologic quality of meta-analyses of general surgery topics published in peer-reviewed journals.


BACKGROUND
Systematic reviews and meta-analysis are used to seek, summarize, and interpret primary studies on a given topic. Accordingly, systematic reviews and meta-analyses of high-quality primary studies may be the highest level of evidence for issues of prevention and treatment in evidence-based medicine. However, not all published meta-analyses are rigorously performed.


METHODS
We searched MEDLINE (from January 1, 1997, to September 1, 2002) and reference lists and solicited general surgery specialists to identify relevant meta-analyses. Inclusion criteria were use of meta-analytic methods to pool the results of primary studies in general surgery on issues of diagnosis, causation, prognosis, or treatment. Our search strategies identified 487 potentially relevant articles. After excluding articles based on a priori criteria, 51 meta-analyses fulfilled eligibility criteria. In duplicate and independently, 2 reviewers assessed the quality of these meta-analyses using a 10-item index called the Overview Quality Assessment Questionnaire.


RESULTS
Overall concordance between 2 independent reviewers was good (interobserver agreement 81%, and a kappa of 0.62 (95% CI 0.55-0.69). Of 51 relevant articles, 38 were published in surgical journals. Most studies had major methodologic flaws (median score of 3.3, scale of 1-7). Factors associated with low overall scientific quality included the absence of any prior meta-analyses publications by authors and meta-analyses produced by surgical department members without external collaboration.


CONCLUSIONS
This critical appraisal of meta-analyses published in the general surgery literature demonstrates frequent methodologic flaws. The quality of these reports limits the validity of the findings and the inferences that can be made about the primary studies reviewed. To improve the quality of future meta-analyses, we recommend following guidelines for the optimal conduct and reporting of meta-analyses in general surgery.
Dixon
Elijah
E
Department of Surgery, University of Calgary, Calgary, Alberta, Canada.
Hameed
Morad
M
Sutherland
Francis
F
Cook
Deborah J
DJ
Doig
Christopher
C
D013502:General Surgery; D015201:Meta-Analysis as Topic; D011643:Publishing / Q000592:standards; D012196:Review Literature as Topic
J Gastrointest Surg. 1999 May-Jun;3(3):225-32
10481115
Ann Thorac Surg. 2001 Jul;72(1):306-13
11465217
Lancet. 1998 Aug 22;352(9128):609-13
9746022
Ann Emerg Med. 2001 Nov;38(5):518-26
11679863
Semin Liver Dis. 1999;19(4):457-73
10643629
Drug Saf. 2000 Nov;23(5):449-61
11085349
JAMA. 1990 Mar 9;263(10):1392-5
2406473
Lancet. 1995 Mar 25;345(8952):772-6
7891492
J Clin Epidemiol. 1991;44(1):91-8
1824710
Arch Surg. 2000 Aug;135(8):948-58
10922258
Br J Surg. 1998 Feb;85(2):171-8
9501809
Am Surg. 2001 May;67(5):421-6
11379640
Onkologie. 2000 Dec;23(6):597-602
11441269
Br J Cancer. 2000 Mar;82(5):1017-21
10737383
Aust N Z J Surg. 1997 Nov;67(11):750-4
9396988
Br J Surg. 2002 Jun;89(6):695-703
12027978
Br J Surg. 1997 Jun;84(6):855-7
9189108
Dis Colon Rectum. 1998 Feb;41(2):180-9
9556242
Dis Esophagus. 2001;14(3-4):212-7
11869322
J Clin Epidemiol. 1991;44(11):1271-8
1834807
Can J Surg. 2002 Jun;45(3):173-80
12067168
Dis Colon Rectum. 1998 Jul;41(7):875-82; discussion 882-3
9678373
JAMA. 1995 Feb 1;273(5):408-12
7823387
Surg Endosc. 1999 Jul;13(7):689-94
10384076
JAMA. 2002 Jun 12;287(22):2973-82
12052127
Br J Surg. 2001 Jul;88(7):913-30
11442521
Am J Surg. 2002 Jul;184(1):6-10
12135710
Control Clin Trials. 1997 Dec;18(6):568-79; discussion 661-6
9408719
Br J Surg. 1997 Jun;84(6):750-9
9189079
JAMA. 1998 Apr 8;279(14):1089-93
9546568
BMJ. 2002 Apr 6;324(7341):813
11934773
Int J Technol Assess Health Care. 1996 Spring;12(2):195-208
8707495
Ann Surg. 2002 Jul;236(1):17-27
12131081
Ann R Coll Surg Engl. 1998;80 Suppl 1:S1-80
11432408
Dis Colon Rectum. 1999 Nov;42(11):1424-8; discussion 1428-31
10566530
Can J Surg. 1999 Oct;42(5):377-83
10526524
J Reprod Med. 1999 Apr;44(4):325-31
10319300
Surg Endosc. 2002 Jul;16(7):1098-104
12165830
Lancet. 2000 Feb 26;355(9205):756-7
10703836
Am J Gastroenterol. 1997 Jan;92(1):109-13
8995948
Am J Surg. 1998 Dec;176(6):666-70
9926810
Cancer. 1999 Feb 1;85(3):616-28
10091735
BMJ. 2001 Oct 6;323(7316):773-6
11588077
Obes Surg. 2000 Oct;10(5):391-401
11054242
Stat Med. 2002 Jun 15;21(11):1513-24
12111917
Arch Surg. 2002 Apr;137(4):458-61
11926952
J Clin Epidemiol. 2001 Jul;54(7):651-4
11438404
J Hepatol. 2001 Feb;34(2):342-5
11281568
Surg Laparosc Endosc. 1999 Jan;9(1):17-26
9950122
Dis Colon Rectum. 1998 Sep;41(9):1116-26
9749495
JAMA. 1995 Dec 13;274(22):1800-4
7500513
Am Surg. 1998 Nov;64(11):1050-8
9798767
J Surg Res. 2000 May 15;90(2):161-5
10792958
J Clin Epidemiol. 2000 Dec;53(12):1193-9
11146264
Surgery. 2000 Feb;127(2):136-41
10686977
Haemostasis. 1997 Mar-Apr;27(2):65-74
9212354
Ann Intern Med. 2001 Dec 4;135(11):982-9
11730399
BMJ. 1997 Sep 13;315(7109):629-34
9310563
Surgery. 2000 Dec;128(6):1075-81
11114645
Am J Surg. 2001 Nov;182(5):470-5
11754853
Cancer J Sci Am. 1997 Jan-Feb;3(1):6-12
9072310
Ann Surg. 2002 Mar;235(3):322-32
11882753
Am J Surg. 2000 Jul;180(1):58-61
11036143
Ann Surg. 1999 Apr;229(4):467-77
10203078
Anesth Analg. 2001 Mar;92(3):700-9
11226105
JAMA. 1998 Jul 15;280(3):278-80
9676681
Liver Transpl. 2000 May;6(3):302-8
10827230
Ann Surg. 2001 Nov;234(5):590-606
11685021
N Engl J Med. 1987 Feb 19;316(8):450-5
3807986
Ann Surg. 2000 Mar;231(3):436-42
10714638
Ann Surg Oncol. 1999 Jan-Feb;6(1):109-16
10030423
Ann Surg. 1999 Feb;229(2):174-80
10024097
J Clin Gastroenterol. 1997 Dec;25(4):628-33
9451677
Dis Colon Rectum. 2000 Jul;43(7):911-9
10910235
N Engl J Med. 1997 Aug 21;337(8):536-42
9262498