The contribution of databases to the results of systematic reviews: a cross-sectional study.
BACKGROUND
One of the best sources for high quality information about healthcare interventions is a systematic review. A well-conducted systematic review includes a comprehensive literature search. There is limited empiric evidence to guide the extent of searching, in particular the number of electronic databases that should be searched. We conducted a cross-sectional quantitative analysis to examine the potential impact of selective database searching on results of meta-analyses.


METHODS
Our sample included systematic reviews (SRs) with at least one meta-analysis from three Cochrane Review Groups: Acute Respiratory Infections (ARI), Infectious Diseases (ID), Developmental Psychosocial and Learning Problems (DPLP) (n = 129). Outcomes included: 1) proportion of relevant studies indexed in each of 10 databases; and 2) changes in results and statistical significance of primary meta-analysis for studies identified in Medline only and in Medline plus each of the other databases.


RESULTS
Due to variation across topics, we present results by group (ARI n = 57, ID n = 38, DPLP n = 34). For ARI, identification of relevant studies was highest for Medline (85 %) and Embase (80 %). Restricting meta-analyses to trials that appeared in Medline + Embase yielded fewest changes in statistical significance: 53/55 meta-analyses showed no change. Point estimates changed in 12 cases; in 7 the change was less than 20 %. For ID, yield was highest for Medline (92 %), Embase (81 %), and BIOSIS (67 %). Restricting meta-analyses to trials that appeared in Medline + BIOSIS yielded fewest changes with 1 meta-analysis changing in statistical significance. Point estimates changed in 8 of 31 meta-analyses; change less than 20 % in all cases. For DPLP, identification of relevant studies was highest for Medline (75 %) and Embase (62 %). Restricting meta-analyses to trials that appeared in Medline + PsycINFO resulted in only one change in significance. Point estimates changed for 13 of 33 meta-analyses; less than 20 % in 9 cases.


CONCLUSIONS
Majority of relevant studies can be found within a limited number of databases. Results of meta-analyses based on the majority of studies did not differ in most cases. There were very few cases of changes in statistical significance. Effect estimates changed in a minority of meta-analyses but in most the change was small. Results did not change in a systematic manner (i.e., regularly over- or underestimating treatment effects), suggesting that selective searching may not introduce bias in terms of effect estimates.
Cochrane Child Health, University of Alberta, ECHA4-472, 11405-87 Avenue, Edmonton, AB, T6G 1C9, Canada. hartling@ualberta.ca.|Alberta Research Center for Health Evidence (ARCHE), University of Alberta, 11405-87 Avenue, Edmonton, AB, T6G 1C9, Canada. hartling@ualberta.ca.
Lisa
L
Hartling
Alberta Research Center for Health Evidence (ARCHE), University of Alberta, 11405-87 Avenue, Edmonton, AB, T6G 1C9, Canada.
Robin
R
Featherstone
Alberta Research Center for Health Evidence (ARCHE), University of Alberta, 11405-87 Avenue, Edmonton, AB, T6G 1C9, Canada.
Megan
M
Nuspl
Alberta Research Center for Health Evidence (ARCHE), University of Alberta, 11405-87 Avenue, Edmonton, AB, T6G 1C9, Canada.
Kassi
K
Shave
Alberta Research Center for Health Evidence (ARCHE), University of Alberta, 11405-87 Avenue, Edmonton, AB, T6G 1C9, Canada.
Donna M
DM
Dryden
Alberta Research Center for Health Evidence (ARCHE), University of Alberta, 11405-87 Avenue, Edmonton, AB, T6G 1C9, Canada.
Ben
B
Vandermeer
Bias; Knowledge synthesis; Literature searching; Meta-analysis; Systematic reviews
Health Info Libr J. 2013 Mar;30(1):49-58
23413793
PLoS Med. 2007 Mar 27;4(3):e78
17388659
Cochrane Database Syst Rev. 2014 Jun 17;(6):CD004381
24936966
PLoS Med. 2016 May 24;13(5):e1002028
27218655
Complement Ther Med. 2004 Dec;12(4):228-32
15649836
J Clin Epidemiol. 2011 Jan;64(1):6-10
19926445
J Clin Epidemiol. 2009 Dec;62(12):1261-7
19364634
BMC Med Res Methodol. 2015 Mar 29;15:26
25888523
Cochrane Database Syst Rev. 2013 Mar 28;(3):CD004559
23543533
Int J Technol Assess Health Care. 2014 Apr;30(2):173-8
24774535
Syst Rev. 2013 Dec 23;2:115
24360284
BMC Med Res Methodol. 2005 Jan 28;5(1):6
15679886
Syst Rev. 2014 Nov 11;3:135
25387523
J Clin Epidemiol. 2015 Dec;68(12):1451-62.e3
26278023
J Clin Epidemiol. 2015 Sep;68(9):1076-84
26279401
Diabet Med. 2005 Oct;22(10):1386-93
16176201
Implement Sci. 2010 Jul 19;5:56
20642853
J Clin Epidemiol. 2003 Oct;56(10):943-55
14568625
Control Clin Trials. 2000 Oct;21(5):476-87
11018564
J Clin Epidemiol. 2008 Apr;61(4):357-364
18313560
Int J Technol Assess Health Care. 2012 Apr;28(2):133-7
22559754
J Public Health Res. 2014 Jul 01;3(2):177
25343133
J Epidemiol Community Health. 2005 Sep;59(9):804-8
16100321
BMC Med. 2015 Sep 16;13:224
26377409
Cochrane Database Syst Rev. 2012 Dec 12;12:CD001957
23235586
J Clin Epidemiol. 2005 Sep;58(9):867-73
16085190
Int J Technol Assess Health Care. 2003 Fall;19(4):591-603
15095765
Cochrane Database Syst Rev. 2008 Apr 16;(2):CD006358
18425950
Health Libr Rev. 1999 Sep;16(3):151-6
10620849
Int J Technol Assess Health Care. 2014 Jan;30(1):20-7
24451157
Health Technol Assess. 2003;7(34):iii, ix-x, 1-51
14609481
Int J Technol Assess Health Care. 2003 Winter;19(1):168-78
12701949
BMC Pediatr. 2010 May 20;10:34
20487565
Cochrane Database Syst Rev. 2013 Feb 28;(2):CD008898
23450592
BMC Med Res Methodol. 2005 Jan 08;5:2
15638944