title: US studies may overestimate effect sizes in softer research.
abstract: Many biases affect scientific research, causing a waste of resources, posing a threat to human health, and hampering scientific progress. These problems are hypothesized to be worsened by lack of consensus on theories and methods, by selective publication processes, and by career systems too heavily oriented toward productivity, such as those adopted in the United States (US). Here, we extracted 1,174 primary outcomes appearing in 82 meta-analyses published in health-related biological and behavioral research sampled from the Web of Science categories Genetics & Heredity and Psychiatry and measured how individual results deviated from the overall summary effect size within their respective meta-analysis. We found that primary studies whose outcome included behavioral parameters were generally more likely to report extreme effects, and those with a corresponding author based in the US were more likely to deviate in the direction predicted by their experimental hypotheses, particularly when their outcome did not include additional biological parameters. Nonbehavioral studies showed no such "US effect" and were subject mainly to sampling variance and small-study effects, which were stronger for non-US countries. Although this latter finding could be interpreted as a publication bias against non-US authors, the US effect observed in behavioral research is unlikely to be generated by editorial biases. Behavioral studies have lower methodological consensus and higher noise, making US researchers potentially more likely to express an underlying propensity to report strong and significant findings.
authors: Science, Technology and Innovation Studies, The University of Edinburgh, Edinburgh EH1 1LZ, United Kingdom.
authors: Daniele
authors: D
authors: Fanelli
authors: John P A
authors: JP
authors: Ioannidis
mesh_terms: D035841:Behavioral Research; D035843:Biomedical Research; D035081:Editorial Policies; D006801:Humans; D015201:Meta-Analysis as Topic; D017594:Publication Bias; D011643:Publishing; D014481:United States
keywords: publish or perish; questionable research practices; research bias; scientific misconduct; soft science
references: CMAJ. 2004 Sep 28;171(7):735-40
references: 15451835
references: BMC Med Res Methodol. 2006 Aug 01;6:37
references: 16879753
references: PLoS One. 2010 Apr 21;5(4):e10271
references: 20422014
references: Nat Genet. 2001 Nov;29(3):306-9
references: 11600885
references: Epidemiology. 2008 Sep;19(5):640-8
references: 18633328
references: BMJ. 2005 Apr 2;330(7494):753
references: 15681569
references: Perspect Psychol Sci. 2011 Mar;6(2):163-71
references: 26162135
references: Psychol Sci. 2011 Nov;22(11):1359-66
references: 22006061
references: Nature. 2010 Jan 14;463(7278):142-3
references: 20075887
references: Braz J Med Biol Res. 2003 Sep;36(9):1135-41
references: 12937778
references: Psychol Methods. 2012 Mar;17(1):120-8
references: 21787082
references: JAMA. 2004 May 26;291(20):2457-65
references: 15161896
references: Health Technol Assess. 2010 Feb;14(8):iii, ix-xi, 1-193
references: 20181324
references: Behav Res Methods. 2011 Sep;43(3):666-78
references: 21494917
references: J Bone Joint Surg Am. 2007 May;89(5):1010-8
references: 17473138
references: World Psychiatry. 2007 Feb;6(1):19-24
references: 17342215
references: JAMA. 1998 Jul 15;280(3):246-7
references: 9676670
references: PLoS Med. 2005 Aug;2(8):e124
references: 16060722
references: J Clin Epidemiol. 2005 Jun;58(6):543-9
references: 15878467
references: Nature. 2011 Feb 24;470(7335):437
references: 21350443
references: Psychon Bull Rev. 2012 Apr;19(2):151-6
references: 22351589
references: Psychol Med. 2008 Aug;38(8):1213-4
references: 18598581
references: PLoS One. 2013 Jun 26;8(6):e66938
references: 23840557
references: Neurologia. 2011 Jan-Feb;26(1):1-5
references: 21216501
references: Psychol Sci. 2012 May 1;23(5):524-32
references: 22508865
references: J Eval Clin Pract. 2008 Oct;14(5):951-7
references: 19018930
references: PLoS One. 2013;8(1):e53548
references: 23372659
references: Proc Natl Acad Sci U S A. 2012 Oct 16;109(42):17028-33
references: 23027971
references: PLoS One. 2010 Apr 07;5(4):e10068
references: 20383332
references: Perspect Psychol Sci. 2012 Nov;7(6):670-88
references: 26168129
references: PLoS One. 2009 May 29;4(5):e5738
references: 19478950