title: A surge of p-values between 0.041 and 0.049 in recent decades (but negative results are increasing rapidly too).
abstract: It is known that statistically significant (positive) results are more likely to be published than non-significant (negative) results. However, it has been unclear whether any increasing prevalence of positive results is stronger in the "softer" disciplines (social sciences) than in the "harder" disciplines (physical sciences), and whether the prevalence of negative results is decreasing over time. Using Scopus, we searched the abstracts of papers published between 1990 and 2013, and measured longitudinal trends of multiple expressions of positive versus negative results, including p-values between 0.041 and 0.049 versus p-values between 0.051 and 0.059, textual reporting of "significant difference" versus "no significant difference," and the reporting of p < 0.05 versus p > 0.05. We found no support for a "hierarchy of sciences" with physical sciences at the top and social sciences at the bottom. However, we found large differences in reporting practices between disciplines, with p-values between 0.041 and 0.049 over 1990-2013 being 65.7 times more prevalent in the biological sciences than in the physical sciences. The p-values near the significance threshold of 0.05 on either side have both increased but with those p-values between 0.041 and 0.049 having increased to a greater extent (2013-to-1990 ratio of the percentage of papers = 10.3) than those between 0.051 and 0.059 (ratio = 3.6). Contradictorily, p < 0.05 has increased more slowly than p > 0.05 (ratios = 1.4 and 4.8, respectively), while the use of "significant difference" has shown only a modest increase compared to "no significant difference" (ratios = 1.5 and 1.1, respectively). We also compared reporting of significance in the United States, Asia, and Europe and found that the results are too inconsistent to draw conclusions on cross-cultural differences in significance reporting. We argue that the observed longitudinal trends are caused by negative factors, such as an increase of questionable research practices, but also by positive factors, such as an increase of quantitative research and structured reporting.
authors: Department of BioMechanical Engineering, Delft University of Technology , Delft , The Netherlands.
authors: Joost Cf
authors: JC
authors: de Winter
authors: Department of BioMechanical Engineering, Delft University of Technology , Delft , The Netherlands.
authors: Dimitra
authors: D
authors: Dodou
keywords: Bias; Biological sciences; Physical sciences; Science policy; Significant differences; Social sciences
references: J Evol Biol. 2007 May;20(3):1082-9
references: 17465918
references: BMJ. 2002 Dec 21;325(7378):1450-1
references: 12493662
references: Biostatistics. 2014 Jan;15(1):1-12
references: 24068246
references: Proc Natl Acad Sci U S A. 2013 Sep 10;110(37):15031-6
references: 23980165
references: Q J Exp Psychol (Hove). 2015;68(4):829-32
references: 25484109
references: J Pers Soc Psychol. 2015 Feb;108(2):275-97
references: 25603376
references: Biostatistics. 2014 Jan;15(1):23-7; discussion 39-45
references: 24068250
references: Trends Mol Med. 2003 Apr;9(4):135-8
references: 12727138
references: Science. 2014 Sep 19;345(6203):1502-5
references: 25170047
references: Psychon Bull Rev. 2007 Oct;14(5):779-804
references: 18087943
references: Proc Natl Acad Sci U S A. 2014 Feb 18;111(7):E712-3
references: 24497511
references: Lancet. 2000 Aug 19;356(9230):635-8
references: 10968436
references: Biol Rev Camb Philos Soc. 2002 May;77(2):211-22
references: 12056747
references: N Engl J Med. 2004 Sep 16;351(12):1250-1
references: 15356289
references: J Chronic Dis. 1979;32(1-2):51-63
references: 447779
references: BMC Psychol. 2013 Feb 27;1(1):2
references: 25566354
references: PLoS Med. 2008 Oct 7;5(10):e201
references: 18844432
references: Eur J Cancer. 2007 Nov;43(17):2559-79
references: 17981458
references: J Psychiatry Neurosci. 2012 May;37(3):149-52
references: 22515987
references: PLoS Med. 2005 Aug;2(8):e124
references: 16060722
references: Biometrics. 2000 Jun;56(2):455-63
references: 10877304
references: Psychol Sci. 2011 Nov;22(11):1359-66
references: 22006061
references: Stat Med. 2008 Feb 28;27(5):746-63
references: 17592831
references: PLoS Med. 2005 Dec;2(12):e334
references: 16285839
references: Behav Res Methods. 2006 Feb;38(1):24-7
references: 16817510
references: BMJ. 2010 Feb 15;340:c365
references: 20156912
references: PLoS One. 2008 Aug 28;3(8):e3081
references: 18769481
references: Q J Exp Psychol (Hove). 2012;65(11):2271-9
references: 22853650
references: Biostatistics. 2014 Jan;15(1):13-6; discussion 39-45
references: 24068247
references: PLoS One. 2010 Apr 07;5(4):e10068
references: 20383332
references: Am J Med Genet B Neuropsychiatr Genet. 2008 Sep 5;147B(6):964-72
references: 18361430
references: Neuroinformatics. 2012 Jan;10(1):67-80
references: 21643733
references: BMJ. 2006 Jul 29;333(7561):231-4
references: 16854948
references: Psychol Sci. 2012;23(11):1314-23
references: 23012269
references: Perspect Psychol Sci. 2012 Nov;7(6):555-61
references: 26168112
references: J Cereb Blood Flow Metab. 2010 Jul;30(7):1263-4
references: 20596038
references: PLoS One. 2012;7(10):e46363
references: 23056287
references: Am Psychol. 2005 Feb-Mar;60(2):170-80
references: 15740449
references: Control Clin Trials. 1998 Apr;19(2):159-66
references: 9551280
references: J Clin Epidemiol. 2000 Feb;53(2):207-16
references: 10729693
references: BMJ. 2003 May 31;326(7400):1167-70
references: 12775614
references: J Exp Psychol Gen. 2014 Apr;143(2):534-47
references: 23855496
references: Q J Exp Psychol (Hove). 2013;66(12):2303-9
references: 24205936
references: Perspect Psychol Sci. 2012 Nov;7(6):531-6
references: 26168109
references: PLoS One. 2014 Sep 05;9(9):e105825
references: 25192357
references: Contemp Clin Trials. 2008 Mar;29(2):109-13
references: 17919992
references: Scientometrics. 2010 Sep;84(3):575-603
references: 20700371
references: Cochrane Database Syst Rev. 2009 Jan 21;(1):MR000006
references: 19160345
references: Psychon Bull Rev. 2014 Oct;21(5):1180-7
references: 24638826
references: Biostatistics. 2014 Jan;15(1):28-36; discussion 39-45
references: 24068251
references: Trends Cogn Sci. 2014 May;18(5):235-41
references: 24656991
references: JAMA. 2010 May 26;303(20):2058-64
references: 20501928
references: Psychol Methods. 2014 Sep;19(3):409-27
references: 24773354
references: Science. 2011 Jan 14;331(6014):176-82
references: 21163965
references: Nature. 2012 May 16;485(7398):298-300
references: 22596136
references: Perspect Psychol Sci. 2012 Nov;7(6):661-9
references: 26168128
references: PLoS One. 2009 May 29;4(5):e5738
references: 19478950
references: BMJ. 2004 Oct 16;329(7471):883
references: 15469946
references: Nat Genet. 2010 Jul;42(7):558-60
references: 20581876
references: Stat Med. 2003 Jul 15;22(13):2113-26
references: 12820277
references: Stat Med. 2002 Apr 15;21(7):985-94; discussion 995-999, 1001
references: 11921006
references: J Med Libr Assoc. 2011 Apr;99(2):160-3
references: 21464855
references: Lancet. 2014 Jan 18;383(9913):257-66
references: 24411650