Conducting Meta-Analyses Based on p Values: Reservations and Recommendations for Applying p-Uniform and p-Curve.
Because of overwhelming evidence of publication bias in psychology, techniques to correct meta-analytic estimates for such bias are greatly needed. The methodology on which the p-uniform and p-curve methods are based has great promise for providing accurate meta-analytic estimates in the presence of publication bias. However, in this article, we show that in some situations, p-curve behaves erratically, whereas p-uniform may yield implausible estimates of negative effect size. Moreover, we show that (and explain why) p-curve and p-uniform result in overestimation of effect size under moderate-to-large heterogeneity and may yield unpredictable bias when researchers employ p-hacking. We offer hands-on recommendations on applying and interpreting results of meta-analyses in general and p-uniform and p-curve in particular. Both methods as well as traditional methods are applied to a meta-analysis on the effect of weight on judgments of importance. We offer guidance for applying p-uniform or p-curve using R and a user-friendly web application for applying p-uniform.
Department of Methodology and Statistics, Tilburg University R.C.M.vanAert@tilburguniversity.edu.
Robbie C M
RC
van Aert
Department of Methodology and Statistics, Tilburg University.
Jelte M
JM
Wicherts
Department of Methodology and Statistics, Tilburg University Department of Social and Behavioral Sciences, Utrecht University.
Marcel A L M
MA
van Assen
D003627:Data Interpretation, Statistical; D006801:Humans; D020407:Internet; D015201:Meta-Analysis as Topic; D015233:Models, Statistical; D011584:Psychology / Q000379:methods; D017594:Publication Bias; D012984:Software
heterogeneity; meta-analysis; p-curve; p-hacking; p-uniform
Biometrics. 1997 Mar;53(1):11-22
9147587
Behav Res Methods. 2016 Dec;48(4):1205-1226
26497820
Stat Med. 2002 Jun 15;21(11):1539-58
12111919
Q J Exp Psychol (Hove). 2015;68(4):829-32
25484109
Perspect Psychol Sci. 2012 Nov;7(6):543-54
26168111
Psychol Sci. 2009 Sep;20(9):1169-74
19686292
Science. 2010 Jun 25;328(5986):1712-5
20576894
J Exp Psychol Gen. 2012 May;141(2):211-6
21767044
Br J Math Stat Psychol. 2010 Nov;63(Pt 3):665-94
20497626
BMJ. 2009 Aug 07;339:b2981
19666685
Psychol Methods. 2012 Mar;17(1):120-8
21787082
Psychol Methods. 2015 Sep;20(3):293-309
25401773
Psychol Sci. 2011 Nov;22(11):1359-66
22006061
Psychon Bull Rev. 2016 Feb;23(1):74-86
26126776
J Exp Psychol Gen. 2015 Dec;144(6):1137-45
26595841
J Exp Psychol Gen. 2014 Feb;143(1):414-24
23398182
PLoS One. 2013 Oct 01;8(10):e76010
24098419
Perspect Psychol Sci. 2014 Nov;9(6):666-81
26186117
PLoS One. 2016 Feb 17;11(2):e0149144
26886098
Psychol Sci. 2012 May 1;23(5):524-32
22508865
PLoS One. 2014 Jul 29;9(7):e103360
25072606
Res Synth Methods. 2014 Mar;5(1):60-78
26054026
PeerJ. 2016 Apr 11;4:e1935
27077017
PLoS One. 2013 Nov 11;8(11):e78307
24244302
PLoS One. 2015 Aug 04;10(8):e0134808
26241042
J Exp Psychol Gen. 2014 Apr;143(2):534-47
23855496
BMC Med Res Methodol. 2009 Jan 12;9:2
19138428
Br J Math Stat Psychol. 2009 Feb;62(Pt 1):97-128
18001516
Psychol Methods. 2014 Sep;19(3):409-27
24773354
J Clin Epidemiol. 2006 Oct;59(10):1023-32
16980141
Exp Psychol. 2013;60(4):255-9
23548985
Epidemiology. 2008 Sep;19(5):640-8
18633328