title: Better P-curves: Making P-curve analysis more robust to errors, fraud, and ambitious P-hacking, a Reply to Ulrich and Miller (2015).
abstract: When studies examine true effects, they generate right-skewed p-curves, distributions of statistically significant results with more low (.01 s) than high (.04 s) p values. What else can cause a right-skewed p-curve? First, we consider the possibility that researchers report only the smallest significant p value (as conjectured by Ulrich & Miller, 2015), concluding that it is a very uncommon problem. We then consider more common problems, including (a) p-curvers selecting the wrong p values, (b) fake data, (c) honest errors, and (d) ambitiously p-hacked (beyond p < .05) results. We evaluate the impact of these common problems on the validity of p-curve analysis, and provide practical solutions that substantially increase its robustness.
authors.affiliation: Wharton School, University of Pennsylvania.
authors.forename: Uri
authors.initials: U
authors.lastname: Simonsohn
authors.affiliation: Wharton School, University of Pennsylvania.
authors.forename: Joseph P
authors.initials: JP
authors.lastname: Simmons
authors.affiliation: Haas School of Business, University of California, Berkeley.
authors.forename: Leif D
authors.initials: LD
authors.lastname: Nelson
mesh_terms: D003627:Data Interpretation, Statistical; D006801:Humans; D017594:Publication Bias / Q000706:statistics & numerical data*; D012107:Research Design / Q000706:statistics & numerical data*; D015871:Scientific Misconduct / Q000706:statistics & numerical data*