title: The Search for Significance: A Few Peculiarities in the Distribution of P Values in Experimental Psychology Literature.
abstract: In this project I investigate the use and possible misuse of p values in papers published in five (high-ranked) journals in experimental psychology. I use a data set of over 135'000 p values from more than five thousand papers. I inspect (1) the way in which the p values are reported and (2) their distribution. The main findings are following: first, it appears that some authors choose the mode of reporting their results in an arbitrary way. Moreover, they often end up doing it in such a way that makes their findings seem more statistically significant than they really are (which is well known to improve the chances for publication). Specifically, they frequently report p values "just above" significance thresholds directly, whereas other values are reported by means of inequalities (e.g. "p<.1"), they round the p values down more eagerly than up and appear to choose between the significance thresholds and between one- and two-sided tests only after seeing the data. Further, about 9.2% of reported p values are inconsistent with their underlying statistics (e.g. F or t) and it appears that there are "too many" "just significant" values. One interpretation of this is that researchers tend to choose the model or include/discard observations to bring the p value to the right side of the threshold.
authors.affiliation: University of Warsaw, Dluga 44/50, 00-241, Warsaw, Poland.
authors.forename: Michał
authors.initials: M
authors.lastname: Krawczyk
mesh_terms: D003627:Data Interpretation, Statistical; D006801:Humans; D011336:Probability; D011589:Psychology, Experimental / Q000379:methods / Q000706:statistics & numerical data*; D017594:Publication Bias / Q000706:statistics & numerical data*; D011642:Publications / Q000706:statistics & numerical data