title: Reanalyzing Head et al. (2015): investigating the robustness of widespreadp-hacking.
abstract: Head et al. (2015) provided a large collection ofp-values that, from their perspective, indicates widespread statistical significance seeking (i.e.,p-hacking). This paper inspects this result for robustness. Theoretically, thep-value distribution should be a smooth, decreasing function, but the distribution of reportedp-values shows systematically more reportedp-values for .01, .02, .03, .04, and .05 thanp-values reported to three decimal places, due to apparent tendencies to roundp-values to two decimal places. Head et al. (2015) correctly argue that an aggregatep-value distribution could show a bump below .05 when left-skewp-hacking occurs frequently. Moreover, the elimination ofp= .045 andp= .05, as done in the original paper, is debatable. Given that eliminatingp= .045 is a result of the need for symmetric bins and systematically morep-values are reported to two decimal places than to three decimal places, I did not excludep= .045 andp= .05. I conducted Fisher's method .04 <p< .05 and reanalyzed the data by adjusting the bin selection to .03875 <p≤ .04 versus .04875 <p≤ .05. Results of the reanalysis indicate that no evidence for left-skewp-hacking remains when we look at the entire range between .04 <p< .05 or when we inspect the second-decimal. Taking into account reporting tendencies when selecting the bins to compare is especially important because this dataset does not allow for the recalculation of thep-values. Moreover, inspecting the bins that include two-decimal reportedp-values potentially increases sensitivity if strategic rounding down ofp-values as a form ofp-hacking is widespread. Given the far-reaching implications of supposed widespreadp-hacking throughout the sciences Head et al. (2015), it is important that these findings are robust to data analysis choices if the conclusion is to be considered unequivocal. Although no evidence of widespread left-skewp-hacking is found in this reanalysis, this does not mean that there is nop-hacking at all. These results nuance the conclusion by Head et al. (2015), indicating that the results are not robust and that the evidence for widespread left-skewp-hacking is ambiguous at best.
authors.affiliation: Department of Methodology and Statistics, Tilburg University , Tilburg , The Netherlands.
authors.forename: Chris H J
authors.initials: CH
authors.lastname: Hartgerink
keywords: Nhst; P-hacking; P-values; Qrps; Questionable research practices; Reanalysis