A comment on replication, p-values and evidence.
It is conventionally thought that a small p-value confers high credibility on the observed alternative hypothesis, and that a repetition of the same experiment will have a high probability of resulting again in statistical significance. It is shown that if the observed difference is the true one, the probability of repeating a statistically significant result, the 'replication probability', is substantially lower than expected. The reason for this is a mistake that generates other seeming paradoxes: the interpretation of the post-trial p-value in the same way as the pre-trial alpha error. The replication probability can be used as a frequentist counterpart of Bayesian and likelihood methods to show that p-values overstate the evidence against the null hypothesis.
Goodman
S N
SN
Johns Hopkins University School of Medicine, Department of Oncology, Baltimore, MD 21205.
D015982:Bias; D006801:Humans; D011336:Probability; D015203:Reproducibility of Results