title: A comment on replication, p-values and evidence.
abstract: It is conventionally thought that a small p-value confers high credibility on the observed alternative hypothesis, and that a repetition of the same experiment will have a high probability of resulting again in statistical significance. It is shown that if the observed difference is the true one, the probability of repeating a statistically significant result, the 'replication probability', is substantially lower than expected. The reason for this is a mistake that generates other seeming paradoxes: the interpretation of the post-trial p-value in the same way as the pre-trial alpha error. The replication probability can be used as a frequentist counterpart of Bayesian and likelihood methods to show that p-values overstate the evidence against the null hypothesis.
authors.affiliation: Johns Hopkins University School of Medicine, Department of Oncology, Baltimore, MD 21205.
authors.forename: S N
authors.initials: SN
authors.lastname: Goodman
mesh_terms: D015982:Bias; D006801:Humans; D011336:Probability; D015203:Reproducibility of Results