P(rep) misestimates the probability of replication.
The probability of "replication," P(rep), has been proposed as a means of identifying replicable and reliable effects in the psychological sciences. We conduct a basic test of P(rep) that reveals that it misestimates the true probability of replication, especially for small effects. We show how these general problems with P(rep) play out in practice, when it is applied to predict the replicability of observed effects over a series of experiments. Our results show that, over any plausible series of experiments, the true probabilities of replication will be very different from those predicted by P(rep). We discuss some basic problems in the formulation of P(rep) that are responsible for its poor performance, and conclude that P(rep) is not a useful statistic for psychological science.
Iverson
Geoffrey J
GJ
University of California, Irvine, California, USA.
Lee
Michael D
MD
Wagenmakers
Eric-Jan
EJ
D015982:Bias; D003627:Data Interpretation, Statistical; D006801:Humans; D011336:Probability; D011589:Psychology, Experimental / Q000706:statistics & numerical data*; D015203:Reproducibility of Results; D013223:Statistics as Topic