What is the probability of replicating a statistically significant effect?
If an initial experiment produces a statistically significant effect, what is the probability that this effect will be replicated in a follow-up experiment? I argue that this seemingly fundamental question can be interpreted in two very different ways and that its answer is, in practice, virtually unknowable under either interpretation. Although the data from an initial experiment can be used to estimate one type of replication probability, this estimate will rarely be precise enough to be of any use. The other type of replication probability is also unknowable, because it depends on unknown aspects of the research context. Thus, although it would be nice to know the probability of replicating a significant effect, researchers must accept the fact that they generally cannot determine this information, whichever type of replication probability they seek.
Miller
Jeff
J
University of Otago, Dunedin, New Zealand. miller@psy.otago.ac.nz
D003625:Data Collection / Q000706:statistics & numerical data; D003627:Data Interpretation, Statistical; D006801:Humans; D011336:Probability; D011589:Psychology, Experimental / Q000706:statistics & numerical data*; D015203:Reproducibility of Results; D012106:Research / Q000706:statistics & numerical data*