Replication is not coincidence: reply to Iverson, Lee, and Wagenmakers (2009).
Iverson, Lee, and Wagenmakers (2009) claimed that Killeen's (2005) statistic prep overestimates the "true probability of replication." We show that Iverson et al. confused the probability of replication of an observed direction of effect with a probability of coincidence--the probability that two future experiments will return the same sign. The theoretical analysis is punctuated with a simulation of the predictions of prep for a realistic random effects world of representative parameters, when those are unknown a priori. We emphasize throughout that prep is intended to evaluate the probability of a replication outcome after observations, not to estimate a parameter. Hence, the usual conventional criteria (unbiasedness, minimum variance estimator) for judging estimators are not appropriate for probabilities such as p and prep.
Lecoutre
Bruno
B
CNRS and Université de Rouen, Rouen, France.
Killeen
Peter R
PR
D015982:Bias; D003627:Data Interpretation, Statistical; D006801:Humans; D011336:Probability; D011589:Psychology, Experimental / Q000706:statistics & numerical data*; D015203:Reproducibility of Results
Psychol Methods. 2010 Jun;15(2):158-71
20515237
Psychon Bull Rev. 2009 Apr;16(2):424-9
19293117
Psychon Bull Rev. 2008 Feb;15(1):16-27
18605475
Perspect Psychol Sci. 2008 Jul;3(4):286-300
26158948
Psychon Bull Rev. 2006 Aug;13(4):549-62
17201351
Psychol Sci. 2005 Dec;16(12):1009-12
16313669
Br J Math Stat Psychol. 2009 Nov;62(Pt 3):485-7
19017433
Psychon Bull Rev. 2009 Aug;16(4):617-40
19648445