title: Replication is not coincidence: reply to Iverson, Lee, and Wagenmakers (2009).
abstract: Iverson, Lee, and Wagenmakers (2009) claimed that Killeen's (2005) statistic prep overestimates the "true probability of replication." We show that Iverson et al. confused the probability of replication of an observed direction of effect with a probability of coincidence--the probability that two future experiments will return the same sign. The theoretical analysis is punctuated with a simulation of the predictions of prep for a realistic random effects world of representative parameters, when those are unknown a priori. We emphasize throughout that prep is intended to evaluate the probability of a replication outcome after observations, not to estimate a parameter. Hence, the usual conventional criteria (unbiasedness, minimum variance estimator) for judging estimators are not appropriate for probabilities such as p and prep.
authors.affiliation: CNRS and Université de Rouen, Rouen, France.
authors.forename: Bruno
authors.initials: B
authors.lastname: Lecoutre
authors.forename: Peter R
authors.initials: PR
authors.lastname: Killeen
mesh_terms: D015982:Bias; D003627:Data Interpretation, Statistical; D006801:Humans; D011336:Probability; D011589:Psychology, Experimental / Q000706:statistics & numerical data*; D015203:Reproducibility of Results
references.citation: Psychol Methods. 2010 Jun;15(2):158-71
references.pmid: 20515237
references.citation: Psychon Bull Rev. 2009 Apr;16(2):424-9
references.pmid: 19293117
references.citation: Psychon Bull Rev. 2008 Feb;15(1):16-27
references.pmid: 18605475
references.citation: Perspect Psychol Sci. 2008 Jul;3(4):286-300
references.pmid: 26158948
references.citation: Psychon Bull Rev. 2006 Aug;13(4):549-62
references.pmid: 17201351
references.citation: Psychol Sci. 2005 Dec;16(12):1009-12
references.pmid: 16313669
references.citation: Br J Math Stat Psychol. 2009 Nov;62(Pt 3):485-7
references.pmid: 19017433
references.citation: Psychon Bull Rev. 2009 Aug;16(4):617-40
references.pmid: 19648445