title: Using prediction markets to estimate the reproducibility of scientific research.
abstract: Concerns about a lack of reproducibility of statistically significant results have recently been raised in many fields, and it has been argued that this lack comes at substantial economic costs. We here report the results from prediction markets set up to quantify the reproducibility of 44 studies published in prominent psychology journals and replicated in the Reproducibility Project: Psychology. The prediction markets predict the outcomes of the replications well and outperform a survey of market participants' individual forecasts. This shows that prediction markets are a promising tool for assessing the reproducibility of published scientific results. The prediction markets also allow us to estimate probabilities for the hypotheses being true at different testing stages, which provides valuable information regarding the temporal dynamics of scientific discovery. We find that the hypotheses being tested in psychology typically have low prior probabilities of being true (median, 9%) and that a "statistically significant" finding needs to be confirmed in a well-powered replication to have a high probability of being true. We argue that prediction markets could be used to obtain speedy information about reproducibility at low cost and could potentially even be used to determine which studies to replicate to optimally allocate limited resources into replications.
authors.affiliation: Department of Economics, Stockholm School of Economics, SE-113 83 Stockholm, Sweden; anna.dreber@hhs.se.
authors.forename: Anna
authors.initials: A
authors.lastname: Dreber
authors.affiliation: New Zealand Institute for Advanced Study, Massey University, Auckland 0745, New Zealand; Wissenschaftskolleg zu Berlin-Institute for Advanced Study, D-14193 Berlin, Germany;
authors.forename: Thomas
authors.initials: T
authors.lastname: Pfeiffer
authors.affiliation: Sveriges Riksbank, SE-103 37 Stockholm, Sweden;
authors.forename: Johan
authors.initials: J
authors.lastname: Almenberg
authors.affiliation: Department of Economics, Stockholm School of Economics, SE-113 83 Stockholm, Sweden;
authors.forename: Siri
authors.initials: S
authors.lastname: Isaksson
authors.affiliation: Consensus Point, Nashville, TN 37203;
authors.forename: Brad
authors.initials: B
authors.lastname: Wilson
authors.affiliation: John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138;
authors.forename: Yiling
authors.initials: Y
authors.lastname: Chen
authors.affiliation: Department of Psychology, University of Virginia, Charlottesville, VA 22904; Center for Open Science, Charlottesville, VA 22903.
authors.forename: Brian A
authors.initials: BA
authors.lastname: Nosek
authors.affiliation: Department of Economics, Stockholm School of Economics, SE-113 83 Stockholm, Sweden;
authors.forename: Magnus
authors.initials: M
authors.lastname: Johannesson
mesh_terms: D003132:Commerce; D005544:Forecasting; D011336:Probability; D015203:Reproducibility of Results; D012106:Research; D012586:Science; D011795:Surveys and Questionnaires
keywords: prediction markets; replications; reproducibility