title: Promises and pitfalls of Web-based experimentation in the advance of replicable psychological science: A reply to Plant (2015).
abstract: In a recent letter, Plant (2015) reminded us that proper calibration of our laboratory experiments is important for the progress of psychological science. Therefore, carefully controlled laboratory studies are argued to be preferred over Web-based experimentation, in which timing is usually more imprecise. Here we argue that there are many situations in which the timing of Web-based experimentation is acceptable and that online experimentation provides a very useful and promising complementary toolbox to available lab-based approaches. We discuss examples in which stimulus calibration or calibration against response criteria is necessary and situations in which this is not critical. We also discuss how online labor markets, such as Amazon's Mechanical Turk, allow researchers to acquire data in more diverse populations and to test theories along more psychological dimensions. Recent methodological advances that have produced more accurate browser-based stimulus presentation are also discussed. In our view, online experimentation is one of the most promising avenues to advance replicable psychological science in the near future.
authors.affiliation: Institute of Psychology, Leiden University, Leiden, The Netherlands. HvanSteenbergen@fsw.leidenuniv.nl.|, Wassenaarseweg 52, 2333 AK, Leiden, The Netherlands. HvanSteenbergen@fsw.leidenuniv.nl.|Leiden Institute for Brain and Cognition, Leiden University, Leiden, The Netherlands. HvanSteenbergen@fsw.leidenuniv.nl.
authors.forename: Henk
authors.initials: H
authors.lastname: van Steenbergen
authors.affiliation: Institute of Psychology, Leiden University, Leiden, The Netherlands.|Leiden Institute for Brain and Cognition, Leiden University, Leiden, The Netherlands.
authors.forename: Bruno R
authors.initials: BR
authors.lastname: Bocanegra
mesh_terms: D035841:Behavioral Research; D006801:Humans; D020407:Internet; D011584:Psychology; D012107:Research Design
keywords: Critical timing; Online experimentation; Psychological science; Qualtrics Reaction Time Engine (QRTEngine); Reaction times; Replication; Web-based psychology experiments