title: Could millisecond timing errors in commonly used equipment be a cause of replication failure in some neuroscience studies?
abstract: Neuroscience is a rapidly expanding field in which complex studies and equipment setups are the norm. Often these push boundaries in terms of what technology can offer, and increasingly they make use of a wide range of stimulus materials and interconnected equipment (e.g., magnetic resonance imaging, electroencephalography, magnetoencephalography, eyetrackers, biofeedback, etc.). The software that bonds the various constituent parts together itself allows for ever more elaborate investigations to be carried out with apparent ease. However, research over the last decade has suggested a growing, yet underacknowledged, problem with obtaining millisecond-accurate timing in some computer-based studies. Crucially, timing inaccuracies can affect not just response time measurements, but also stimulus presentation and the synchronization between equipment. This is not a new problem, but rather one that researchers may have assumed had been solved with the advent of faster computers, state-of-the-art equipment, and more advanced software. In this article, we highlight the potential sources of error, their causes, and their likely impact on replication. Unfortunately, in many applications, inaccurate timing is not easily resolved by utilizing ever-faster computers, newer equipment, or post-hoc statistical manipulation. To ensure consistency across the field, we advocate that researchers self-validate the timing accuracy of their own equipment whilst running the actual paradigm in situ.
authors: Black Box ToolKit Ltd, P.O. Box 3802, Sheffield, S25 9AG, UK, r.plant@blackboxtoolkit.com.
authors: Richard R
authors: RR
authors: Plant
authors: Philip T
authors: PT
authors: Quinlan
mesh_terms: D003201:Computers; D004569:Electroencephalography / Q000379:methods; D008279:Magnetic Resonance Imaging / Q000379:methods; D009488:Neurosciences / Q000295:instrumentation / Q000379:methods*; D011930:Reaction Time; D012984:Software; D013997:Time Factors
references: Behav Res Methods Instrum Comput. 2003 May;35(2):276-84
references: 12834085
references: Exp Brain Res. 2012 Feb;216(3):457-62
references: 22105336
references: Behav Res Methods. 2005 Feb;37(1):139-47
references: 16097354
references: Behav Res Methods Instrum Comput. 2004 May;36(2):291-303
references: 15354695
references: Behav Res Methods. 2008 Nov;40(4):1170-6
references: 19001409
references: Vision Res. 2011 Sep 1;51(17):1966-71
references: 21787801
references: Perspect Psychol Sci. 2012 Nov;7(6):528-30
references: 26168108
references: NMR Biomed. 2002 Nov-Dec;15(7-8):456-67
references: 12489095
references: Behav Res Methods. 2007 Aug;39(3):365-70
references: 17958146
references: Infancy. 2012 Jan;17(1):9-32
references: 32693503
references: Behav Res Methods. 2011 Jun;43(2):353-62
references: 21416303
references: Behav Res Methods Instrum Comput. 2002 May;34(2):218-26
references: 12109015
references: Behav Res Methods. 2010 Feb;42(1):205-11
references: 20160300
references: Behav Res Methods. 2009 Aug;41(3):598-614
references: 19587169