title: Collecting psycholinguistic response time data using Amazon mechanical Turk.
abstract: Researchers in linguistics and related fields have recently begun exploiting online crowd-sourcing tools, like Amazon Mechanical Turk (AMT), to gather behavioral data. While this method has been successfully validated for various offline measures--grammaticality judgment or other forced-choice tasks--its use for mainstream psycholinguistic research remains limited. This is because psycholinguistic effects are often dependent on relatively small differences in response times, and there remains some doubt as to whether precise timing measurements can be gathered over the web. Here we show that three classic psycholinguistic effects can in fact be replicated using AMT in combination with open-source software for gathering response times client-side. Specifically, we find reliable effects of subject definiteness, filler-gap dependency processing, and agreement attraction in self-paced reading tasks using approximately the same numbers of participants and/or trials as similar laboratory studies. Our results suggest that psycholinguists can and should be taking advantage of AMT and similar online crowd-sourcing marketplaces as a fast, low-resource alternative to traditional laboratory research.
authors.affiliation: Linguistics Program, Department of English, George Mason University, Fairfax, Virginia, United States of America.
authors.forename: Kelly
authors.initials: K
authors.lastname: Enochson
authors.affiliation: Linguistics Program, Department of English, George Mason University, Fairfax, Virginia, United States of America; School of Philosophy, Psychology and Language Sciences, University of Edinburgh, Edinburgh, United Kingdom.
authors.forename: Jennifer
authors.initials: J
authors.lastname: Culbertson
mesh_terms: D000328:Adult; D063045:Crowdsourcing / Q000379:methods*; D006801:Humans; D020407:Internet; D011578:Psycholinguistics; D011930:Reaction Time; D055815:Young Adult
references.citation: PLoS One. 2013 Jun 21;8(6):e67769
references.pmid: 23805326
references.citation: Cognition. 1998 Aug;68(1):B13-29
references.pmid: 9775519
references.citation: Cogn Psychol. 1991 Jan;23(1):45-93
references.pmid: 2001615
references.citation: Psychon Bull Rev. 2012 Oct;19(5):847-57
references.pmid: 22829343
references.citation: Cognition. 2002 Aug;85(1):79-112
references.pmid: 12086714
references.citation: PLoS One. 2013;8(3):e57410
references.pmid: 23516406
references.citation: Proc Natl Acad Sci U S A. 2014 Apr 22;111(16):5842-7
references.pmid: 24706789
references.citation: Cogn Sci. 2013 Apr;37(3):578-91
references.pmid: 23363004
references.citation: J Mem Lang. 2013 Apr;68(3):null
references.pmid: 24403724
references.citation: Behav Res Methods. 2011 Mar;43(1):155-67
references.pmid: 21287108
references.citation: Behav Res Methods. 2009 Feb;41(1):1-12
references.pmid: 19182118
references.citation: Cogn Psychol. 2004 Aug;49(1):1-46
references.pmid: 15193971
references.citation: Language (Baltim). 2010 Jun;86(2):366-415
references.pmid: 22661792
references.citation: PLoS One. 2013;8(10):e77661
references.pmid: 24204909
references.citation: Behav Brain Sci. 2010 Jun;33(2-3):61-83; discussion 83-135
references.pmid: 20550733