title: A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory.
abstract: Amazon's Mechanical Turk (AMT) is a Web application that provides instant access to thousands of potential participants for survey-based psychology experiments, such as the acceptability judgment task used extensively in syntactic theory. Because AMT is a Web-based system, syntacticians may worry that the move out of the experimenter-controlled environment of the laboratory and onto the user-controlled environment of AMT could adversely affect the quality of the judgment data collected. This article reports a quantitative comparison of two identical acceptability judgment experiments, each with 176 participants (352 total): one conducted in the laboratory, and one conducted on AMT. Crucial indicators of data quality--such as participant rejection rates, statistical power, and the shape of the distributions of the judgments for each sentence type--are compared between the two samples. The results suggest that aside from slightly higher participant rejection rates, AMT data are almost indistinguishable from laboratory data.
authors.affiliation: Department of Cognitive Sciences, University of California, 3151 Social Science Plaza A, Irvine, CA 92697-5100, USA. jsprouse@uci.edu
authors.forename: Jon
authors.initials: J
authors.lastname: Sprouse
mesh_terms: D003365:Costs and Cost Analysis; D003625:Data Collection; D003627:Data Interpretation, Statistical; D004739:England; D005260:Female; D006801:Humans; D007088:Illusions / Q000523:psychology; D020407:Internet; D007600:Judgment; D007802:Language; D008037:Linguistics / Q000379:methods / Q000706:statistics & numerical data; D008297:Male; D011578:Psycholinguistics / Q000379:methods* / Q000706:statistics & numerical data*; D011597:Psychomotor Performance / Q000502:physiology; D015203:Reproducibility of Results; D012984:Software; D055815:Young Adult