title: Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants.
abstract: Participant attentiveness is a concern for many researchers using Amazon's Mechanical Turk (MTurk). Although studies comparing the attentiveness of participants on MTurk versus traditional subject pool samples have provided mixed support for this concern, attention check questions and other methods of ensuring participant attention have become prolific in MTurk studies. Because MTurk is a population that learns, we hypothesized that MTurkers would be more attentive to instructions than are traditional subject pool samples. In three online studies, participants from MTurk and collegiate populations participated in a task that included a measure of attentiveness to instructions (an instructional manipulation check: IMC). In all studies, MTurkers were more attentive to the instructions than were college students, even on novel IMCs (Studies 2 and 3), and MTurkers showed larger effects in response to a minute text manipulation. These results have implications for the sustainable use of MTurk samples for social science research and for the conclusions drawn from research with MTurk and college subject pool samples.
authors.affiliation: Department of Psychology, University of Michigan, 3233 East Hall, 530 Church Street, Ann Arbor, MI, 48109-1043, USA. djhauser@umich.edu.
authors.forename: David J
authors.initials: DJ
authors.lastname: Hauser
authors.affiliation: University of Southern California, Los Angeles, CA, USA.
authors.forename: Norbert
authors.initials: N
authors.lastname: Schwarz
mesh_terms: D001288:Attention; D003625:Data Collection / Q000379:methods*; D005260:Female; D006801:Humans; D020407:Internet; D007858:Learning; D008297:Male; D012942:Social Sciences; D011795:Surveys and Questionnaires
keywords: College students; Instructional manipulation checks; MTurk; Participant attentiveness