title: Participation and contribution in crowdsourced surveys.
abstract: This paper identifies trends within and relationships between the amount of participation and the quality of contributions in three crowdsourced surveys. Participants were asked to perform a collective problem solving task that lacked any explicit incentive: they were instructed not only to respond to survey questions but also to pose new questions that they thought might-if responded to by others-predict an outcome variable of interest to them. While the three surveys had very different outcome variables, target audiences, methods of advertisement, and lengths of deployment, we found very similar patterns of collective behavior. In particular, we found that: the rate at which participants submitted new survey questions followed a heavy-tailed distribution; the distribution in the types of questions posed was similar; and many users posed non-obvious yet predictive questions. By analyzing responses to questions that contained a built-in range of valid response we found that less than 0.2% of responses lay outside of those ranges, indicating that most participants tend to respond honestly to surveys of this form, even without explicit incentives for honesty. While we did not find a significant relationship between the quantity of participation and the quality of contribution for both response submissions and question submissions, we did find several other more nuanced participant behavior patterns, which did correlate with contribution in one of the three surveys. We conclude that there exists an optimal time for users to pose questions early on in their participation, but only after they have submitted a few responses to other questions. This suggests that future crowdsourced surveys may attract more predictive questions by prompting users to pose new questions at specific times during their participation and limiting question submission at non-optimal times.
authors.affiliation: Computer Science Department (graduated), University of Vermont, Burlington, United States of America.
authors.forename: Robert
authors.initials: R
authors.lastname: Swain
authors.affiliation: School of Business Administration, University of Vermont, Burlington, United States of America.
authors.forename: Alex
authors.initials: A
authors.lastname: Berger
authors.affiliation: Computer Science Department, University of Vermont, Burlington, United States of America; Complex Systems Center, University of Vermont, Burlington, United States of America.
authors.forename: Josh
authors.initials: J
authors.lastname: Bongard
authors.affiliation: School of Engineering, University of Vermont, Burlington, United States of America; Complex Systems Center, University of Vermont, Burlington, United States of America.
authors.forename: Paul
authors.initials: P
authors.lastname: Hines
mesh_terms: D000465:Algorithms; D063045:Crowdsourcing / Q000379:methods; D006801:Humans; D015233:Models, Statistical; D011795:Surveys and Questionnaires
references.citation: AJNR Am J Neuroradiol. 2013 Oct;34(10):1863-5
references.pmid: 23292527
references.citation: Psychol Bull. 1989 Mar;105(2):260-75
references.pmid: 2648443
references.citation: Science. 2009 Apr 3;324(5923):81-5
references.pmid: 19342586
references.citation: Nature. 2010 Aug 5;466(7307):756-60
references.pmid: 20686574
references.citation: PLoS One. 2014;9(2):e87756
references.pmid: 24505310
references.citation: Perspect Psychol Sci. 2011 Jan;6(1):3-5
references.pmid: 26162106
references.citation: Behav Res Methods. 2012 Mar;44(1):1-23
references.pmid: 21717266