title: Assessing Pictograph Recognition: A Comparison of Crowdsourcing and Traditional Survey Approaches.
abstract: BACKGROUND
  Compared to traditional methods of participant recruitment, online crowdsourcing platforms provide a fast and low-cost alternative. Amazon Mechanical Turk (MTurk) is a large and well-known crowdsourcing service. It has developed into the leading platform for crowdsourcing recruitment.
  
  
  OBJECTIVE
  To explore the application of online crowdsourcing for health informatics research, specifically the testing of medical pictographs.
  
  
  METHODS
  A set of pictographs created for cardiovascular hospital discharge instructions was tested for recognition. This set of illustrations (n=486) was first tested through an in-person survey in a hospital setting (n=150) and then using online MTurk participants (n=150). We analyzed these survey results to determine their comparability.
  
  
  RESULTS
  Both the demographics and the pictograph recognition rates of online participants were different from those of the in-person participants. In the multivariable linear regression model comparing the 2 groups, the MTurk group scored significantly higher than the hospital sample after adjusting for potential demographic characteristics (adjusted mean difference 0.18, 95% CI 0.08-0.28, P<.001). The adjusted mean ratings were 2.95 (95% CI 2.89-3.02) for the in-person hospital sample and 3.14 (95% CI 3.07-3.20) for the online MTurk sample on a 4-point Likert scale (1=totally incorrect, 4=totally correct).
  
  
  CONCLUSIONS
  The findings suggest that crowdsourcing is a viable complement to traditional in-person surveys, but it cannot replace them.
authors.affiliation: Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, United States. Jinqiu.kuang@utah.edu.
authors.forename: Jinqiu
authors.identifier: http://orcid.org/0000-0002-5458-1387
authors.initials: J
authors.lastname: Kuang
authors.forename: Lauren
authors.identifier: http://orcid.org/0000-0003-4514-2486
authors.initials: L
authors.lastname: Argo
authors.forename: Greg
authors.identifier: http://orcid.org/0000-0002-6292-276X
authors.initials: G
authors.lastname: Stoddard
authors.forename: Bruce E
authors.identifier: http://orcid.org/0000-0003-3963-2964
authors.initials: BE
authors.lastname: Bray
authors.forename: Qing
authors.identifier: http://orcid.org/0000-0002-8353-7473
authors.initials: Q
authors.lastname: Zeng-Treitler
mesh_terms: D000328:Adult; D063045:Crowdsourcing / Q000379:methods*; D003710:Demography; D005260:Female; D006801:Humans; D008297:Male; D063886:Patient Discharge Summaries; D011795:Surveys and Questionnaires / Q000706:statistics & numerical data*
keywords: Amazon Mechanical Turk; cardiovascular; crowdsourcing; patient discharge summaries; pictograph recognition