title: The cure: design and evaluation of a crowdsourcing game for gene selection for breast cancer survival prediction.
abstract: BACKGROUND
  Molecular signatures for predicting breast cancer prognosis could greatly improve care through personalization of treatment. Computational analyses of genome-wide expression datasets have identified such signatures, but these signatures leave much to be desired in terms of accuracy, reproducibility, and biological interpretability. Methods that take advantage of structured prior knowledge (eg, protein interaction networks) show promise in helping to define better signatures, but most knowledge remains unstructured. Crowdsourcing via scientific discovery games is an emerging methodology that has the potential to tap into human intelligence at scales and in modes unheard of before.
  
  
  OBJECTIVE
  The main objective of this study was to test the hypothesis that knowledge linking expression patterns of specific genes to breast cancer outcomes could be captured from players of an open, Web-based game. We envisioned capturing knowledge both from the player's prior experience and from their ability to interpret text related to candidate genes presented to them in the context of the game.
  
  
  METHODS
  We developed and evaluated an online game called The Cure that captured information from players regarding genes for use as predictors of breast cancer survival. Information gathered from game play was aggregated using a voting approach, and used to create rankings of genes. The top genes from these rankings were evaluated using annotation enrichment analysis, comparison to prior predictor gene sets, and by using them to train and test machine learning systems for predicting 10 year survival.
  
  
  RESULTS
  Between its launch in September 2012 and September 2013, The Cure attracted more than 1000 registered players, who collectively played nearly 10,000 games. Gene sets assembled through aggregation of the collected data showed significant enrichment for genes known to be related to key concepts such as cancer, disease progression, and recurrence. In terms of the predictive accuracy of models trained using this information, these gene sets provided comparable performance to gene sets generated using other methods, including those used in commercial tests. The Cure is available on the Internet.
  
  
  CONCLUSIONS
  The principal contribution of this work is to show that crowdsourcing games can be developed as a means to address problems involving domain knowledge. While most prior work on scientific discovery games and crowdsourcing in general takes as a premise that contributors have little or no expertise, here we demonstrated a crowdsourcing system that succeeded in capturing expert knowledge.
authors: The Scripps Research Institute, Department of Molecular and Experimental Medicine, La Jolla, CA, United States. bgood@scripps.edu.
authors: Benjamin M
authors: http://orcid.org/0000-0002-7334-7852
authors: BM
authors: Good
authors: Salvatore
authors: http://orcid.org/0000-0002-7544-2992
authors: S
authors: Loguercio
authors: Obi L
authors: http://orcid.org/0000-0002-0843-4271
authors: OL
authors: Griffith
authors: Max
authors: http://orcid.org/0000-0001-7587-232X
authors: M
authors: Nanis
authors: Chunlei
authors: http://orcid.org/0000-0002-2629-6124
authors: C
authors: Wu
authors: Andrew I
authors: http://orcid.org/0000-0002-9859-4104
authors: AI
authors: Su
keywords: Web applications; artificial intelligence; breast neoplasms; collaborative and social computing systems and tools; computer games; crowdsourcing; feature selection; gene expression; supervised learning; survival analysis
references: Genome Biol. 2013;14(10):R110
references: 24176112
references: Nucleic Acids Res. 2013 Jul;41(Web Server issue):W77-83
references: 23703215
references: PLoS Comput Biol. 2011 Sep;7(9):e1002180
references: 21980275
references: Oncologist. 2007 Jun;12(6):631-5
references: 17602054
references: Nat Rev Clin Oncol. 2011 Aug 30;9(1):58-64
references: 21878891
references: Oncologist. 2008 May;13(5):477-93
references: 18515733
references: Sci Transl Med. 2013 Apr 17;5(181):181ra50
references: 23596202
references: Nature. 2002 Jan 31;415(6871):530-6
references: 11823860
references: Sci Transl Med. 2013 Apr 17;5(181):181re1
references: 23596205
references: J Med Internet Res. 2012 Nov 29;14(6):e167
references: 23196001
references: PLoS One. 2009 Dec 07;4(12):e8161
references: 19997592
references: Nature. 2010 Aug 5;466(7307):756-60
references: 20686574
references: PLoS Comput Biol. 2013;9(2):e1002920
references: 23468608
references: Bioinformatics. 2013 Aug 15;29(16):1925-33
references: 23782614
references: PLoS Comput Biol. 2012;8(5):e1002511
references: 22615549
references: Nature. 2006 Jan 19;439(7074):353-7
references: 16273092
references: PLoS Comput Biol. 2011 Oct;7(10):e1002240
references: 22028643
references: PLoS One. 2012;7(3):e31362
references: 22412834
references: Nature. 2012 Apr 18;486(7403):346-52
references: 22522925
references: Int J Cancer. 2013 Mar 1;132(5):1133-45
references: 22752881
references: Breast Cancer Res Treat. 2008 Jul;110(2):235-44
references: 17899371
references: Genome Med. 2013 Oct 11;5(10):92
references: 24112773