title: Crowdsourcing for translational research: analysis of biomarker expression using cancer microarrays.
abstract: BACKGROUND
  Academic pathology suffers from an acute and growing lack of workforce resource. This especially impacts on translational elements of clinical trials, which can require detailed analysis of thousands of tissue samples. We tested whether crowdsourcing - enlisting help from the public - is a sufficiently accurate method to score such samples.
  
  
  METHODS
  We developed a novel online interface to train and test lay participants on cancer detection and immunohistochemistry scoring in tissue microarrays. Lay participants initially performed cancer detection on lung cancer images stained for CD8, and we measured how extending a basic tutorial by annotated example images and feedback-based training affected cancer detection accuracy. We then applied this tutorial to additional cancer types and immunohistochemistry markers - bladder/ki67, lung/EGFR, and oesophageal/CD8 - to establish accuracy compared with experts. Using this optimised tutorial, we then tested lay participants' accuracy on immunohistochemistry scoring of lung/EGFR and bladder/p53 samples.
  
  
  RESULTS
  We observed that for cancer detection, annotated example images and feedback-based training both improved accuracy compared with a basic tutorial only. Using this optimised tutorial, we demonstrate highly accurate (>0.90 area under curve) detection of cancer in samples stained with nuclear, cytoplasmic and membrane cell markers. We also observed high Spearman correlations between lay participants and experts for immunohistochemistry scoring (0.91 (0.78, 0.96) and 0.97 (0.91, 0.99) for lung/EGFR and bladder/p53 samples, respectively).
  
  
  CONCLUSIONS
  These results establish crowdsourcing as a promising method to screen large data sets for biomarkers in cancer pathology research across a range of cancers and immunohistochemical stains.
authors.affiliation: Cancer Research UK, 407 St John Street, London EC1V 4AD, UK.
authors.forename: Jonathan
authors.initials: J
authors.lastname: Lawson
authors.affiliation: Cancer Research UK, 407 St John Street, London EC1V 4AD, UK.
authors.forename: Rupesh J
authors.initials: RJ
authors.lastname: Robinson-Vyas
authors.affiliation: Cancer Research UK, 407 St John Street, London EC1V 4AD, UK.
authors.forename: Janette P
authors.initials: JP
authors.lastname: McQuillan
authors.affiliation: Cancer Research UK, 407 St John Street, London EC1V 4AD, UK.
authors.forename: Andy
authors.initials: A
authors.lastname: Paterson
authors.affiliation: Cancer Research UK, 407 St John Street, London EC1V 4AD, UK.
authors.forename: Sarah
authors.initials: S
authors.lastname: Christie
authors.affiliation: Cancer Research UK, 407 St John Street, London EC1V 4AD, UK.
authors.forename: Matthew
authors.initials: M
authors.lastname: Kidza-Griffiths
authors.affiliation: Cancer Research UK Cambridge Institute, University of Cambridge, Robinson Way, Cambridge CB2 0RE, UK.
authors.forename: Leigh-Anne
authors.initials: LA
authors.lastname: McDuffus
authors.affiliation: Southampton CRUK Centre, University of Southampton Faculty of Medicine, Tremona Road, Southampton SO16 6YD, UK.
authors.forename: Karwan A
authors.initials: KA
authors.lastname: Moutasim
authors.affiliation: Cancer Research UK, 407 St John Street, London EC1V 4AD, UK.|Southampton CRUK Centre, University of Southampton Faculty of Medicine, Tremona Road, Southampton SO16 6YD, UK.
authors.forename: Emily C
authors.initials: EC
authors.lastname: Shaw
authors.affiliation: CRUK/MRC Oxford Institute for Radiation Oncology, University of Oxford, Roosevelt Drive, Oxford OX3 7DQ, UK.
authors.forename: Anne E
authors.initials: AE
authors.lastname: Kiltie
authors.affiliation: Cancer Research UK Cambridge Institute, University of Cambridge, Robinson Way, Cambridge CB2 0RE, UK.
authors.forename: William J
authors.initials: WJ
authors.lastname: Howat
authors.affiliation: Cancer Research UK Leeds Centre, University of Leeds, Beckett Street, Leeds LS9 7TF, UK.
authors.forename: Andrew M
authors.initials: AM
authors.lastname: Hanby
authors.affiliation: Southampton CRUK Centre, University of Southampton Faculty of Medicine, Tremona Road, Southampton SO16 6YD, UK.
authors.forename: Gareth J
authors.initials: GJ
authors.lastname: Thomas
authors.affiliation: Cancer Research UK, 407 St John Street, London EC1V 4AD, UK.
authors.forename: Peter
authors.initials: P
authors.lastname: Smittenaar
mesh_terms: D014408:Biomarkers, Tumor / Q000378:metabolism*; D063045:Crowdsourcing / Q000379:methods*; D003627:Data Interpretation, Statistical; D006801:Humans; D007091:Image Processing, Computer-Assisted / Q000379:methods; D007150:Immunohistochemistry; D009369:Neoplasms / Q000378:metabolism*; D018579:Patient Selection; D046888:Tissue Array Analysis; D057170:Translational Research, Biomedical / Q000379:methods*
chemical_list: D014408:Biomarkers, Tumor