Crowdsourcing the General Public for Large Scale Molecular Pathology Studies in Cancer.
BACKGROUND
Citizen science, scientific research conducted by non-specialists, has the potential to facilitate biomedical research using available large-scale data, however validating the results is challenging. The Cell Slider is a citizen science project that intends to share images from tumors with the general public, enabling them to score tumor markers independently through an internet-based interface.


METHODS
From October 2012 to June 2014, 98,293 Citizen Scientists accessed the Cell Slider web page and scored 180,172 sub-images derived from images of 12,326 tissue microarray cores labeled for estrogen receptor (ER). We evaluated the accuracy of Citizen Scientist's ER classification, and the association between ER status and prognosis by comparing their test performance against trained pathologists.


RESULTS
The area under ROC curve was 0.95 (95% CI 0.94 to 0.96) for cancer cell identification and 0.97 (95% CI 0.96 to 0.97) for ER status. ER positive tumors scored by Citizen Scientists were associated with survival in a similar way to that scored by trained pathologists. Survival probability at 15 years were 0.78 (95% CI 0.76 to 0.80) for ER-positive and 0.72 (95% CI 0.68 to 0.77) for ER-negative tumors based on Citizen Scientists classification. Based on pathologist classification, survival probability was 0.79 (95% CI 0.77 to 0.81) for ER-positive and 0.71 (95% CI 0.67 to 0.74) for ER-negative tumors. The hazard ratio for death was 0.26 (95% CI 0.18 to 0.37) at diagnosis and became greater than one after 6.5 years of follow-up for ER scored by Citizen Scientists, and 0.24 (95% CI 0.18 to 0.33) at diagnosis increasing thereafter to one after 6.7 (95% CI 4.1 to 10.9) years of follow-up for ER scored by pathologists.


CONCLUSIONS
Crowdsourcing of the general public to classify cancer pathology data for research is viable, engages the public and provides accurate ER data. Crowdsourced classification of research data may offer a valid solution to problems of throughput requiring human input.
Department of Oncology, University of Cambridge, Cambridge, UK ; Department of Gynecology and Obstetrics, Ribeirao Preto School of Medicine, University of Sao Paulo, Ribeirao Preto, Brazil.
Francisco J
FJ
Candido Dos Reis
Department of Physics (Astrophysics), University of Oxford, Oxford, UK.
Stuart
S
Lynn
Cancer Research UK, Cambridge Institute, Cambridge, UK.
H Raza
HR
Ali
University of Southampton, Southampton, UK.
Diana
D
Eccles
University of Leeds, Leeds, UK.
Andrew
A
Hanby
Addenbrookes Hospital NHS Trust, Cambridge, UK.
Elena
E
Provenzano
Cancer Research UK, Cambridge Institute, Cambridge, UK.
Carlos
C
Caldas
Cancer Research UK, Cambridge Institute, Cambridge, UK.
William J
WJ
Howat
Cancer Research UK, Cambridge Institute, Cambridge, UK.
Leigh-Anne
LA
McDuffus
Cancer Research UK, Cambridge Institute, Cambridge, UK.
Bin
B
Liu
Institute of Cancer Research, London, UK.
Frances
F
Daley
Institute of Cancer Research, London, UK.
Penny
P
Coulson
Cancer Research UK, London, UK.
Rupesh J
RJ
Vyas
Cancer Research UK, London, UK.
Leslie M
LM
Harris
Cancer Research UK, London, UK.
Joanna M
JM
Owens
Cancer Research UK, London, UK.
Amy F M
AF
Carton
Cancer Research UK, London, UK.
Janette P
JP
McQuillan
Cancer Research UK, London, UK.
Andy M
AM
Paterson
Cancer Research UK, London, UK.
Zohra
Z
Hirji
Cancer Research UK, London, UK.
Sarah K
SK
Christie
Cancer Research UK, London, UK.
Amber R
AR
Holmes
Netherlands Cancer Institute, Antoni van Leeuwenhoek Hospital, Amsterdam, The Netherlands.
Marjanka K
MK
Schmidt
Institute of Cancer Research, London, UK.
Montserrat
M
Garcia-Closas
Department of Oncology, University of Cambridge, Cambridge, UK ; Department of Public Health and Primary Care, University of Cambridge, Cambridge, UK.
Douglas F
DF
Easton
Department of Public Health and Primary Care, University of Cambridge, Cambridge, UK.
Manjeet K
MK
Bolla
Department of Public Health and Primary Care, University of Cambridge, Cambridge, UK.
Qin
Q
Wang
Human Genotyping (CEGEN) Unit, Human Cancer Genetics Program, Spanish National Cancer Research Centre (CNIO), Madrid, Spain ; Biomedical Network on Rare Diseases (CIBERER), Madrid, Spain.
Javier
J
Benitez
Cancer Epidemiology Centre, Cancer Council Victoria, Melbourne, Australia ; Centre for Epidemiology and Biostatistics, Melbourne School of Population and Global Health, The University of Melbourne, Australia.
Roger L
RL
Milne
Institute of Clinical Medicine, Pathology and Forensic Medicine, University of Eastern Finland, Finland.
Arto
A
Mannermaa
Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, MN, USA.
Fergus
F
Couch
Department of Human Genetics & Department of Pathology, Leiden University Medical Center, Leiden, The Netherlands.
Peter
P
Devilee
Department of Surgical Oncology, Leiden University Medical Center, Leiden, The Netherlands.
Robert A E M
RA
Tollenaar
Department of Medical Oncology, Family Cancer Clinic, Erasmus MC Cancer Institute, Rotterdam, The Netherlands.
Caroline
C
Seynaeve
Sheffield Cancer Research, Department of Oncology, University of Sheffield, Sheffield, UK.
Angela
A
Cox
Academic Unit of Pathology, Department of Neuroscience, University of Sheffield, Sheffield, UK.
Simon S
SS
Cross
Department of Oncology, University of Cambridge, Cambridge, UK.
Fiona M
FM
Blows
Netherlands Cancer Institute, Antoni van Leeuwenhoek Hospital, Amsterdam, The Netherlands.
Joyce
J
Sanders
Netherlands Cancer Institute, Antoni van Leeuwenhoek Hospital, Amsterdam, The Netherlands.
Renate
R
de Groot
National Cancer Institute, USA.
Jonine
J
Figueroa
National Cancer Institute, USA.
Mark
M
Sherman
Department of Medical Oncology, Family Cancer Clinic, Erasmus MC Cancer Institute, Rotterdam, The Netherlands.
Maartje
M
Hooning
Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany ; Division of Preventive Oncology, German Cancer Research Center (DKFZ), Heidelberg, Germany ; German Cancer Consortium (DKTK), German Cancer Research Center (DKFZ), Heidelberg, Germany.
Hermann
H
Brenner
Saarland Cancer Registry, Saarbrücken, Germany.
Bernd
B
Holleczek
Saarland Cancer Registry, Saarbrücken, Germany.
Christa
C
Stegmaier
Department of Physics (Astrophysics), University of Oxford, Oxford, UK.
Chris
C
Lintott
Department of Oncology, University of Cambridge, Cambridge, UK ; Department of Public Health and Primary Care, University of Cambridge, Cambridge, UK.
Paul D P
PD
Pharoah
D001943:Breast Neoplasms / Q000401:mortality / Q000473:pathology*; D063045:Crowdsourcing; D005260:Female; D006801:Humans; D053208:Kaplan-Meier Estimate; D057089:Pathology, Molecular; D016016:Proportional Hazards Models; D012372:ROC Curve; D011960:Receptors, Estrogen / Q000378:metabolism
D011960:Receptors, Estrogen
Breast cancer; Citizen science; Crowd science; Crowdsourcing