title: Is the crowd better as an assistant or a replacement in ontology engineering? An exploration through the lens of the Gene Ontology.
abstract: Biomedical ontologies contain errors. Crowdsourcing, defined as taking a job traditionally performed by a designated agent and outsourcing it to an undefined large group of people, provides scalable access to humans. Therefore, the crowd has the potential to overcome the limited accuracy and scalability found in current ontology quality assurance approaches. Crowd-based methods have identified errors in SNOMED CT, a large, clinical ontology, with an accuracy similar to that of experts, suggesting that crowdsourcing is indeed a feasible approach for identifying ontology errors. This work uses that same crowd-based methodology, as well as a panel of experts, to verify a subset of the Gene Ontology (200 relationships). Experts identified 16 errors, generally in relationships referencing acids and metals. The crowd performed poorly in identifying those errors, with an area under the receiver operating characteristic curve ranging from 0.44 to 0.73, depending on the methods configuration. However, when the crowd verified what experts considered to be easy relationships with useful definitions, they performed reasonably well. Notably, there are significantly fewer Google search results for Gene Ontology concepts than SNOMED CT concepts. This disparity may account for the difference in performance - fewer search results indicate a more difficult task for the worker. The number of Internet search results could serve as a method to assess which tasks are appropriate for the crowd. These results suggest that the crowd fits better as an expert assistant, helping experts with their verification by completing the easy tasks and allowing experts to focus on the difficult tasks, rather than an expert replacement.
authors.affiliation: Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, CA 94305-5479, United States; Biomedical Informatics Training Program, Stanford University, Stanford, CA 94305-5479, United States.
authors.forename: Jonathan M
authors.initials: JM
authors.lastname: Mortensen
authors.affiliation: Biomedical Informatics Training Program, Stanford University, Stanford, CA 94305-5479, United States.
authors.forename: Natalie
authors.initials: N
authors.lastname: Telis
authors.affiliation: Institute of Computational Health Sciences, University of California, San Francisco, San Francisco, CA 94143, United States.
authors.forename: Jacob J
authors.initials: JJ
authors.lastname: Hughey
authors.affiliation: Department of Pediatrics, Stanford University, Stanford, CA 94305-5479, United States.
authors.forename: Hua
authors.initials: H
authors.lastname: Fan-Minogue
authors.affiliation: Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA 91125, United States.
authors.forename: Kimberly
authors.initials: K
authors.lastname: Van Auken
authors.affiliation: Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, CA 94305-5479, United States.
authors.forename: Michel
authors.initials: M
authors.lastname: Dumontier
authors.affiliation: Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, CA 94305-5479, United States. Electronic address: musen@stanford.edu.
authors.forename: Mark A
authors.initials: MA
authors.lastname: Musen
mesh_terms: D000465:Algorithms; D000704:Analysis of Variance; D019540:Area Under Curve; D019295:Computational Biology / Q000379:methods; D063045:Crowdsourcing / Q000379:methods*; D063990:Gene Ontology; D006801:Humans; D020407:Internet; D057386:Search Engine; D012984:Software; D039061:Systematized Nomenclature of Medicine; D013647:Task Performance and Analysis
keywords: Crowdsourcing; Gene Ontology; Ontology engineering