title: Using the wisdom of the crowds to find critical errors in biomedical ontologies: a study of SNOMED CT.
abstract: OBJECTIVE
  The verification of biomedical ontologies is an arduous process that typically involves peer review by subject-matter experts. This work evaluated the ability of crowdsourcing methods to detect errors in SNOMED CT (Systematized Nomenclature of Medicine Clinical Terms) and to address the challenges of scalable ontology verification.
  
  
  METHODS
  We developed a methodology to crowdsource ontology verification that uses micro-tasking combined with a Bayesian classifier. We then conducted a prospective study in which both the crowd and domain experts verified a subset of SNOMED CT comprising 200 taxonomic relationships.
  
  
  RESULTS
  The crowd identified errors as well as any single expert at about one-quarter of the cost. The inter-rater agreement (κ) between the crowd and the experts was 0.58; the inter-rater agreement between experts themselves was 0.59, suggesting that the crowd is nearly indistinguishable from any one expert. Furthermore, the crowd identified 39 previously undiscovered, critical errors in SNOMED CT (eg, 'septic shock is a soft-tissue infection').
  
  
  CONCLUSIONS
  The results show that the crowd can indeed identify errors in SNOMED CT that experts also find, and the results suggest that our method will likely perform well on similar ontologies. The crowd may be particularly useful in situations where an expert is unavailable, budget is limited, or an ontology is too large for manual error checking. Finally, our results suggest that the online anonymous crowd could successfully complete other domain-specific tasks.
  
  
  CONCLUSIONS
  We have demonstrated that the crowd can address the challenges of scalable ontology verification, completing not only intuitive, common-sense tasks, but also expert-level, knowledge-intensive tasks.
authors.affiliation: Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA Biomedical Informatics Training Program, Stanford University, Stanford, California, USA.
authors.forename: Jonathan M
authors.initials: JM
authors.lastname: Mortensen
authors.affiliation: Biomedical Informatics Training Program, Stanford University, Stanford, California, USA Faculty of Medicine, University of Calgary, Calgary, Canada.
authors.forename: Evan P
authors.initials: EP
authors.lastname: Minty
authors.affiliation: Biomedical Informatics Training Program, Stanford University, Stanford, California, USA.
authors.forename: Michael
authors.initials: M
authors.lastname: Januszyk
authors.affiliation: Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA.
authors.forename: Timothy E
authors.initials: TE
authors.lastname: Sweeney
authors.affiliation: School of Computer Science, University of Manchester, Manchester, UK.
authors.forename: Alan L
authors.initials: AL
authors.lastname: Rector
authors.affiliation: Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA Google Inc., Mountain View, California, USA.
authors.forename: Natalya F
authors.initials: NF
authors.lastname: Noy
authors.affiliation: Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA Biomedical Informatics Training Program, Stanford University, Stanford, California, USA.
authors.forename: Mark A
authors.initials: MA
authors.lastname: Musen
mesh_terms: D001499:Bayes Theorem; D064229:Biological Ontologies; D063045:Crowdsourcing; D004194:Disease / Q000145:classification*; D006801:Humans; D039061:Systematized Nomenclature of Medicine
keywords: SNOMED CT; biomedical ontology; crowdsourcing; ontology engineering