title: Evaluation of a UMLS Auditing Process of Semantic Type Assignments.
abstract: The UMLS is a terminological system that integrates many source terminologies. Each concept in the UMLS is assigned one or more semantic types from the Semantic Network, an upper level ontology for biomedicine. Due to the complexity of the UMLS, errors exist in the semantic type assignments. Finding assignment errors may unearth modeling errors. Even with sophisticated tools, discovering assignment errors requires manual review. In this paper we describe the evaluation of an auditing project of UMLS semantic type assignments. We studied the performance of the auditors who reviewed potential errors. We found that four auditors, interacting according to a multi-step protocol, identified a high rate of errors (one or more errors in 81% of concepts studied) and that results were sufficiently reliable (0.67 to 0.70) for the two most common types of errors. However, reliability was low for each individual auditor, suggesting that review of potential errors is resource-intensive.
authors.affiliation: SHRP, University of Medicine and Dentistry of New Jersey.
authors.forename: Huanying Helen
authors.initials: HH
authors.lastname: Gu
authors.forename: George
authors.initials: G
authors.lastname: Hripcsak
authors.forename: Yan
authors.initials: Y
authors.lastname: Chen
authors.forename: C Paul
authors.initials: CP
authors.lastname: Morrey
authors.forename: Gai
authors.initials: G
authors.lastname: Elhanan
authors.forename: James
authors.initials: J
authors.lastname: Cimino
authors.forename: James
authors.initials: J
authors.lastname: Geller
authors.forename: Yehoshua
authors.initials: Y
authors.lastname: Perl
mesh_terms: D000043:Abstracting and Indexing; D012660:Semantics; D017432:Unified Medical Language System