title: Formative evaluation of ontology learning methods for entity discovery by using existing ontologies as reference standards.
abstract: OBJECTIVE
  Developing a two-step method for formative evaluation of statistical Ontology Learning (OL) algorithms that leverages existing biomedical ontologies as reference standards.
  
  
  METHODS
  In the first step optimum parameters are established. A 'gap list' of entities is generated by finding the set of entities present in a later version of the ontology that are not present in an earlier version of the ontology. A named entity recognition system is used to identify entities in a corpus of biomedical documents that are present in the 'gap list', generating a reference standard. The output of the algorithm (new entity candidates), produced by statistical methods, is subsequently compared against this reference standard. An OL method that performs perfectly will be able to learn all of the terms in this reference standard. Using evaluation metrics and precision-recall curves for different thresholds and parameters, we compute the optimum parameters for each method. In the second step, human judges with expertise in ontology development evaluate each candidate suggested by the algorithm configured with the optimum parameters previously established. These judgments are used to compute two performance metrics developed from our previous work: Entity Suggestion Rate (ESR) and Entity Acceptance Rate (EAR).
  
  
  RESULTS
  Using this method, we evaluated two statistical OL methods for OL in two medical domains. For the pathology domain, we obtained 49% ESR, 28% EAR with the Lin method and 52% ESR, 39% EAR with the Church method. For the radiology domain, we obtain 87% ESA, 9% EAR using Lin method and 96% ESR, 16% EAR using Church method.
  
  
  CONCLUSIONS
  This method is sufficiently general and flexible enough to permit comparison of any OL method for a specific corpus and ontology of interest.
authors.affiliation: Department of Biomedical Informatics, School of Medicine, University of Pittsburgh, Pittsburgh, PA, USA. kaihong@pitt.edu
authors.forename: K
authors.initials: K
authors.lastname: Liu
authors.forename: K J
authors.initials: KJ
authors.lastname: Mitchell
authors.forename: W W
authors.initials: WW
authors.lastname: Chapman
authors.forename: G K
authors.initials: GK
authors.lastname: Savova
authors.forename: N
authors.initials: N
authors.lastname: Sioutos
authors.forename: D L
authors.initials: DL
authors.lastname: Rubin
authors.forename: R S
authors.initials: RS
authors.lastname: Crowley
mesh_terms: D000046:Academic Medical Centers; D000465:Algorithms; D001185:Artificial Intelligence / Q000592:standards*; D064229:Biological Ontologies; D006801:Humans; D008492:Medical Informatics Computing / Q000592:standards*; D016347:Medical Records Systems, Computerized; D009323:Natural Language Processing; D010340:Pathology, Surgical; D010363:Pattern Recognition, Automated / Q000592:standards*; D010414:Pennsylvania; D011873:Radiology Information Systems; D012015:Reference Standards; D009626:Terminology as Topic; D018875:Vocabulary, Controlled