Corpus-based statistical screening for phrase identification.
OBJECTIVE
The authors study the extraction of useful phrases from a natural language database by statistical methods. The aim is to leverage human effort by providing preprocessed phrase lists with a high percentage of useful material.


METHODS
The approach is to develop six different scoring methods that are based on different aspects of phrase occurrence. The emphasis here is not on lexical information or syntactic structure but rather on the statistical properties of word pairs and triples that can be obtained from a large database.


METHODS
The Unified Medical Language System (UMLS) incorporates a large list of humanly acceptable phrases in the medical field as a part of its structure. The authors use this list of phrases as a gold standard for validating their methods. A good method is one that ranks the UMLS phrases high among all phrases studied. Measurements are 11-point average precision values and precision-recall curves based on the rankings.


RESULTS
The authors find of six different scoring methods that each proves effective in identifying UMLS quality phrases in a large subset of MEDLINE. These methods are applicable both to word pairs and word triples. All six methods are optimally combined to produce composite scoring methods that are more effective than any single method. The quality of the composite methods appears sufficient to support the automatic placement of hyperlinks in text at the site of highly ranked phrases.


CONCLUSIONS
Statistical scoring methods provide a promising approach to the extraction of useful phrases from a natural language database for the purpose of indexing or providing hyperlinks in text.
Kim
W
W
National Library of Medicine, Bethesda, Maryland 20894, USA. wonkim@ncbi.nlm.nih.gov
Wilbur
W J
WJ
D000043:Abstracting and Indexing; D019146:Hypermedia; D016247:Information Storage and Retrieval / Q000379:methods*; D016239:MEDLINE; D009323:Natural Language Processing; D013223:Statistics as Topic; D017432:Unified Medical Language System; D018875:Vocabulary, Controlled