title: Intrinsic evaluation of text mining tools may not predict performance on realistic tasks.
abstract: Biomedical text mining and other automated techniques are beginning to achieve performance which suggests that they could be applied to aid database curators. However, few studies have evaluated how these systems might work in practice. In this article we focus on the problem of annotating mutations in Protein Data Bank (PDB) entries, and evaluate the relationship between performance of two automated techniques, a text-mining-based approach (MutationFinder) and an alignment-based approach, in intrinsic versus extrinsic evaluations. We find that high performance on gold standard data (an intrinsic evaluation) does not necessarily translate to high performance for database annotation (an extrinsic evaluation). We show that this is in part a result of lack of access to the full text of journal articles, which appears to be critical for comprehensive database annotation by text mining. Additionally, we evaluate the accuracy and completeness of manually annotated mutation data in the PDB, and find that it is far from perfect. We conclude that currently the most cost-effective and reliable approach for database annotation might incorporate manual and automatic annotation methods.
authors.affiliation: Center for Computational Pharmacology, University of Colorado Health Sciences Center, Aurora, CO, USA.
authors.forename: J Gregory
authors.initials: JG
authors.lastname: Caporaso
authors.forename: Nita
authors.initials: N
authors.lastname: Deshpande
authors.forename: J Lynn
authors.initials: JL
authors.lastname: Fink
authors.forename: Philip E
authors.initials: PE
authors.lastname: Bourne
authors.forename: K Bretonnel
authors.initials: KB
authors.lastname: Cohen
authors.forename: Lawrence
authors.initials: L
authors.lastname: Hunter
mesh_terms: D019295:Computational Biology; D016208:Databases, Factual; D030562:Databases, Protein; D016247:Information Storage and Retrieval; D009154:Mutation; D011506:Proteins / Q000235:genetics; D016415:Sequence Alignment / Q000706:statistics & numerical data
chemical_list: D011506:Proteins