title: An overview of the BioCreative 2012 Workshop Track III: interactive text mining task.
abstract: In many databases, biocuration primarily involves literature curation, which usually involves retrieving relevant articles, extracting information that will translate into annotations and identifying new incoming literature. As the volume of biological literature increases, the use of text mining to assist in biocuration becomes increasingly relevant. A number of groups have developed tools for text mining from a computer science/linguistics perspective, and there are many initiatives to curate some aspect of biology from the literature. Some biocuration efforts already make use of a text mining tool, but there have not been many broad-based systematic efforts to study which aspects of a text mining tool contribute to its usefulness for a curation task. Here, we report on an effort to bring together text mining tool developers and database biocurators to test the utility and usability of tools. Six text mining systems presenting diverse biocuration tasks participated in a formal evaluation, and appropriate biocurators were recruited for testing. The performance results from this evaluation indicate that some of the systems were able to improve efficiency of curation by speeding up the curation task significantly (∼1.7- to 2.5-fold) over manual curation. In addition, some of the systems were able to improve annotation accuracy when compared with the performance on the manually curated set. In terms of inter-annotator agreement, the factors that contributed to significant differences for some of the systems included the expertise of the biocurator on the given curation task, the inherent difficulty of the curation and attention to annotation guidelines. After the task, annotators were asked to complete a survey to help identify strengths and weaknesses of the various systems. The analysis of this survey highlights how important task completion is to the biocurators' overall experience of a system, regardless of the system's high score on design, learnability and usability. In addition, strategies to refine the annotation guidelines and systems documentation, to adapt the tools to the needs and query types the end user might have and to evaluate performance in terms of efficiency, user interface, result export and traditional evaluation metrics have been analyzed during this task. This analysis will help to plan for a more intense study in BioCreative IV.
authors: Center for Bioinformatics and Computational Biology, University of Delaware, Newark, DE 19711, USA. arighi@dbi.udel.edu
authors: Cecilia N
authors: CN
authors: Arighi
authors: Ben
authors: B
authors: Carterette
authors: K Bretonnel
authors: KB
authors: Cohen
authors: Martin
authors: M
authors: Krallinger
authors: W John
authors: WJ
authors: Wilbur
authors: Petra
authors: P
authors: Fey
authors: Robert
authors: R
authors: Dodson
authors: Laurel
authors: L
authors: Cooper
authors: Ceri E
authors: CE
authors: Van Slyke
authors: Wasila
authors: W
authors: Dahdul
authors: Paula
authors: P
authors: Mabee
authors: Donghui
authors: D
authors: Li
authors: Bethany
authors: B
authors: Harris
authors: Marc
authors: M
authors: Gillespie
authors: Silvia
authors: S
authors: Jimenez
authors: Phoebe
authors: P
authors: Roberts
authors: Lisa
authors: L
authors: Matthews
authors: Kevin
authors: K
authors: Becker
authors: Harold
authors: H
authors: Drabkin
authors: Susan
authors: S
authors: Bello
authors: Luana
authors: L
authors: Licata
authors: Andrew
authors: A
authors: Chatr-aryamontri
authors: Mary L
authors: ML
authors: Schaeffer
authors: Julie
authors: J
authors: Park
authors: Melissa
authors: M
authors: Haendel
authors: Kimberly
authors: K
authors: Van Auken
authors: Yuling
authors: Y
authors: Li
authors: Juancarlos
authors: J
authors: Chan
authors: Hans-Michael
authors: HM
authors: Muller
authors: Hong
authors: H
authors: Cui
authors: James P
authors: JP
authors: Balhoff
authors: Johnny
authors: J
authors: Chi-Yang Wu
authors: Zhiyong
authors: Z
authors: Lu
authors: Chih-Hsuan
authors: CH
authors: Wei
authors: Catalina O
authors: CO
authors: Tudor
authors: Kalpana
authors: K
authors: Raja
authors: Suresh
authors: S
authors: Subramani
authors: Jeyakumar
authors: J
authors: Natarajan
authors: Juan Miguel
authors: JM
authors: Cejuela
authors: Pratibha
authors: P
authors: Dubey
authors: Cathy
authors: C
authors: Wu
mesh_terms: D057225:Data Mining; D019992:Databases as Topic; D004282:Documentation; D004493:Education; D006801:Humans; D012984:Software; D013997:Time Factors