title: Imitating manual curation of text-mined facts in biomedicine.
abstract: Text-mining algorithms make mistakes in extracting facts from natural-language texts. In biomedical applications, which rely on use of text-mined data, it is critical to assess the quality (the probability that the message is correctly extracted) of individual facts--to resolve data conflicts and inconsistencies. Using a large set of almost 100,000 manually produced evaluations (most facts were independently reviewed more than once, producing independent evaluations), we implemented and tested a collection of algorithms that mimic human evaluation of facts provided by an automated information-extraction system. The performance of our best automated classifiers closely approached that of our human evaluators (ROC score close to 0.95). Our hypothesis is that, were we to use a larger number of human experts to evaluate any given sentence, we could implement an artificial-intelligence curator that would perform the classification job at least as accurately as an average individual human evaluator. We illustrated our analysis by visualizing the predicted accuracy of the text-mined relations involving the term cocaine.
authors.affiliation: Department of Electrical Engineering, Columbia University, New York, New York, United States of America.
authors.forename: Raul
authors.initials: R
authors.lastname: Rodriguez-Esteban
authors.forename: Ivan
authors.initials: I
authors.lastname: Iossifov
authors.forename: Andrey
authors.initials: A
authors.lastname: Rzhetsky
mesh_terms: D000043:Abstracting and Indexing / Q000379:methods*; D000465:Algorithms; D001185:Artificial Intelligence; D001331:Automation; D035843:Biomedical Research; D003042:Cocaine; D019295:Computational Biology; D003198:Computer Simulation; D012680:Sensitivity and Specificity
chemical_list: D003042:Cocaine