title: Functional evaluation of out-of-the-box text-mining tools for data-mining tasks.
abstract: OBJECTIVE
  The trade-off between the speed and simplicity of dictionary-based term recognition and the richer linguistic information provided by more advanced natural language processing (NLP) is an area of active discussion in clinical informatics. In this paper, we quantify this trade-off among text processing systems that make different trade-offs between speed and linguistic understanding. We tested both types of systems in three clinical research tasks: phase IV safety profiling of a drug, learning adverse drug-drug interactions, and learning used-to-treat relationships between drugs and indications.
  
  
  METHODS
  We first benchmarked the accuracy of the NCBO Annotator and REVEAL in a manually annotated, publically available dataset from the 2008 i2b2 Obesity Challenge. We then applied the NCBO Annotator and REVEAL to 9 million clinical notes from the Stanford Translational Research Integrated Database Environment (STRIDE) and used the resulting data for three research tasks.
  
  
  RESULTS
  There is no significant difference between using the NCBO Annotator and REVEAL in the results of the three research tasks when using large datasets. In one subtask, REVEAL achieved higher sensitivity with smaller datasets.
  
  
  CONCLUSIONS
  For a variety of tasks, employing simple term recognition methods instead of advanced NLP methods results in little or no impact on accuracy when using large datasets. Simpler dictionary-based methods have the advantage of scaling well to very large datasets. Promoting the use of simple, dictionary-based methods for population level analyses can advance adoption of NLP in practice.
authors.affiliation: Program in Biomedical Informatics, Stanford University, Stanford, California, USA.
authors.forename: Kenneth
authors.initials: K
authors.lastname: Jung
authors.affiliation: Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA.
authors.forename: Paea
authors.initials: P
authors.lastname: LePendu
authors.affiliation: Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA.
authors.forename: Srinivasan
authors.initials: S
authors.lastname: Iyer
authors.affiliation: Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA.
authors.forename: Anna
authors.initials: A
authors.lastname: Bauer-Mehren
authors.affiliation: Program in Biomedical Informatics, Stanford University, Stanford, California, USA.
authors.forename: Bethany
authors.initials: B
authors.lastname: Percha
authors.affiliation: Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA.
authors.forename: Nigam H
authors.initials: NH
authors.lastname: Shah
mesh_terms: D001185:Artificial Intelligence; D057225:Data Mining / Q000379:methods*; D019992:Databases as Topic; D004347:Drug Interactions; D064420:Drug-Related Side Effects and Adverse Reactions; D057286:Electronic Health Records; D006801:Humans; D009323:Natural Language Processing; D009765:Obesity
keywords: electronic health records; natural language processing; text mining