SparkText: Biomedical Text Mining on Big Data Framework.
BACKGROUND
Many new biomedical research articles are published every day, accumulating rich information, such as genetic variants, genes, diseases, and treatments. Rapid yet accurate text mining on large-scale scientific literature can discover novel knowledge to better understand human diseases and to improve the quality of disease diagnosis, prevention, and treatment.


RESULTS
In this study, we designed and developed an efficient text mining framework called SparkText on a Big Data infrastructure, which is composed of Apache Spark data streaming and machine learning methods, combined with a Cassandra NoSQL database. To demonstrate its performance for classifying cancer types, we extracted information (e.g., breast, prostate, and lung cancers) from tens of thousands of articles downloaded from PubMed, and then employed Naïve Bayes, Support Vector Machine (SVM), and Logistic Regression to build prediction models to mine the articles. The accuracy of predicting a cancer type by SVM using the 29,437 full-text articles was 93.81%. While competing text-mining tools took more than 11 hours, SparkText mined the dataset in approximately 6 minutes.


CONCLUSIONS
This study demonstrates the potential for mining large-scale scientific articles on a Big Data infrastructure, with real-time update from new articles published daily. SparkText can be extended to other areas of biomedical research.
Biomedical Informatics Research Center, Marshfield Clinic Research Foundation, Marshfield, WI, 54449, United States of America.
Zhan
Z
Ye
Center for Human Genetics, Marshfield Clinic Research Foundation, Marshfield, WI, 54449, United States of America.|Department of Computer Science, University of Wisconsin-Milwaukee, Milwaukee, WI, 53211, United States of America.
Ahmad P
AP
Tafti
Department of Epidemiology and Biostatistics, Case Western Reserve University, Cleveland, OH, 44106, United States of America.
Karen Y
KY
He
Zilkha Neurogenetic Institute, University of Southern California, Los Angeles, CA, 90089, United States of America.|Department of Psychiatry, University of Southern California, Los Angeles, CA, 90089, United States of America.
Kai
K
Wang
Biomedical Informatics Research Center, Marshfield Clinic Research Foundation, Marshfield, WI, 54449, United States of America.|Center for Human Genetics, Marshfield Clinic Research Foundation, Marshfield, WI, 54449, United States of America.|Computation and Informatics in Biology and Medicine, University of Wisconsin-Madison, Madison, WI, 53706, United States of America.
Max M
http://orcid.org/0000-0002-5347-6374
MM
He
Nat Rev Cancer. 2015 Nov;15(11):639-52
26493645
Adv Bioinformatics. 2013;2013:320436
23365570
Drug Discov Today. 2005 Mar 15;10(6):439-45
15808823
Pac Symp Biocomput. 2000;(12):365-6
10902184
BMC Bioinformatics. 2008 Jan 14;9:22
18194557
Neuroinformatics. 2012 Oct;10(4):409-13
22426993
J Biomed Inform. 2013 Apr;46(2):200-11
23159498
Nat Rev Genet. 2012 Dec;13(12):829-39
23150036
Eur J Hum Genet. 2006 May;14(5):535-42
16493445
Nat Rev Cancer. 2015 Dec;15(12):698
26511457
Nucleic Acids Res. 2008 Jan;36(Database issue):D842-6
17932060
Nat Rev Cancer. 2015 Dec;15(12):747-56
26536825
Bull Med Libr Assoc. 2000 Jul;88(3):265-6
10928714
Brief Bioinform. 2016 May;17(3):468-78
26220932
Bioinformatics. 2004 Oct 12;20(15):2479-81
15073010
Cognition. 2014 Sep;132(3):429-36
24908342
PLoS One. 2012;7(4):e33427
22511921