title: Crowdsourcing Twitter annotations to identify first-hand experiences of prescription drug use.
abstract: Self-reported patient data has been shown to be a valuable knowledge source for post-market pharmacovigilance. In this paper we propose using the popular micro-blogging service Twitter to gather evidence about adverse drug reactions (ADRs) after firstly having identified micro-blog messages (also know as "tweets") that report first-hand experience. In order to achieve this goal we explore machine learning with data crowdsourced from laymen annotators. With the help of lay annotators recruited from CrowdFlower we manually annotated 1548 tweets containing keywords related to two kinds of drugs: SSRIs (eg. Paroxetine), and cognitive enhancers (eg. Ritalin). Our results show that inter-annotator agreement (Fleiss' kappa) for crowdsourcing ranks in moderate agreement with a pair of experienced annotators (Spearman's Rho=0.471). We utilized the gold standard annotations from CrowdFlower for automatically training a range of supervised machine learning models to recognize first-hand experience. F-Score values are reported for 6 of these techniques with the Bayesian Generalized Linear Model being the best (F-Score=0.64 and Informedness=0.43) when combined with a selected set of features obtained by using information gain criteria.
authors.affiliation: Department of Informatics, National Institute of Informatics, Japan; The Graduate University for Advanced Studies, Japan. Electronic address: nestoralvaro@nii.ac.jp.
authors.forename: Nestor
authors.initials: N
authors.lastname: Alvaro
authors.affiliation: Department of Biomedical Informatics, University of Utah, USA. Electronic address: mike.conway@utah.edu.
authors.forename: Mike
authors.initials: M
authors.lastname: Conway
authors.affiliation: Medical Informatics, Kaiser Permanente Southern California, USA. Electronic address: Son.Doan@kp.org.
authors.forename: Son
authors.initials: S
authors.lastname: Doan
authors.affiliation: Institute for Information Systems, Technische Universität Braunschweig, Germany. Electronic address: lofi@ifis.cs.tu-bs.de.
authors.forename: Christoph
authors.initials: C
authors.lastname: Lofi
authors.affiliation: Stratified Medical, London, UK. Electronic address: john.overington@stratifiedmedical.com.
authors.forename: John
authors.initials: J
authors.lastname: Overington
authors.affiliation: Department of Theoretical and Applied Linguistics, University of Cambridge, UK; European Bioinformatics Institute (EMBL-EBI), Cambridge, UK. Electronic address: nhc30@cam.ac.uk.
authors.forename: Nigel
authors.initials: N
authors.lastname: Collier
mesh_terms: D063045:Crowdsourcing; D011307:Drug Prescriptions; D006801:Humans; D061108:Social Media
keywords: Crowdsourcing; Natural language processing; Pharmacovigilance; Twitter