title: Assessing Electronic Cigarette-Related Tweets for Sentiment and Content Using Supervised Machine Learning.
abstract: BACKGROUND
  Electronic cigarettes (e-cigarettes) continue to be a growing topic among social media users, especially on Twitter. The ability to analyze conversations about e-cigarettes in real-time can provide important insight into trends in the public's knowledge, attitudes, and beliefs surrounding e-cigarettes, and subsequently guide public health interventions.
  
  
  OBJECTIVE
  Our aim was to establish a supervised machine learning algorithm to build predictive classification models that assess Twitter data for a range of factors related to e-cigarettes.
  
  
  METHODS
  Manual content analysis was conducted for 17,098 tweets. These tweets were coded for five categories: e-cigarette relevance, sentiment, user description, genre, and theme. Machine learning classification models were then built for each of these five categories, and word groupings (n-grams) were used to define the feature space for each classifier.
  
  
  RESULTS
  Predictive performance scores for classification models indicated that the models correctly labeled the tweets with the appropriate variables between 68.40% and 99.34% of the time, and the percentage of maximum possible improvement over a random baseline that was achieved by the classification models ranged from 41.59% to 80.62%. Classifiers with the highest performance scores that also achieved the highest percentage of the maximum possible improvement over a random baseline were Policy/Government (performance: 0.94; % improvement: 80.62%), Relevance (performance: 0.94; % improvement: 75.26%), Ad or Promotion (performance: 0.89; % improvement: 72.69%), and Marketing (performance: 0.91; % improvement: 72.56%). The most appropriate word-grouping unit (n-gram) was 1 for the majority of classifiers. Performance continued to marginally increase with the size of the training dataset of manually annotated data, but eventually leveled off. Even at low dataset sizes of 4000 observations, performance characteristics were fairly sound.
  
  
  CONCLUSIONS
  Social media outlets like Twitter can uncover real-time snapshots of personal sentiment, knowledge, attitudes, and behavior that are not as accessible, at this scale, through any other offline platform. Using the vast data available through social media presents an opportunity for social science and public health methodologies to utilize computational methodologies to enhance and extend research and practice. This study was successful in automating a complex five-category manual content analysis of e-cigarette-related content on Twitter using machine learning techniques. The study details machine learning model specifications that provided the best accuracy for data related to e-cigarettes, as well as a replicable methodology to allow extension of these methods to additional topics.
authors: ICF International, Rockville, MD, United States.
authors: Heather
authors: http://orcid.org/0000-0002-7275-1810
authors: H
authors: Cole-Lewis
authors: Arun
authors: http://orcid.org/0000-0001-9882-884X
authors: A
authors: Varghese
authors: Amy
authors: http://orcid.org/0000-0002-0164-0208
authors: A
authors: Sanders
authors: Mary
authors: http://orcid.org/0000-0003-0482-7323
authors: M
authors: Schwarz
authors: Jillian
authors: http://orcid.org/0000-0001-9103-5484
authors: J
authors: Pugatch
authors: Erik
authors: http://orcid.org/0000-0002-9723-4078
authors: E
authors: Augustson
mesh_terms: D000465:Algorithms; D001294:Attitude to Health; D066300:Electronic Nicotine Delivery Systems; D006801:Humans; D000069550:Machine Learning; D040541:Marketing; D011634:Public Health; D061108:Social Media
keywords: Twitter; e-cigarette; machine learning; social media