title: A study of the effectiveness of machine learning methods for classification of clinical interview fragments into a large number of categories.
abstract: This study examines the effectiveness of state-of-the-art supervised machine learning methods in conjunction with different feature types for the task of automatic annotation of fragments of clinical text based on codebooks with a large number of categories. We used a collection of motivational interview transcripts consisting of 11,353 utterances, which were manually annotated by two human coders as the gold standard, and experimented with state-of-art classifiers, including Naïve Bayes, J48 Decision Tree, Support Vector Machine (SVM), Random Forest (RF), AdaBoost, DiscLDA, Conditional Random Fields (CRF) and Convolutional Neural Network (CNN) in conjunction with lexical, contextual (label of the previous utterance) and semantic (distribution of words in the utterance across the Linguistic Inquiry and Word Count dictionaries) features. We found out that, when the number of classes is large, the performance of CNN and CRF is inferior to SVM. When only lexical features were used, interview transcripts were automatically annotated by SVM with the highest classification accuracy among all classifiers of 70.8%, 61% and 53.7% based on the codebooks consisting of 17, 20 and 41 codes, respectively. Using contextual and semantic features, as well as their combination, in addition to lexical ones, improved the accuracy of SVM for annotation of utterances in motivational interview transcripts with a codebook consisting of 17 classes to 71.5%, 74.2%, and 75.1%, respectively. Our results demonstrate the potential of using machine learning methods in conjunction with lexical, semantic and contextual features for automatic annotation of clinical interview transcripts with near-human accuracy.
authors.affiliation: Department of Computer Science, Wayne State University, 5057 Woodward Ave, Detroit, MI 48202, USA.
authors.forename: Mehedi
authors.initials: M
authors.lastname: Hasan
authors.affiliation: Department of Computer Science, Wayne State University, 5057 Woodward Ave, Detroit, MI 48202, USA.
authors.forename: Alexander
authors.initials: A
authors.lastname: Kotov
authors.affiliation: Pediatric Prevention Research Center, School of Medicine, Wayne State University, 540 E Canfield St, Detroit, MI 48201, USA.
authors.forename: April
authors.initials: A
authors.lastname: Carcone
authors.affiliation: Department of Computer Science, Wayne State University, 5057 Woodward Ave, Detroit, MI 48202, USA.
authors.forename: Ming
authors.initials: M
authors.lastname: Dong
authors.affiliation: Pediatric Prevention Research Center, School of Medicine, Wayne State University, 540 E Canfield St, Detroit, MI 48201, USA.
authors.forename: Sylvie
authors.initials: S
authors.lastname: Naar
authors.affiliation: Department of Dietetics and Nutrition, Florida International University, 11200 SW 8th St, Miami, FL 33199, USA.
authors.forename: Kathryn Brogan
authors.initials: KB
authors.lastname: Hartlieb
mesh_terms: D001499:Bayes Theorem; D066289:Data Curation / Q000379:methods*; D003663:Decision Trees; D000069550:Machine Learning; D012660:Semantics; D060388:Support Vector Machine
keywords: Annotation of clinical text; Deep learning; Machine learning; Motivational interviewing; Text classification