title: A Comparison of Natural Language Processing Methods for Automated Coding of Motivational Interviewing.
abstract: Motivational interviewing (MI) is an efficacious treatment for substance use disorders and other problem behaviors. Studies on MI fidelity and mechanisms of change typically use human raters to code therapy sessions, which requires considerable time, training, and financial costs. Natural language processing techniques have recently been utilized for coding MI sessions using machine learning techniques, rather than human coders, and preliminary results have suggested these methods hold promise. The current study extends this previous work by introducing two natural language processing models for automatically coding MI sessions via computer. The two models differ in the way they semantically represent session content, utilizing either 1) simple discrete sentence features (DSF model) and 2) more complex recursive neural networks (RNN model). Utterance- and session-level predictions from these models were compared to ratings provided by human coders using a large sample of MI sessions (N=341 sessions; 78,977 clinician and client talk turns) from 6 MI studies. Results show that the DSF model generally had slightly better performance compared to the RNN model. The DSF model had "good" or higher utterance-level agreement with human coders (Cohen's kappa>0.60) for open and closed questions, affirm, giving information, and follow/neutral (all therapist codes); considerably higher agreement was obtained for session-level indices, and many estimates were competitive with human-to-human agreement. However, there was poor agreement for client change talk, client sustain talk, and therapist MI-inconsistent behaviors. Natural language processing methods provide accurate representations of human derived behavioral codes and could offer substantial improvements to the efficiency and scale in which MI mechanisms of change research and fidelity monitoring are conducted.
authors.affiliation: University of UT, Department of Educational Psychology and Social Research Institute, 395 South 1500 East #111, United States. Electronic address: Michael.Tanana@utah.edu.
authors.forename: Michael
authors.initials: M
authors.lastname: Tanana
authors.affiliation: University of Washington, Department of Psychiatry and Behavioral Sciences, Box 354944, Seattle, WA 98195-4944, United States. Electronic address: khallgre@uw.edu.
authors.forename: Kevin A
authors.initials: KA
authors.lastname: Hallgren
authors.affiliation: University of Utah, Department of Psychology, 1705 Campus Center Drive, Room 327, Salt Lake City, UT United States. Electronic address: zac.imel@utah.edu.
authors.forename: Zac E
authors.initials: ZE
authors.lastname: Imel
authors.affiliation: University of Washington, Department of Psychiatry and Behavioral Sciences, Box 354944, Seattle, WA 98195-4944, United States. Electronic address: datkins@uw.edu.
authors.forename: David C
authors.initials: DC
authors.lastname: Atkins
authors.affiliation: University of Utah, School of Computing, 50S. Central Campus Drive Room 3190, Salt Lake City, UT, United States. Electronic address: svivek@cs.utah.edu.
authors.forename: Vivek
authors.initials: V
authors.lastname: Srikumar
mesh_terms: D001330:Electronic Data Processing / Q000379:methods*; D005260:Female; D006282:Health Personnel; D006801:Humans; D008297:Male; D062405:Motivational Interviewing / Q000379:methods*; D009323:Natural Language Processing; D019966:Substance-Related Disorders / Q000628:therapy
keywords: Behavioral coding; Discrete sentence feature model; Machine learning; Motivational interviewing; Natural language processing; Recursive neural network; Treatment integrity