Automatic classification of communication logs into implementation stages via text analysis.
To improve the quality, quantity, and speed of implementation, careful monitoring of the implementation process is required. However, some health organizations have such limited capacity to collect, organize, and synthesize information relevant to its decision to implement an evidence-based program, the preparation steps necessary for successful program adoption, the fidelity of program delivery, and the sustainment of this program over time. When a large health system implements an evidence-based program across multiple sites, a trained intermediary or broker may provide such monitoring and feedback, but this task is labor intensive and not easily scaled up for large numbers of sites. We present a novel approach to producing an automated system of monitoring implementation stage entrances and exits based on a computational analysis of communication log notes generated by implementation brokers. Potentially discriminating keywords are identified using the definitions of the stages and experts' coding of a portion of the log notes. A machine learning algorithm produces a decision rule to classify remaining, unclassified log notes.



We applied this procedure to log notes in the implementation trial of multidimensional treatment foster care in the California 40-county implementation trial (CAL-40) project, using the stages of implementation completion (SIC) measure. We found that a semi-supervised non-negative matrix factorization method accurately identified most stage transitions. Another computational model was built for determining the start and the end of each stage.



This automated system demonstrated feasibility in this proof of concept challenge. We provide suggestions on how such a system can be used to improve the speed, quality, quantity, and sustainment of implementation. The innovative methods presented here are not intended to replace the expertise and judgement of an expert rater already in place. Rather, these can be used when human monitoring and feedback is too expensive to use or maintain. These methods rely on digitized text that already exists or can be collected with minimal to no intrusiveness and can signal when additional attention or remediation is required during implementation. Thus, resources can be allocated according to need rather than universally applied, or worse, not applied at all due to their cost.
Department of Computer Science, Florida Atlantic University, 777 Glades Road EE 403, Boca Raton, FL, USA.
Dingding
D
Wang
Department of Computer Science and Center for Computational Science, University of Miami, 1320 S. Dixie Highway, Miami, FL, USA.
Mitsunori
M
Ogihara
Center for Prevention Implementation Methodology, Department of Psychiatry and Behavioral Sciences, Feinberg School of Medicine Northwestern University, 750 N. Lake Shore Dr., Chicago, IL, USA.
Carlos
C
Gallo
Center for Prevention Implementation Methodology, Department of Psychiatry and Behavioral Sciences, Feinberg School of Medicine Northwestern University, 750 N. Lake Shore Dr., Chicago, IL, USA.
Juan A
JA
Villamar
Center for Prevention Implementation Methodology, Department of Psychiatry and Behavioral Sciences, Feinberg School of Medicine Northwestern University, 750 N. Lake Shore Dr., Chicago, IL, USA.
Justin D
JD
Smith
Center for Prevention Implementation Methodology, Department of Psychiatry and Behavioral Sciences, Feinberg School of Medicine Northwestern University, 750 N. Lake Shore Dr., Chicago, IL, USA.
Wouter
W
Vermeer
Department of Health Policy and Management, University of North Carolina, Chapel Hill, 135 Dauer Drive, Chapel Hill, NC, USA.
Gracelyn
G
Cruden
Center for Prevention Implementation Methodology, Department of Psychiatry and Behavioral Sciences, Feinberg School of Medicine Northwestern University, 750 N. Lake Shore Dr., Chicago, IL, USA.
Nanette
N
Benbow
Center for Prevention Implementation Methodology, Department of Psychiatry and Behavioral Sciences, Feinberg School of Medicine Northwestern University, 750 N. Lake Shore Dr., Chicago, IL, USA. hendricks.brown@northwestern.edu.
C Hendricks
0000-0002-0294-2419
CH
Brown
D002140:California; D003142:Communication; D003198:Computer Simulation; D057225:Data Mining; D004059:Diffusion of Innovation; D005240:Feasibility Studies; D005581:Foster Home Care; D006801:Humans; D016247:Information Storage and Retrieval; D000069550:Machine Learning; D008433:Mathematics; D008490:Medical Informatics / Q000379:methods*; D011996:Records; D012680:Sensitivity and Specificity; D057170:Translational Research, Biomedical
Machine learning; Social systems informatics; Text mining; Unobtrusive measures