Blending Qualitative and Computational Linguistics Methods for Fidelity Assessment: Experience with the Familias Unidas Preventive Intervention.
Careful fidelity monitoring and feedback are critical to implementing effective interventions. A wide range of procedures exist to assess fidelity; most are derived from observational assessments (Schoenwald and Garland, Psycholog Assess 25:146-156, 2013). However, these fidelity measures are resource intensive for research teams in efficacy/effectiveness trials, and are often unattainable or unmanageable for the host organization to rate when the program is implemented on a large scale. We present a first step towards automated processing of linguistic patterns in fidelity monitoring of a behavioral intervention using an innovative mixed methods approach to fidelity assessment that uses rule-based, computational linguistics to overcome major resource burdens. Data come from an effectiveness trial of the Familias Unidas intervention, an evidence-based, family-centered preventive intervention found to be efficacious in reducing conduct problems, substance use and HIV sexual risk behaviors among Hispanic youth. This computational approach focuses on "joining," which measures the quality of the working alliance of the facilitator with the family. Quantitative assessments of reliability are provided. Kappa scores between a human rater and a machine rater for the new method for measuring joining reached 0.83. Early findings suggest that this approach can reduce the high cost of fidelity measurement and the time delay between fidelity assessment and feedback to facilitators; it also has the potential for improving the quality of intervention fidelity ratings.
Department of Psychiatry and Behavioral Sciences, Fienberg School of Medicine, Northwestern University, 680 N. Lake Shore Drive, Suite 1400, Chicago, IL, 60611, USA, carlos.gallo@northwestern.edu.
Carlos
C
Gallo
Hilda
H
Pantin
Juan
J
Villamar
Guillermo
G
Prado
Maria
M
Tapia
Mitsunori
M
Ogihara
Gracelyn
G
Cruden
C Hendricks
CH
Brown
D000293:Adolescent; D019955:Conduct Disorder / Q000517:prevention & control; D055317:Evidence-Based Practice; D005192:Family Health; D005260:Female; D015658:HIV Infections / Q000517:prevention & control; D006302:Health Services Research; D006630:Hispanic or Latino; D006801:Humans; D008037:Linguistics; D000069550:Machine Learning; D008297:Male; D011315:Preventive Medicine; D017064:Process Assessment, Health Care; D011368:Professional-Family Relations; D036301:Qualitative Research; D016032:Randomized Controlled Trials as Topic; D015203:Reproducibility of Results; D012309:Risk-Taking; D012572:School Health Services; D012725:Sexual Behavior; D013223:Statistics as Topic; D047568:Unsafe Sex / Q000517:prevention & control; D014741:Video Recording