title: Observational measures of implementer fidelity for a school-based preventive intervention: development, reliability, and validity.
abstract: Current measures of implementer fidelity often fail to adequately measure core constructs of adherence and competence, and their relationship to outcomes can be mixed. To address these limitations, we used observational methods to assess these constructs and their relationships to proximal outcomes in a randomized trial of a school-based preventive intervention (Rochester Resilience Project) designed to strengthen emotion self-regulation skills in first-third graders with elevated aggressive-disruptive behaviors. Within the intervention group (n = 203), a subsample (n = 76) of students was selected to reflect the overall sample. Implementers were 10 paraprofessionals. Videotaped observations of three lessons from year 1 of the intervention (14 lessons) were coded for each implementer-child dyad on adherence (content) and competence (quality). Using multilevel modeling, we examined how much of the variance in the fidelity measures was attributed to implementer and to the child within implementer. Both measures had large and significant variance accounted for by implementer (competence, 68 %; adherence, 41 %); child within implementer did not account for significant variance indicating that ratings reflected stable qualities of the implementer rather than the child. Raw adherence and competence scores shared 46 % of variance (r = .68). Controlling for baseline differences and age, the amount (adherence) and quality (competence) of program delivered predicted children's enhanced response to the intervention on both child and parent reports after 6 months, but not on teacher report of externalizing behavior. Our findings support the use of multiple observations for measuring fidelity and that adherence and competence are important components of fidelity which could be assessed by many programs using these methods.
authors.affiliation: Department of Psychiatry, School of Medicine and Dentistry, University of Rochester, Rochester, NY, USA, wendi_cross@urmc.rochester.edu.
authors.forename: Wendi
authors.initials: W
authors.lastname: Cross
authors.forename: Jennifer
authors.initials: J
authors.lastname: West
authors.forename: Peter A
authors.initials: PA
authors.lastname: Wyman
authors.forename: Karen
authors.initials: K
authors.lastname: Schmeelk-Cone
authors.forename: Yinglin
authors.initials: Y
authors.lastname: Xia
authors.forename: Xin
authors.initials: X
authors.lastname: Tu
authors.forename: Michael
authors.initials: M
authors.lastname: Teisl
authors.forename: C Hendricks
authors.initials: CH
authors.lastname: Brown
authors.forename: Marion
authors.initials: M
authors.lastname: Forgatch
mesh_terms: D000328:Adult; D002648:Child; D005260:Female; D006801:Humans; D008297:Male; D008875:Middle Aged; D002653:Child Behavior Disorders / Q000517:prevention & control; D016730:Program Development; D015397:Program Evaluation; D015203:Reproducibility of Results; D012572:School Health Services / Q000458:organization & administration; D014743:Videotape Recording
references.citation: Health Educ Behav. 2004 Apr;31(2):242-57
references.pmid: 15090124
references.citation: Psychol Assess. 2013 Mar;25(1):146-56
references.pmid: 22888981
references.citation: Early Child Res Q. 2010 3rd Quarter;25(3):329-347
references.pmid: 34531638
references.citation: Drug Alcohol Depend. 2000 Jan 1;57(3):225-38
references.pmid: 10661673
references.citation: Prev Sci. 2011 Sep;12(3):235-46
references.pmid: 21671090
references.citation: Health Educ Res. 2010 Aug;25(4):531-41
references.pmid: 20142414
references.citation: Clin Psychol (New York). 2011 Jun;18(2):142-147
references.pmid: 21691439
references.citation: J Pers Assess. 2003 Jun;80(3):217-22
references.pmid: 12763696
references.citation: Implement Sci. 2009 Aug 07;4:50
references.pmid: 19664226
references.citation: Implement Sci. 2008 Mar 19;3:17
references.pmid: 18353186
references.citation: J Abnorm Child Psychol. 2010 Jul;38(5):683-94
references.pmid: 20180011
references.citation: Am J Community Psychol. 1991 Aug;19(4):585-602
references.pmid: 1755437
references.citation: J Abnorm Child Psychol. 2010 Jul;38(5):707-20
references.pmid: 20180009
references.citation: J Consult Clin Psychol. 1993 Aug;61(4):620-30
references.pmid: 8370857
references.citation: Am J Public Health. 2012 Jul;102(7):1274-81
references.pmid: 22594758
references.citation: Adm Policy Ment Health. 2011 Jan;38(1):32-43
references.pmid: 20957425
references.citation: Implement Sci. 2008 May 29;3:30
references.pmid: 18510750
references.citation: J Child Serv. 2011;6(1):18-33
references.pmid: 21922026
references.citation: Prev Sci. 2006 Mar;7(1):43-56
references.pmid: 16572301
references.citation: Dev Psychopathol. 1998 Spring;10(2):165-85
references.pmid: 9635220
references.citation: Adm Policy Ment Health. 2011 Jan;38(1):4-23
references.pmid: 21197565
references.citation: Am J Community Psychol. 2008 Jun;41(3-4):327-50
references.pmid: 18322790
references.citation: J Consult Clin Psychol. 2010 Apr;78(2):200-11
references.pmid: 20350031
references.citation: Am J Public Health. 1999 Sep;89(9):1322-7
references.pmid: 10474547
references.citation: Nurs Res. 2010 May-Jun;59(3):158-65
references.pmid: 20404777
references.citation: Suicide Life Threat Behav. 2013 Dec;43(6):676-91
references.pmid: 23889494
references.citation: Behav Ther. 2005;36(4):357-65
references.pmid: 16718303
references.citation: Med Care Res Rev. 2012 Apr;69(2):123-57
references.pmid: 22203646
references.citation: Adm Policy Ment Health. 2011 Mar;38(2):65-76
references.pmid: 20957426
references.citation: J Consult Clin Psychol. 2008 Aug;76(4):544-55
references.pmid: 18665684
references.citation: Adm Policy Ment Health. 2011 Jul;38(4):313-23
references.pmid: 21626444
references.citation: J Am Acad Child Adolesc Psychiatry. 2000 Jan;39(1):85-93
references.pmid: 10638071
references.citation: Implement Sci. 2012 Apr 18;7:32
references.pmid: 22512914
references.citation: Am J Prev Med. 2001 Jan;20(1 Suppl):38-47
references.pmid: 11146259