title: Measuring Fidelity and Adaptation: Reliability of a Instrument for School-Based Prevention Programs.
abstract: There is a need to standardize methods for assessing fidelity and adaptation. Such standardization would allow program implementation to be examined in a manner that will be useful for understanding the moderating role of fidelity in dissemination research. This article describes a method for collecting data about fidelity of implementation for school-based prevention programs, including measures of adherence, quality of delivery, dosage, participant engagement, and adaptation. We report about the reliability of these methods when applied by four observers who coded video recordings of teachers delivering All Stars, a middle school drug prevention program. Interrater agreement for scaled items was assessed for an instrument designed to evaluate program fidelity. Results indicated sound interrater reliability for items assessing adherence, dosage, quality of teaching, teacher understanding of concepts, and program adaptations. The interrater reliability for items assessing potential program effectiveness, classroom management, achievement of activity objectives, and adaptation valences was improved by dichotomizing the response options for these items. The item that assessed student engagement demonstrated only modest interrater reliability and was not improved through dichotomization. Several coder pairs were discordant on items that overall demonstrated good interrater reliability. Proposed modifications to the coding manual and protocol are discussed.
authors: Tanglewood Research, Inc., Greensboro, NC, USA dana@tanglewood.net.
authors: Dana C
authors: DC
authors: Bishop
authors: The Pacific Institute for Research and Evaluation, Chapel Hill, NC, USA.
authors: Melinda M
authors: MM
authors: Pankratz
authors: Tanglewood Research, Inc., Greensboro, NC, USA.
authors: William B
authors: WB
authors: Hansen
authors: Tanglewood Research, Inc., Greensboro, NC, USA.
authors: Jordan
authors: J
authors: Albritton
authors: Tanglewood Research, Inc., Greensboro, NC, USA.
authors: Lauren
authors: L
authors: Albritton
authors: Tanglewood Research, Inc., Greensboro, NC, USA.
authors: Joann
authors: J
authors: Strack
mesh_terms: D000293:Adolescent; D003625:Data Collection / Q000379:methods; D006801:Humans; D017146:Models, Organizational; D015397:Program Evaluation; D015203:Reproducibility of Results; D012572:School Health Services / Q000458:organization & administration / Q000592:standards; D019966:Substance-Related Disorders / Q000517:prevention & control
keywords: adaptation; fidelity; fidelity measurement; observers; prevention; reliability