title: Issues of rigor and feasibility when observing the quality of program implementation: a case study.
abstract: Program evaluators have paid little attention in the literature to the manner in which measuring the quality of implementation with observations requires tradeoffs between rigor (reliability and validity) and program evaluation feasibility. We present a case example of how we addressed rigor in light of feasibility concerns when developing and conducting observations for measuring the quality of implementation of a small education professional development program. We discuss the results of meta-evaluative analyses of the reliability of the quality observations, and we present conclusions about conducting observations in a rigorous and feasible manner. The results show that the feasibility constraints that we faced did not notably reduce the rigor of our methods.
authors.affiliation: University of Hawaii at Manoa, Curriculum Research & Development Group, 1776 University Avenue, Honolulu, HI 96822, United States. Electronic address: brandon@hawaii.edu.
authors.forename: Paul R
authors.initials: PR
authors.lastname: Brandon
authors.affiliation: University of Hawaii at Manoa, Curriculum Research & Development Group, 1776 University Avenue, Honolulu, HI 96822, United States. Electronic address: blawton@hawaii.edu.
authors.forename: Brian E
authors.initials: BE
authors.lastname: Lawton
authors.affiliation: University of Hawaii at Manoa, Curriculum Research & Development Group, 1776 University Avenue, Honolulu, HI 96822, United States. Electronic address: georgeha@hawaii.edu.
authors.forename: George M
authors.initials: GM
authors.lastname: Harrison
mesh_terms: D001154:Art; D003363:Cost Control / Q000379:methods; D004517:Education, Professional / Q000191:economics / Q000379:methods / Q000592:standards*; D005240:Feasibility Studies; D006801:Humans; D019370:Observation; D019982:Organizational Case Studies; D011361:Professional Competence / Q000592:standards*; D016730:Program Development / Q000191:economics / Q000379:methods / Q000592:standards*; D015397:Program Evaluation / Q000191:economics / Q000379:methods / Q000592:standards*; D015203:Reproducibility of Results; D013663:Teaching / Q000379:methods
keywords: Observations; Quality of program implementation; Tradeoffs between feasibility and rigor