title: Measuring stakeholder participation in evaluation: an empirical validation of the Participatory Evaluation Measurement Instrument (PEMI).
abstract: BACKGROUND
  Stakeholder participation is an important trend in the field of program evaluation. Although a few measurement instruments have been proposed, they either have not been empirically validated or do not cover the full content of the concept.
  
  
  OBJECTIVE
  This study consists of a first empirical validation of a measurement instrument that fully covers the content of participation, namely the Participatory Evaluation Measurement Instrument (PEMI). It specifically examines (1) the intercoder reliability of scores derived by two research assistants on published evaluation cases; (2) the convergence between the scores of coders and those of key respondents (i.e., authors); and (3) the convergence between the authors' scores on the PEMI and the Evaluation Involvement Scale (EIS).
  
  
  METHODS
  A purposive sample of 40 cases drawn from the evaluation literature was used to assess reliability. One author per case in this sample was then invited to participate in a survey; 25 fully usable questionnaires were received.
  
  
  METHODS
  Stakeholder participation was measured on nominal and ordinal scales. Cohen's κ, the intraclass correlation coefficient, and Spearman's ρ were used to assess reliability and convergence.
  
  
  RESULTS
  Reliability results ranged from fair to excellent. Convergence between coders' and authors' scores ranged from poor to good. Scores derived from the PEMI and the EIS were moderately associated.
  
  
  CONCLUSIONS
  Evidence from this study is strong in the case of intercoder reliability and ranges from weak to strong in the case of convergent validation. Globally, this suggests that the PEMI can produce scores that are both reliable and valid.
authors.affiliation: Department of Political Science, Université Laval, Quebec City, Quebec, Canada G1V 0A6. pierre-marc.daigneault.1@ulaval.ca
authors.forename: Pierre-Marc
authors.initials: PM
authors.lastname: Daigneault
authors.forename: Steve
authors.initials: S
authors.lastname: Jacob
authors.forename: Joël
authors.initials: J
authors.lastname: Tremblay
mesh_terms: D055812:Community-Based Participatory Research / Q000379:methods* / Q000592:standards; D006801:Humans; D015588:Observer Variation; D015397:Program Evaluation / Q000379:methods*; D011594:Psychometrics / Q000295:instrumentation*; D015203:Reproducibility of Results