title: The reliability of multisource feedback in competency-based assessment programs: the effects of multiple occasions and assessor groups.
abstract: OBJECTIVE
  Residency programs around the world use multisource feedback (MSF) to evaluate learners' performance. Studies of the reliability of MSF show mixed results. This study aimed to identify the reliability of MSF as practiced across occasions with varying numbers of assessors from different professional groups (physicians and nonphysicians) and the effect on the reliability of the assessment for different competencies when completed by both groups.
  
  
  METHODS
  The authors collected data from 2008 to 2012 from electronically completed MSF questionnaires. In total, 428 residents completed 586 MSF occasions, and 5,020 assessors provided feedback. The authors used generalizability theory to analyze the reliability of MSF for multiple occasions, different competencies, and varying numbers of assessors and assessor groups across multiple occasions.
  
  
  RESULTS
  A reliability coefficient of 0.800 can be achieved with two MSF occasions completed by at least 10 assessors per group or with three MSF occasions completed by 5 assessors per group. Nonphysicians' scores for the "Scholar" and "Health advocate" competencies and physicians' scores for the "Health advocate" competency had a negative effect on the composite reliability.
  
  
  CONCLUSIONS
  A feasible number of assessors per MSF occasion can reliably assess residents' performance. Scores from a single occasion should be interpreted cautiously. However, every occasion can provide valuable feedback for learning. This research confirms that the (unique) characteristics of different assessor groups should be considered when interpreting MSF results. Reliability seems to be influenced by the included assessor groups and competencies. These findings will enhance the utility of MSF during residency training.
authors.affiliation: J.M.W. Moonen-van Loon is postdoctoral researcher, Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands. K. Overeem is postdoctoral researcher, Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands. M.J.B. Govaerts is assistant professor, Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands. B.H. Verhoeven is pediatric surgeon, Department of Surgery, Radboud University Medical Center, Nijmegen, and assistant professor, Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands. C.P.M. van der Vleuten is professor of education, Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands. E.W. Driessen is associate professor of education, Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands.
authors.forename: Joyce M W
authors.initials: JM
authors.lastname: Moonen-van Loon
authors.forename: Karlijn
authors.initials: K
authors.lastname: Overeem
authors.forename: Marjan J B
authors.initials: MJ
authors.lastname: Govaerts
authors.forename: Bas H
authors.initials: BH
authors.lastname: Verhoeven
authors.forename: Cees P M
authors.initials: CP
authors.lastname: van der Vleuten
authors.forename: Erik W
authors.initials: EW
authors.lastname: Driessen
mesh_terms: D000328:Adult; D002983:Clinical Competence; D003625:Data Collection / Q000379:methods; D004503:Education, Medical, Graduate; D004650:Employee Performance Appraisal / Q000379:methods*; D005180:Faculty, Medical; D005246:Feedback; D005260:Female; D006761:Hospitals; D006801:Humans; D007396:Internship and Residency; D007400:Interprofessional Relations; D008297:Male; D009426:Netherlands; D018024:Peer Review, Health Care; D015203:Reproducibility of Results; D011795:Surveys and Questionnaires