title: Can student self-ratings be compared with peer ratings? A study of measurement invariance of multisource feedback.
abstract: Measurement invariance is a prerequisite for comparing measurement scores from different groups. In medical education, multi-source feedback (MSF) is utilized to assess core competencies, including the professionalism. However, little attention has been paid to the measurement invariance of assessment instruments; that is, whether an instrument holds the same meaning across different rater groups. To examine the measurement invariance of the National Taiwan University professionalism MSF (NTU P-MSF) in order to determine whether medical students' self-rating can be compared to their peers' rating. An eight-factor model was specified for confirmatory factor analysis to examine the construct validity of the NTU P-MSF. Cronbach's alpha was computed for the items of each domain to evaluate internal consistent reliability. The same eight-factor model was used for multi-group confirmatory factor analyses. Four hierarchical models were specified to test configural (i.e., identical factor-item relationship), metric (i.e., identical factor loadings), scalar (i.e., identical intercepts), and error variance across self-rating and peer rating groups. One hundred and twenty second-year medical students from weekly discussion groups conducted as part of a medical professionalism course agreed to use the NTU P-MSF to assess themselves or their discussion group peers. NTU P-MSF assessment scores were a good fit for the eight-factor model among self group and peer group. The Cronbach's alpha coefficients of students' NTU P-MSF scores and peers' scores ranged from 0.76 to 0.89 and 0.84 to 0.91, respectively indicating that the NTU P-MSF scores also have good internal consistent reliability between both groups. In addition, same factor structure and similar factor loadings and intercepts of NTU P-MSF scores between both groups indicate that NTU P-MSF scores had configural, metric, and scalar invariance. Thus, students' self-assessments and peer assessments can be compared in terms of the constructs of NTU P-MSF scores, change in NTU P-MSF scores, and its factor scores. This study demonstrates how to investigate the measurement invariance of a professionalism MSF and contributes to the discussion on self- and peer assessment in medical education.
authors.affiliation: Department of Medical Education and Bioethics, National Taiwan University College of Medicine, No. 1, Ren-Ai Road, Section 1, Taipei, Taiwan.
authors.forename: Keng-Lin
authors.initials: KL
authors.lastname: Lee
authors.affiliation: Department of Medical Education and Bioethics, National Taiwan University College of Medicine, No. 1, Ren-Ai Road, Section 1, Taipei, Taiwan.
authors.forename: Shih-Li
authors.initials: SL
authors.lastname: Tsai
authors.affiliation: Department of Medical Education and Bioethics, National Taiwan University College of Medicine, No. 1, Ren-Ai Road, Section 1, Taipei, Taiwan.
authors.forename: Yu-Ting
authors.initials: YT
authors.lastname: Chiu
authors.affiliation: Department of Medical Education and Bioethics, National Taiwan University College of Medicine, No. 1, Ren-Ai Road, Section 1, Taipei, Taiwan. mjho@ntu.edu.tw.
authors.forename: Ming-Jung
authors.identifier: http://orcid.org/0000-0003-1415-8282
authors.initials: MJ
authors.lastname: Ho
mesh_terms: D004521:Educational Measurement / Q000379:methods* / Q000592:standards; D005260:Female; D000068156:Formative Feedback; D006801:Humans; D008297:Male; D010379:Peer Group; D011594:Psychometrics; D015203:Reproducibility of Results; D012647:Self-Assessment; D013337:Students, Medical; D011795:Surveys and Questionnaires; D013624:Taiwan
keywords: Measurement invariance; Multisource feedback; Peer-assessment; Professionalism; Self-assessment