Improving the residency admissions process by integrating a professionalism assessment: a validity and feasibility study.
The purpose of this study was to provide validity and feasibility evidence in measuring professionalism using the Professionalism Mini-Evaluation Exercise (P-MEX) scores as part of a residency admissions process. In 2012 and 2013, three standardized-patient-based P-MEX encounters were administered to applicants invited for an interview at the University of Geneva Pediatrics Residency Program. Validity evidence was gathered for P-MEX content (item analysis); response process (qualitative feedback); internal structure (inter-rater reliability with intraclass correlation and Generalizability); relations to other variables (correlations); and consequences (logistic regression to predict admission). To improve reliability, Kane's formula was used to create an applicant composite score using P-MEX, structured letter of recommendation (SLR), and structured interview (SI) scores. Applicant rank lists using composite scores versus faculty global ratings were compared using the Wilcoxon signed-rank test. Seventy applicants were assessed. Moderate associations were found between pairwise correlations of P-MEX scores and SLR (r = 0.25, P = .036), SI (r = 0.34, P = .004), and global ratings (r = 0.48, P < .001). Generalizability of the P-MEX using three cases was moderate (G-coefficient = 0.45). P-MEX scores had the greatest correlation with acceptance (r = 0.56, P < .001), were the strongest predictor of acceptance (OR 4.37, P < .001), and increased pseudo R-squared by 0.20 points. Including P-MEX scores increased composite score reliability from 0.51 to 0.74. Rank lists of applicants using composite score versus global rating differed significantly (z = 5.41, P < .001). Validity evidence supports the use of P-MEX scores to improve the reliability of the residency admissions process by improving applicant composite score reliability.
Department of General Pediatrics, Hôpital des Enfants, Hôpitaux Universitaires de Genève, Geneva, Switzerland. nadia.bajwa@hcuge.ch.|Département de l'enfant et de l'adolescent, Service de pédiatrie générale, Hôpital des Enfants, Hôpitaux Universitaires de Genève, Rue Willy Donzé 6, 1211, Geneva 14, Switzerland. nadia.bajwa@hcuge.ch.
Nadia M
NM
Bajwa
Department of Medical Education, College of Medicine, University of Illinois at Chicago, Chicago, IL, USA.
Rachel
R
Yudkowsky
Department of General Pediatrics, Hôpital des Enfants, Hôpitaux Universitaires de Genève, Geneva, Switzerland.
Dominique
D
Belli
Development and Research in Medical Education, Faculty of Medicine, University of Geneva, Geneva, Switzerland.
Nu Viet
NV
Vu
Department of Medical Education, College of Medicine, University of Illinois at Chicago, Chicago, IL, USA.
Yoon Soo
YS
Park
D002983:Clinical Competence / Q000592:standards; D004521:Educational Measurement / Q000379:methods*; D005240:Feasibility Studies; D006801:Humans; D007396:Internship and Residency / Q000458:organization & administration* / Q000592:standards; D007407:Interviews as Topic; D058996:Quality Improvement; D012570:School Admission Criteria
Assessment; Professionalism; Resident admissions; Validity