A procedural skills OSCE: assessing technical and non-technical skills of internal medicine residents.
Internists are required to perform a number of procedures that require mastery of technical and non-technical skills, however, formal assessment of these skills is often lacking. The purpose of this study was to develop, implement, and gather validity evidence for a procedural skills objective structured clinical examination (PS-OSCE) for internal medicine (IM) residents to assess their technical and non-technical skills when performing procedures. Thirty-five first to third-year IM residents participated in a 5-station PS-OSCE, which combined partial task models, standardized patients, and allied health professionals. Formal blueprinting was performed and content experts were used to develop the cases and rating instruments. Examiners underwent a frame-of-reference training session to prepare them for their rater role. Scores were compared by levels of training, experience, and to evaluation data from a non-procedural OSCE (IM-OSCE). Reliability was calculated using Generalizability analyses. Reliabilities for the technical and non-technical scores were 0.68 and 0.76, respectively. Third-year residents scored significantly higher than first-year residents on the technical (73.5 vs. 62.2%) and non-technical (83.2 vs. 75.1%) components of the PS-OSCE (p < 0.05). Residents who had performed the procedures more frequently scored higher on three of the five stations (p < 0.05). There was a moderate disattenuated correlation (r = 0.77) between the IM-OSCE and the technical component of the PS-OSCE scores. The PS-OSCE is a feasible method for assessing multiple competencies related to performing procedures and this study provides validity evidence to support its use as an in-training examination.
Department of Medicine, University of Ottawa, Ottawa, ON, Canada, dpugh@toh.on.ca.
Debra
D
Pugh
Stanley J
SJ
Hamstra
Timothy J
TJ
Wood
Susan
S
Humphrey-Murto
Claire
C
Touchie
Rachel
R
Yudkowsky
Georges
G
Bordage
D000328:Adult; D002983:Clinical Competence; D004503:Education, Medical, Graduate / Q000592:standards*; D004521:Educational Measurement / Q000379:methods*; D005260:Female; D006801:Humans; D007388:Internal Medicine / Q000193:education*; D007396:Internship and Residency; D008297:Male; D017145:Models, Educational; D009864:Ontario; D015203:Reproducibility of Results