title: Progress testing: is there a role for the OSCE?
abstract: BACKGROUND
  The shift from a time-based to a competency-based framework in medical education has created a need for frequent formative assessments. Many educational programmes use some form of written progress test to identify areas of strength and weakness and to promote continuous improvement in their learners. However, the role of performance-based assessments, such as objective structured clinical examinations (OSCEs), in progress testing remains unclear.
  
  
  OBJECTIVE
  The aims of this paper are to describe the use of an OSCE to assess learners at different stages of training, describe a structure for reporting scores, and provide evidence for the psychometric properties of different rating tools.
  
  
  METHODS
  A 10-station OSCE was administered to internal medicine residents in postgraduate years (PGYs) 1-4. Candidates were assessed using a checklist (CL), a global rating scale (GRS) and a training level rating scale (TLRS). Reliability was calculated for each measure using Cronbach's alpha. Differences in performance by year of training were explored using analysis of variance (anova). Correlations between scores obtained using the different rating instruments were calculated.
  
  
  RESULTS
  Sixty-nine residents participated in the OSCE. Inter-station reliability was greater (0.88) using the TLRS compared with the CL (0.84) and GRS (0.79). Using all three rating instruments, scores varied significantly by year of training (p < 0.001). Scores from the different rating instruments were highly correlated: CL and GRS, r = 0.93; CL and TLRS, r = 0.90, and GRS and TLRS, r = 0.94 (p < 0.001). Candidates received feedback on their performance relative to examiner expectations for their PGY level.
  
  
  CONCLUSIONS
  Scores were found to have high reliability and demonstrated significant differences in performance by year of training. This provides evidence for the validity of using scores achieved on an OSCE as markers of progress in learners at different levels of training. Future studies will focus on assessing individual progress on the OSCE over time.
authors.affiliation: Department of Medicine, University of Ottawa, Ottawa, Ontario, Canada.
authors.forename: Debra
authors.initials: D
authors.lastname: Pugh
authors.forename: Claire
authors.initials: C
authors.lastname: Touchie
authors.forename: Timothy J
authors.initials: TJ
authors.lastname: Wood
authors.forename: Susan
authors.initials: S
authors.lastname: Humphrey-Murto
mesh_terms: D000704:Analysis of Variance; D002170:Canada; D002983:Clinical Competence / Q000592:standards*; D003162:Competency-Based Education / Q000592:standards*; D004521:Educational Measurement / Q000379:methods / Q000592:standards / Q000706:statistics & numerical data*; D005180:Faculty, Medical; D006801:Humans; D007396:Internship and Residency; D011594:Psychometrics; D015203:Reproducibility of Results