title: Comparison of a rational and an empirical standard setting procedure for an OSCE. Objective structured clinical examinations.
abstract: OBJECTIVE
  Earlier studies of absolute standard setting procedures for objective structured clinical examinations (OSCEs) show inconsistent results. This study compared a rational and an empirical standard setting procedure. Reliability and credibility were examined first. The impact of a reality check was then established.
  
  
  METHODS
  The OSCE included 16 stations and was taken by trainees in their final year of postgraduate training in general practice and experienced general practitioners. A modified Angoff (independent judgements, no group discussion) with and without a reality check was used as a rational procedure. A method related to the borderline group procedure, the borderline regression (BR) method, was used as an empirical procedure. Reliability was assessed using generalisability theory. Credibility was assessed by comparing pass rates and by relating the passing scores to test difficulty.
  
  
  RESULTS
  The passing scores were 73.4% for the Angoff procedure without reality check (Angoff I), 66.0% for the Angoff procedure with reality check (Angoff II) and 57.6% for the BR method. The reliabilities (expressed as root mean square errors) were 2.1% for Angoffs I and II, and 0.6% for the BR method. The pass rates of the trainees and GPs were 19% and 9% for Angoff I, 66% and 46% for Angoff II, and 95% and 80% for the BR method, respectively. The correlation between test difficulty and passing score was 0.69 for Angoff I, 0.88 for Angoff II and 0.86 for the BR method.
  
  
  CONCLUSIONS
  The BR method provides a more credible and reliable standard for an OSCE than a modified Angoff procedure. A reality check improves the credibility of the Angoff procedure but does not improve its reliability.
authors.affiliation: National Centre for Evaluation of Postgraduate Training in General Practice (SVUH), Utrecht, the Netherlands. Carol_ann@tiro.nl
authors.forename: Anneke
authors.initials: A
authors.lastname: Kramer
authors.forename: Arno
authors.initials: A
authors.lastname: Muijtjens
authors.forename: Koos
authors.initials: K
authors.lastname: Jansen
authors.forename: Herman
authors.initials: H
authors.lastname: Düsman
authors.forename: Lisa
authors.initials: L
authors.lastname: Tan
authors.forename: Cees
authors.initials: C
authors.lastname: van der Vleuten
mesh_terms: D002983:Clinical Competence / Q000592:standards*; D004501:Education, Medical / Q000592:standards*; D004521:Educational Measurement; D006801:Humans; D015203:Reproducibility of Results