title: Reliability and credibility of an angoff standard setting procedure in progress testing using recent graduates as judges.
abstract: BACKGROUND
  Progress testing is an assessment method that samples the complete domain of knowledge that is considered pertinent to undergraduate medical education. Because of the comprehensive nature of this test, it is very difficult to set a passing score. We obtained a progress test standard using an Angoff procedure with recent graduates as judges. This paper reports on the reliability and credibility of this approach.
  
  
  METHODS
  The Angoff procedure was applied to a sample of 146 progress test items. The items were judged by a panel of eight recently graduated students. Generalizability theory was used to investigate the reliability as a function of the number of items and judges. Credibility was judged by comparing the pass/fail rates resulting from the standard arrived at by the Angoff procedure with those obtained using a relative and a fixed standard.
  
  
  RESULTS
  The results indicate that an acceptable error score can be achieved, yielding a precision within one percentage on the scoring scale, by using 10 judges on a full-length progress test (i.e. 250 items). The pass/fail rates associated with the Angoff standard came closest to those of the relative standard, which takes variations in test difficulty into account. A high correlation was found between item-Angoff estimates and the item P-values.
  
  
  CONCLUSIONS
  The results of this study suggest that the Angoff procedure, using recently graduated students as judges, is an appropriate standard setting method for a progress test.
authors.affiliation: Department: Skillslab, University of Maastricht, The Netherlands.
authors.forename: B H
authors.initials: BH
authors.lastname: Verhoeven
authors.forename: A F
authors.initials: AF
authors.lastname: van der Steeg
authors.forename: A J
authors.initials: AJ
authors.lastname: Scherpbier
authors.forename: A M
authors.initials: AM
authors.lastname: Muijtjens
authors.forename: G M
authors.initials: GM
authors.lastname: Verwijnen
authors.forename: C P
authors.initials: CP
authors.lastname: van der Vleuten
mesh_terms: D004504:Education, Medical, Undergraduate / Q000379:methods*; D004521:Educational Measurement / Q000379:methods*; D006801:Humans; D018794:Problem-Based Learning; D012680:Sensitivity and Specificity