title: Setting standards for teaching evaluation data: an application of the contrasting groups method.
abstract: BACKGROUND
  Teaching evaluations are widely used in retention and remediation decisions. Typically evaluations are reviewed in a global manner and some gestalt is reached.
  
  
  OBJECTIVE
  Apply the Contrasting Groups standard setting methodology to faculty teaching dossiers, to examine resulting "pass-rates" and precision of the decisions.
  
  
  METHODS
  Ten faculty judges set standards for teaching dossiers using a Contrasting Groups approach. Blinded dossiers summarizing clinical (N = 47) and classroom teaching (N = 37) were sorted into piles labeled Unsatisfactory, Satisfactory, Excellent, and Superior. Cut-points were the midpoints between aggregated judge-level mean performances of dossiers placed within adjacent levels.
  
  
  RESULTS
  For the total faculty, the percentage assigned to groups labeled Unsatisfactory through Superior, were 4.1%, 5.9%, 26.1%, and 63.9% and 6.6%, 24.8%, 44.1%, and 24.5% for clinical and classroom teaching respectively. Standard error of measurement was between .20 and .25 with 5 to 7 judges.
  
  
  CONCLUSIONS
  Standard setting methods applied to faculty evaluation data produce precise results. Future work can examine standard stability and acceptability, and methods to combine across teaching venues.
authors.affiliation: Department of Medicine, University of Pennsylvania, 423 Guardian Drive, Philadelphia, PA 19104-6021, USA. sheaja@mail.med.upenn.edu
authors.forename: Judy A
authors.initials: JA
authors.lastname: Shea
authors.forename: Lisa M
authors.initials: LM
authors.lastname: Bellini
authors.forename: Katherine S
authors.initials: KS
authors.lastname: McOwen
authors.forename: John J
authors.initials: JJ
authors.lastname: Norcini