Setting standards for teaching evaluation data: an application of the contrasting groups method.
BACKGROUND
Teaching evaluations are widely used in retention and remediation decisions. Typically evaluations are reviewed in a global manner and some gestalt is reached.


OBJECTIVE
Apply the Contrasting Groups standard setting methodology to faculty teaching dossiers, to examine resulting "pass-rates" and precision of the decisions.


METHODS
Ten faculty judges set standards for teaching dossiers using a Contrasting Groups approach. Blinded dossiers summarizing clinical (N = 47) and classroom teaching (N = 37) were sorted into piles labeled Unsatisfactory, Satisfactory, Excellent, and Superior. Cut-points were the midpoints between aggregated judge-level mean performances of dossiers placed within adjacent levels.


RESULTS
For the total faculty, the percentage assigned to groups labeled Unsatisfactory through Superior, were 4.1%, 5.9%, 26.1%, and 63.9% and 6.6%, 24.8%, 44.1%, and 24.5% for clinical and classroom teaching respectively. Standard error of measurement was between .20 and .25 with 5 to 7 judges.


CONCLUSIONS
Standard setting methods applied to faculty evaluation data produce precise results. Future work can examine standard stability and acceptability, and methods to combine across teaching venues.
Shea
Judy A
JA
Department of Medicine, University of Pennsylvania, 423 Guardian Drive, Philadelphia, PA 19104-6021, USA. sheaja@mail.med.upenn.edu
Bellini
Lisa M
LM
McOwen
Katherine S
KS
Norcini
John J
JJ