title: Comparison of checklist and anchored global rating instruments for performance rating of simulated pediatric emergencies.
abstract: OBJECTIVE
  To compare the psychometric performance of two rating instruments used to assess trainee performance in three clinical scenarios.
  
  
  METHODS
  This study was part of a two-phase, randomized trial with a wait-list control condition assessing the effectiveness of a pediatric emergency medicine curriculum targeting general emergency medicine residents. Residents received 6 hours of instruction either before or after the first assessment. Separate pairs of raters completed either a dichotomous checklist for each of three cases or the Global Performance Assessment Tool (GPAT), an anchored multidimensional scale. A fully crossed person×rater×case generalizability study was conducted. The effect of training year on performance is assessed using multivariate analysis of variance.
  
  
  RESULTS
  The person and person×case components accounted for most of the score variance for both instruments. Using either instrument, scores demonstrated a small but significant increase as training level increased when analyzed using a multivariate analysis of variance. The inter-rater reliability coefficient was >0.9 for both instruments.
  
  
  CONCLUSIONS
  We demonstrate that our checklist and anchored global rating instrument performed in a psychometrically similar fashion with high reliability. As long as proper attention is given to instrument design and testing and rater training, checklists and anchored assessment scales can produce reproducible data for a given population of subjects. The validity of the data arising for either instrument type must be assessed rigorously and with a focus, when practicable, on patient care outcomes.
authors.affiliation: Department of Pediatrics, Feinberg School of Medicine, Augusta Webster, MD, USA. m-adler@northwestern.edu
authors.forename: Mark D
authors.initials: MD
authors.lastname: Adler
authors.forename: John A
authors.initials: JA
authors.lastname: Vozenilek
authors.forename: Jennifer L
authors.initials: JL
authors.lastname: Trainor
authors.forename: Walter J
authors.initials: WJ
authors.lastname: Eppich
authors.forename: Ernest E
authors.initials: EE
authors.lastname: Wang
authors.forename: Jennifer L
authors.initials: JL
authors.lastname: Beaumont
authors.forename: Pamela R
authors.initials: PR
authors.lastname: Aitchison
authors.forename: Paul J
authors.initials: PJ
authors.lastname: Pribaz
authors.forename: Timothy
authors.initials: T
authors.lastname: Erickson
authors.forename: Marcia
authors.initials: M
authors.lastname: Edison
authors.forename: William C
authors.initials: WC
authors.lastname: McGaghie
mesh_terms: D057189:Checklist; D002983:Clinical Competence; D003198:Computer Simulation; D004521:Educational Measurement / Q000379:methods*; D004635:Emergency Medicine / Q000193:education*; D006801:Humans; D007396:Internship and Residency / Q000379:methods*; D008348:Manikins; D010372:Pediatrics / Q000193:education*; D011594:Psychometrics