title: Reliability and validity of checklists and global ratings by standardized students, trained raters, and faculty raters in an objective structured teaching environment.
abstract: BACKGROUND
  Objective structured teaching exercises (OSTEs) are relatively new in medical education, with few studies that have reported reliability and validity.
  
  
  OBJECTIVE
  To systematically examine the impact of OSTE design decisions, including number of cases, choice of raters, and type of scoring systems used.
  
  
  METHODS
  We examined the impact of number of cases and raters using generalizability theory. We also compared scores from standardized students (SS), faculty raters (FR) and trained graduate student raters (TR), and examined the relation between behavior checklist ratings and global perception scores.
  
  
  RESULTS
  Generalizability (g) coefficients for checklist scores were higher for SSs than TRs. The g estimates based on SSs' global scores were higher than g estimates for FRs. SSs' checklist scores were higher than TRs' checklist scores, and SSs' global evaluations were higher than FRs' and TRs' global scores. TRs' relative to SSs' global perceptions correlated more highly with checklist scores.
  
  
  CONCLUSIONS
  SSs provide more generalizable checklist scores than TRs. Generalizability estimates for global scores from SSs and FRs were comparable. SSs are lenient raters compared to TRs and FRs.
authors.affiliation: University of Massachusetts Medical School, Community Faculty Development Center, Worcester, Massachusetts 01655, USA. Mark.Quirk@umassmed.edu
authors.forename: Mark
authors.initials: M
authors.lastname: Quirk
authors.forename: Kathleen
authors.initials: K
authors.lastname: Mazor
authors.forename: Heather-Lyn
authors.initials: HL
authors.lastname: Haley
authors.forename: Scott
authors.initials: S
authors.lastname: Wellman
authors.forename: David
authors.initials: D
authors.lastname: Keller
authors.forename: David
authors.initials: D
authors.lastname: Hatem
authors.forename: Lisa A
authors.initials: LA
authors.lastname: Keller
mesh_terms: D004501:Education, Medical / Q000592:standards*; D004521:Educational Measurement / Q000592:standards*; D005178:Faculty; D006801:Humans; D015203:Reproducibility of Results; D012107:Research Design; D013337:Students, Medical; D013663:Teaching / Q000379:methods* / Q000592:standards