title: Queen's simulation assessment tool: development and validation of an assessment tool for resuscitation objective structured clinical examination stations in emergency medicine.
abstract: BACKGROUND
  The use of high-fidelity simulation is emerging as an effective approach to competency-based assessment in medical education. We aimed to develop and validate a modifiable anchored global assessment scoring tool for simulation-based Objective Structured Clinical Examinations (OSCEs) of resuscitation competence in postgraduate emergency medicine (EM) trainees.
  
  
  METHODS
  The Queen's Simulation Assessment Tool was developed using a modified Delphi technique with a panel of EM physicians. Ten standardized resuscitation OSCE scenarios were administered to EM trainees, and their video-recorded performances were scored by 3 independent and blinded EM attending physicians using the Queen's Simulation Assessment Tool. Correlational analyses and analysis of variance were applied to measure the discriminatory capabilities and interrater reliability of each scenario. A fully crossed generalizability study was conducted for each examination.
  
  
  RESULTS
  Emergency medicine postgraduate trainees at Queen's University (n = 19-25 per station) participated in the study over 3 years. Interrater reliability showed acceptable levels of agreement for each scenario (mean Spearman ρ = 0.75 [0.63-0.87]; mean interclass correlation coefficient, 0.69 [0.58-0.87]). Discriminatory validity was strong, with senior residents outperforming junior residents in all but 1 of the 10 scenarios. Generalizability studies found the trainee and trainee by scenario interactions as the largest contributors to variance, with G coefficients ranging from 0.67 to 0.84. Resident trainees reported comfort being assessed in the simulation environment (3.8/5) and found the simulation-based examination valuable to their learning (4.6/5).
  
  
  CONCLUSIONS
  This study describes the development and validation of a novel modifiable anchored global assessment scoring tool for simulation-based OSCE assessment of resuscitation competence in postgraduate EM trainees.
authors.affiliation: From the Department of Emergency Medicine (A.K.H., J.D.D.), Department of Community Health and Epidemiology (W.P.), and Faculty of Education (D.A.K.), Queen's University, Kingston; Department of Emergency Medicine (L.L.), University of Ottawa, Ottawa, Ontario, Canada.
authors.forename: Andrew Koch
authors.initials: AK
authors.lastname: Hall
authors.forename: Jeffrey Damon
authors.initials: JD
authors.lastname: Dagnone
authors.forename: Lauren
authors.initials: L
authors.lastname: Lacroix
authors.forename: William
authors.initials: W
authors.lastname: Pickett
authors.forename: Don Albert
authors.initials: DA
authors.lastname: Klinger
mesh_terms: D002983:Clinical Competence; D004501:Education, Medical / Q000379:methods*; D004521:Educational Measurement / Q000379:methods* / Q000592:standards; D004635:Emergency Medicine / Q000193:education*; D006801:Humans; D007396:Internship and Residency; D015588:Observer Variation; D015203:Reproducibility of Results; D012151:Resuscitation / Q000193:education*; D000066908:Simulation Training / Q000379:methods* / Q000592:standards