title: Reliability of a core competency checklist assessment in the emergency department: the Standardized Direct Observation Assessment Tool.
abstract: OBJECTIVE
  A Council of Emergency Medicine Residency Directors task force developed the Standardized Direct Observation Assessment Tool (SDOT), a 26-item checklist assessment tool to evaluate Accreditation Council for Graduate Medical Education resident core competencies by direct observation. Each of the checklist items is assigned to one or more of five core competencies. The objective of this study was to test the interrater measurement properties of the SDOT instrument.
  
  
  METHODS
  Two videos of simulated patient-resident-attending physician encounters were produced. Academic emergency medicine faculty members not involved in the development of the form viewed the two encounters and completed the SDOT for each. Faculty demographic data were collected. Data were collected from 82 faculty members at 16 emergency medicine residency programs. The checklist items were used to generate a composite score for each core competency of patient care, medical knowledge, interpersonal and communication skills, professionalism, and systems-based practice.
  
  
  RESULTS
  Univariate analysis demonstrated a high degree of agreement between evaluators in evaluating residents for both videos. Multivariate analysis found no differences in rating by faculty when examined by experience, academic title, site, or previous use of the SDOT.
  
  
  CONCLUSIONS
  Faculty from 16 emergency medicine residency programs had a high interrater agreement when using the SDOT to evaluate resident core competency performance. This study did not test the validity of the tool. This data analysis is mainly descriptive, and scripted video scenarios may not approximate direct observation in the emergency department.
authors.affiliation: Department of Emergency Medicine, Emory University, Atlanta, GA 30305, USA. pshayne@emory.edu
authors.forename: Philip
authors.initials: P
authors.lastname: Shayne
authors.forename: Fiona
authors.initials: F
authors.lastname: Gallahue
authors.forename: Stephan
authors.initials: S
authors.lastname: Rinnert
authors.forename: Craig L
authors.initials: CL
authors.lastname: Anderson
authors.forename: Gene
authors.initials: G
authors.lastname: Hern
authors.forename: Eric
authors.initials: E
authors.lastname: Katz
mesh_terms: D002983:Clinical Competence; D004521:Educational Measurement / Q000379:methods*; D004635:Emergency Medicine / Q000193:education*; D004636:Emergency Service, Hospital; D005180:Faculty, Medical; D006801:Humans; D007396:Internship and Residency / Q000379:methods*; D015999:Multivariate Analysis; D015588:Observer Variation; D011446:Prospective Studies; D015203:Reproducibility of Results; D013647:Task Performance and Analysis; D014481:United States