title: Evaluating the reliability of surgical assessment methods in an orthopedic residency program.
abstract: BACKGROUND
  Orthopedic surgical education in Canada has seen major change in the last 15 years. Work hour restrictions and external influence have led to new approaches for surgical training. With a change toward competency-based educational models under the CanMEDS headings there is a need to ensure the validity of modern assessment methods. Our objective was to evaluate the reliability of a currently used surgical skill assessment tool within an orthopedic residency program, as measured by the Surgical Encounters Form.
  
  
  METHODS
  A surgical assessment tool has previously been created at our institution that comprises 15 items spanning 4 of the CanMEDS competencies. Results were blinded to the primary investigator and coded by a third party. The assessments were collected, and we measured percent agreement using Cronbach's α and Fleiss κ.
  
  
  RESULTS
  Over a 5-month period 11 staff members assessed 10 residents. Eighty-eight assessments were completed in total. Weighted percent agreement was 90.9%. Cronbach's α averaged 0.865 for the medical expert role, 0.920 for technical skills, 0.934 for the communicator role, 1.00 for the collaborator role and 1.00 for the health advocate role. The mean Fleiss κ score was 0.147 (95% confidence interval -0.071 to 0.364), demonstrating low interrater reliability.
  
  
  CONCLUSIONS
  Despite the development of a validated assessment tool to evaluate surgical skills acquisition, interrater reliability results suggest low levels of agreement among assessors.
authors.affiliation: From the Department of Surgery (Smith), the Faculty of Medicine, Health Sciences Centre, (Harnett) and the Department of Orthopaedic Traumatology (Furey), Memorial University of Newfoundland, St. John's, NL.
authors.forename: Nicholas
authors.initials: N
authors.lastname: Smith
authors.affiliation: From the Department of Surgery (Smith), the Faculty of Medicine, Health Sciences Centre, (Harnett) and the Department of Orthopaedic Traumatology (Furey), Memorial University of Newfoundland, St. John's, NL.
authors.forename: John
authors.initials: J
authors.lastname: Harnett
authors.affiliation: From the Department of Surgery (Smith), the Faculty of Medicine, Health Sciences Centre, (Harnett) and the Department of Orthopaedic Traumatology (Furey), Memorial University of Newfoundland, St. John's, NL.
authors.forename: Andrew
authors.initials: A
authors.lastname: Furey
mesh_terms: D002170:Canada; D002983:Clinical Competence / Q000592:standards*; D003162:Competency-Based Education / Q000592:standards*; D004521:Educational Measurement / Q000379:methods / Q000592:standards*; D006801:Humans; D007396:Internship and Residency / Q000592:standards*; D009985:Orthopedics / Q000193:education*; D015203:Reproducibility of Results; D066231:Surgeons / Q000193:education / Q000592:standards*