title: Assessment in orthopedic training-an analysis of rating consistency by using an objective structured examination video.
abstract: OBJECTIVE
  This study describes a fast and efficient method that uses a prevalidated videotape of an objective structured clinical examination (OSCE) in a fracture scenario to evaluate raters and to measure the consistency of raters from different subspecialties and with varying levels of seniority.
  
  
  METHODS
  We videotaped clinical scenarios for the purpose of evaluating residents' communication and clinical assessment skills. All orthopedic staff used prevalidated checklists to assess residents' performance in the videotape at 3 different time points. Cronbach's α was calculated to evaluate the internal consistency of the OSCE checklist construct. Kendall's W and KR-20 were used to investigate rater agreement. Expert validity was calculated to compare OSCE experts with the present raters.
  
  
  RESULTS
  A high Cronbach's α for the 23-item scale regarding global assessment in all 3 tests confirmed construct validity. Kendall's W showed only moderate interrater reliability. KR-20 was 0.96 for the pretest, 0.968 for the posttest, and 0.892 for the long-term test, indicating high internal consistency. The p-value for expert validity was 0.626 (independent t-test, n.s.).
  
  
  CONCLUSIONS
  This efficient and fast video-based assessment of raters was reliable and yielded satisfactory rater consistency and some evidence for validity.
authors: Department of Orthopaedic Surgery, Chang Gung Memorial Hospital, Chang Gung University Collge of Medicine, Taiwan, Republlic of China. alvinchen@adm.cgmh.org.tw
authors: Alvin Chao-Yu
authors: AC
authors: Chen
authors: Mel S
authors: MS
authors: Lee
authors: Wen-Jer
authors: WJ
authors: Chen
authors: Shih-Tseng
authors: ST
authors: Lee
mesh_terms: D002983:Clinical Competence; D004521:Educational Measurement / Q000379:methods*; D006801:Humans; D009985:Orthopedics / Q000193:education*; D014743:Videotape Recording