title: Education research: Bias and poor interrater reliability in evaluating the neurology clinical skills examination.
abstract: OBJECTIVE
  The American Board of Psychiatry and Neurology (ABPN) has recently replaced the traditional, centralized oral examination with the locally administered Neurology Clinical Skills Examination (NEX). The ABPN postulated the experience with the NEX would be similar to the Mini-Clinical Evaluation Exercise, a reliable and valid assessment tool. The reliability and validity of the NEX has not been established.
  
  
  METHODS
  NEX encounters were videotaped at 4 neurology programs. Local faculty and ABPN examiners graded the encounters using 2 different evaluation forms: an ABPN form and one with a contracted rating scale. Some NEX encounters were purposely failed by residents. Cohen's kappa and intraclass correlation coefficients (ICC) were calculated for local vs ABPN examiners.
  
  
  RESULTS
  Ninety-eight videotaped NEX encounters of 32 residents were evaluated by 20 local faculty evaluators and 18 ABPN examiners. The interrater reliability for a determination of pass vs fail for each encounter was poor (kappa 0.32; 95% confidence interval [CI] = 0.11, 0.53). ICC between local faculty and ABPN examiners for each performance rating on the ABPN NEX form was poor to moderate (ICC range 0.14-0.44), and did not improve with the contracted rating form (ICC range 0.09-0.36). ABPN examiners were more likely than local examiners to fail residents.
  
  
  CONCLUSIONS
  There is poor interrater reliability between local faculty and American Board of Psychiatry and Neurology examiners. A bias was detected for favorable assessment locally, which is concerning for the validity of the examination. Further study is needed to assess whether training can improve interrater reliability and offset bias.
authors: Department of Neurology, Henry Ford Hospital, 2799 West Grand Blvd., Detroit, MI 48202, USA. lschuh@neuro.hfh.edu
authors: L A
authors: LA
authors: Schuh
authors: Z
authors: Z
authors: London
authors: R
authors: R
authors: Neel
authors: C
authors: C
authors: Brock
authors: B M
authors: BM
authors: Kissela
authors: L
authors: L
authors: Schultz
authors: D J
authors: DJ
authors: Gelb
mesh_terms: D015982:Bias; D002983:Clinical Competence / Q000592:standards*; D004521:Educational Measurement / Q000379:methods / Q000592:standards; D005069:Evaluation Studies as Topic; D005260:Female; D006801:Humans; D007396:Internship and Residency / Q000592:standards*; D008297:Male; D009462:Neurology / Q000193:education*; D011570:Psychiatry / Q000193:education; D015203:Reproducibility of Results; D014743:Videotape Recording