title: Education Research: unsatisfactory NEX rating correlations: searching for the reasons.
abstract: OBJECTIVE
  To determine whether the previously demonstrated poor correlation between local faculty and external American Board of Psychiatry and Neurology (ABPN) examiners evaluating the Neurology Evaluation Exercise (NEX) is attributable to a difference between raters who know the residents and raters who do not, a difference between raters with ABPN experience and raters without it, or some other factor.
  
  
  METHODS
  Deidentified NEX encounters were videotaped at 2 neurology residency programs. Each video was graded by 1 local faculty examiner, 1 external faculty examiner with ABPN experience, and 1 external faculty examiner without ABPN experience, using the ABPN-sanctioned form. Acceptable/unacceptable rates were compared using Cohen κ statistic.
  
  
  RESULTS
  Fifty-eight videotaped NEX encounters involving 20 residents were evaluated by 12 local faculty examiners, 13 ABPN examiners, and 10 external non-ABPN examiners. The level of agreement between groups failed to meet our target κ of 0.7 (ABPN vs non-ABPN external examiners: κ = 0.47 [95% confidence interval 0.21-0.73]; local vs non-ABPN external examiners: κ = 0.37 [95% confidence interval 0.08-0.66]; local vs ABPN external examiners: κ = 0.40 [95% confidence interval 0.14-0.67]). Local, non-ABPN, and ABPN examiners assigned a failing grade to 13 (22%), 11 (19%), and 16 (28%) of the NEX encounters, respectively.
  
  
  CONCLUSIONS
  The disappointing correlation between local examiners, non-ABPN external examiners, and ABPN external examiners is not solely attributable to bias toward familiar residents. Inadequate training in NEX administration and scoring could be a factor. It is also possible that the NEX is not a valid tool. Further study is necessary.
authors.affiliation: Department of Neurology, University of Michigan, Ann Arbor, MI, USA. zlondon@med.umich.edu
authors.forename: Zachary
authors.initials: Z
authors.lastname: London
authors.forename: Lori
authors.initials: L
authors.lastname: Schuh
authors.forename: Douglas J
authors.initials: DJ
authors.lastname: Gelb
authors.forename: Lonni
authors.initials: L
authors.lastname: Schultz
mesh_terms: D000704:Analysis of Variance; D004501:Education, Medical; D004521:Educational Measurement / Q000379:methods*; D006801:Humans; D009462:Neurology / Q000193:education* / Q000592:standards; D010820:Physicians; D011570:Psychiatry / Q000193:education*; D014743:Videotape Recording