title: Validity and interrater reliability of a regional mock oral board examination.
abstract: BACKGROUND
  There have been decreasing pass rates recently on the American Board of Surgery Certifying Examination (ABSCE). General surgery residents from the University of Pittsburgh Medical Center, the West Penn Allegheny Health System, the Conemaugh Health System, and Mercy Hospital participate in a mock oral board examination, which is similar to the ABSCE. The aims of the study are to compare examinee performance on the mock oral boards with the ABSCE and to evaluate the interrater reliability of examiner pairs.
  
  
  METHODS
  In this retrospective study from 2003 to 2010, outcomes on the mock oral boards and the first attempt of the ABSCE for chief residents were compared for the 4 regional residency programs. Interrater reliability for examiner pairs was evaluated with agreement and kappa statistics. Nonparametric statistics were performed, with α = 0.05.
  
  
  RESULTS
  A total of 32 of 38 (84.2%) chief residents passed the mock oral boards. The median score for each of the 3 rooms was 6 (clear pass). A total of 37 of 38 (97.4%) residents passed the ABSCE. The sensitivity of the mock oral boards was 83.8%, with a positive predictive value of 96.9%, and an accuracy of 81.6%. A total of 25 of 47 (53.2%) examiner pairs were from the same residency institution, whereas 22 of 47 (46.8%) were from different institutions. The median agreement was 100% (interquartile range (IQR) [100% - 100%]). The median kappa statistic was 1.00 (IQR [0.38-1.00]). The Mann-Whitney U tests showed no difference in agreement or kappa for examiner pairs from the same or from different institutions (p> 0.05).
  
  
  CONCLUSIONS
  The mock oral boards have substantial sensitivity and positive predictive value in relation to the ABSCE. There are also very high levels of interrater agreement and interrater reliability. This regional mock oral board examination is valuable for ABSCE preparation.
authors.affiliation: Department of Surgery, University of Pittsburgh School of Medicine, University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania 15213, USA. falconej@upmc.edu
authors.forename: John L
authors.initials: JL
authors.lastname: Falcone
authors.forename: Daniel J
authors.initials: DJ
authors.lastname: Gagne
authors.forename: Kenneth K W
authors.initials: KK
authors.lastname: Lee
authors.forename: Giselle G
authors.initials: GG
authors.lastname: Hamad
mesh_terms: D002568:Certification / Q000592:standards*; D004521:Educational Measurement / Q000592:standards*; D013502:General Surgery / Q000193:education*; D006801:Humans; D007396:Internship and Residency; D011237:Predictive Value of Tests; D015203:Reproducibility of Results; D012189:Retrospective Studies; D013044:Specialty Boards / Q000592:standards; D014481:United States