title: The effect of standardization on the reliability of the Philippine Board of Surgery oral examinations.
abstract: BACKGROUND
  Cognizant of potential problems with validity and reliability, the Philippine Board of Surgery (PBS) undertook standardization of its oral examination procedures. This retrospective analytic study was conducted to determine and compare the reliability of the Philippine Board of Surgery oral examinations before and after standardization.
  
  
  METHODS
  The records of oral examinations from June 29, 2003 to March 28, 2010 were reviewed and measures of reliability (intraclass correlation coefficient and kappa agreement) were computed and compared between the time periods before and after standardization and among different case content modules administered. The proportion of passers between time periods and different content areas were also compared.
  
  
  RESULTS
  All measures of interrater reliability (intraclass correlation coefficient and kappa agreement) increased significantly overall and in all content areas and regardless of examinee outcome. There was also a trend to increased proportion of passers overall and significant improvement in passing rate in 4 out of 6 content areas.
  
  
  CONCLUSIONS
  This study validates the role of standardization of the content and scoring in improving the reliability of the oral examination, which is crucial for high stakes certifying examinations.
authors.affiliation: Department of Surgery, College of Medicine-Philippine General Hospital, University of the Philippines, Manila, Philippines. armandcrs@yahoo.com
authors.forename: Armando C
authors.initials: AC
authors.lastname: Crisostomo
mesh_terms: D000068:Accreditation / Q000379:methods / Q000592:standards*; D016001:Confidence Intervals; D016208:Databases, Factual; D004521:Educational Measurement / Q000379:methods*; D005260:Female; D013502:General Surgery / Q000193:education*; D006801:Humans; D008297:Male; D015588:Observer Variation; D010679:Philippines; D015203:Reproducibility of Results; D012189:Retrospective Studies; D013044:Specialty Boards / Q000592:standards*