title: Scoring the computer-based case simulation component of USMLE Step 3: a comparison of preoperational and operational data.
abstract: OBJECTIVE
  Operational USMLE(TM) computer-based case simulation results were examined to determine the extent to which rater reliability and regression model performance met expectations based on preoperational data.
  
  
  METHODS
  Operational data resulted from Step 3 examinations given between 1999 and 2004. Plots were produced using reliability and multiple correlation coefficients.
  
  
  RESULTS
  Operational testing reliabilities increased over the four years but were lower than the preoperational reliability. Multiple correlation coefficient results are somewhat superior to the results reported during the preoperational period and suggest that the operational scoring algorithms have been relatively consistent.
  
  
  CONCLUSIONS
  Changes in the rater population, changes in the rating task, and enhancements to the training procedures are several factors that can explain the identified differences between preoperational and operational results. The present findings have important implications for test development and test validity.
authors.affiliation: National Board of Medical Examiners, Philadelphia, PA 19104, USA. mmargolis@nbme.org
authors.forename: Melissa J
authors.initials: MJ
authors.lastname: Margolis
authors.forename: Brian E
authors.initials: BE
authors.lastname: Clauser
authors.forename: Polina
authors.initials: P
authors.lastname: Harik
mesh_terms: D000465:Algorithms; D002983:Clinical Competence; D003198:Computer Simulation; D004501:Education, Medical; D004521:Educational Measurement / Q000379:methods* / Q000706:statistics & numerical data; D006801:Humans; D008007:Licensure, Medical; D015588:Observer Variation; D012044:Regression Analysis; D015203:Reproducibility of Results