title: Assessing the impact of modifications to the documentation component's scoring rubric and rater training on USMLE integrated clinical encounter scores.
abstract: BACKGROUND
  Documentation is a subcomponent of the Step 2 Clinical Skills Examination Integrated Clinical Encounter (ICE) component wherein licensed physicians rate examinees on their ability to communicate the findings of the patient encounter, diagnostic impression, and initial patient work-up. The main purpose of this research was to examine the impact of modifications to the scoring rubric and rater training protocol on the psychometric characteristics of the documentation scores.
  
  
  METHODS
  Following the modifications, the variance structure of the ICE components was modeled using multivariate generalizability theory.
  
  
  RESULTS
  The results confirmed the expectations that true score variance for the documentation subcomponent would increase after adopting a modified training protocol and increased rubric specificity.
  
  
  CONCLUSIONS
  In general, results support the commonsense assumption that providing raters with detailed rubrics and comprehensive training will indeed improve measurement outcomes. Although the steps taken here were in the right direction, there remains room for improvement. Efforts are currently under way to further improve both the scoring rubrics and rater training.
authors.affiliation: National Board of Medical Examiners, 3750 Market Street, Philadelphia, PA, 19104, USA. sbaldwin@nbme.org
authors.forename: Su G
authors.initials: SG
authors.lastname: Baldwin
authors.forename: Polina
authors.initials: P
authors.lastname: Harik
authors.forename: Lisa A
authors.initials: LA
authors.lastname: Keller
authors.forename: Brian E
authors.initials: BE
authors.lastname: Clauser
authors.forename: Peter
authors.initials: P
authors.lastname: Baldwin
authors.forename: Thomas A
authors.initials: TA
authors.lastname: Rebbecchi
mesh_terms: D002983:Clinical Competence; D004521:Educational Measurement / Q000379:methods* / Q000592:standards*; D008007:Licensure, Medical; D014481:United States