title: The generalizability of documentation scores from the USMLE Step 2 Clinical Skills examination.
abstract: BACKGROUND
  This research examined various sources of measurement error in the documentation score component of the United States Medical Licensing Examination (USMLE) Step 2 Clinical Skills examination.
  
  
  METHODS
  A generalizability theory framework was employed to examine the documentation ratings for 847 examinees who completed the USMLE Step 2 Clinical Skills examination during an eight-day period in 2006. Each patient note was scored by two different raters allowing for a persons-crossed-with-raters-nested-in-cases design.
  
  
  RESULTS
  The results suggest that inconsistent performance on the part of raters makes a substantially greater contribution to measurement error than case specificity. Double scoring the notes significantly increases precision.
  
  
  CONCLUSIONS
  The results provide guidance for improving operational scoring of the patient notes. Double scoring of the notes may produce an increase in the precision of measurement equivalent to that achieved by lengthening the test by more than 50%. The study also cautions researchers that when examining sources of measurement error, inappropriate data-collection designs may result in inaccurate inferences.
authors.affiliation: National Board of Medical Examiners, 3750 Market Street, Philadelphia, PA 19104, USA. bclauser@nbme.org
authors.forename: Brian E
authors.initials: BE
authors.lastname: Clauser
authors.forename: Polina
authors.initials: P
authors.lastname: Harik
authors.forename: Melissa J
authors.initials: MJ
authors.lastname: Margolis
authors.forename: Janet
authors.initials: J
authors.lastname: Mee
authors.forename: Kimberly
authors.initials: K
authors.lastname: Swygert
authors.forename: Thomas
authors.initials: T
authors.lastname: Rebbecchi
mesh_terms: D002983:Clinical Competence; D015331:Cohort Studies; D003142:Communication; D005793:Generalization, Psychological; D006801:Humans; D008007:Licensure, Medical; D015588:Observer Variation; D016544:Patient Simulation; D010808:Physical Examination; D010817:Physician-Patient Relations; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D014481:United States