title: Validity evidence for a patient note scoring rubric based on the new patient note format of the United States Medical Licensing Examination.
abstract: OBJECTIVE
  This study examines validity evidence for the Patient Note Scoring Rubric, which was developed for a local graduation competency exam (GCE) to assess patient notes written in the new United States Medical Licensing Examination (USMLE) Step 2 Clinical Skills format. The rubric was designed to measure three dimensions: Documentation, justified differential diagnosis (DDX), and Workup.
  
  
  METHODS
  Analyses used GCE data from 170 fourth-year medical students who completed five standardized patient (SP) cases in May 2012. Five physician raters each scored all responses for one case. Internal structure was examined using correlations between dimensions and between cases; a generalizability study was also conducted. Relationship to other variables was examined by correlating patient note scores with SP encounter scores. Consequence was assessed by comparing pass-fail rates between the rubric and the previous global rating. Response process was examined using rater feedback.
  
  
  RESULTS
  Correlations between scores from different dimensions ranged between 0.33 and 0.44. Reliability of scores based on the phi coefficient was 0.43; 15 cases were required to reach a phi coefficient of 0.70. Evidence of case specificity was found. Documentation scores were moderately correlated with SP scores for data gathering (r = 0.47, P < .001). There was no meaningful change in pass-fail rates. Raters' feedback indicated that they required more training for scoring the DDX and Workup dimensions.
  
  
  CONCLUSIONS
  There is initial validity evidence for use of this rubric to score local clinical exams that are based on the new USMLE patient note format.
authors: Dr. Park is assistant professor, Department of Medical Education, University of Illinois at Chicago College of Medicine, Chicago, Illinois. Dr. Lineberry is assistant professor, Department of Medical Education, University of Illinois at Chicago College of Medicine, Chicago, Illinois. Dr. Hyderi is associate dean for curriculum and associate professor, Department of Family Medicine, University of Illinois at Chicago College of Medicine, Chicago, Illinois. Dr. Bordage is professor, Department of Medical Education, University of Illinois at Chicago College of Medicine, Chicago, Illinois. Dr. Riddle is director of faculty development and research assistant professor, Department of Medical Education, University of Illinois at Chicago College of Medicine, Chicago, Illinois. Dr. Yudkowsky is director, Graham Clinical Performance Center, and associate professor, Department of Medical Education, University of Illinois at Chicago College of Medicine, Chicago, Illinois.
authors: Yoon Soo
authors: YS
authors: Park
authors: Matthew
authors: M
authors: Lineberry
authors: Abbas
authors: A
authors: Hyderi
authors: Georges
authors: G
authors: Bordage
authors: Janet
authors: J
authors: Riddle
authors: Rachel
authors: R
authors: Yudkowsky
mesh_terms: D002983:Clinical Competence / Q000592:standards*; D003937:Diagnosis, Differential; D004282:Documentation; D004504:Education, Medical, Undergraduate / Q000592:standards*; D004521:Educational Measurement / Q000379:methods*; D005246:Feedback; D006801:Humans; D008007:Licensure, Medical; D015203:Reproducibility of Results; D012577:Schools, Medical; D014481:United States