title: Examining rater and occasion influences in observational assessments obtained from within the clinical environment.
abstract: BACKGROUND
  When ratings of student performance within the clerkship consist of a variable number of ratings per clinical teacher (rater), an important measurement question arises regarding how to combine such ratings to accurately summarize performance. As previous G studies have not estimated the independent influence of occasion and rater facets in observational ratings within the clinic, this study was designed to provide estimates of these two sources of error.
  
  
  METHODS
  During 2 years of an emergency medicine clerkship at a large midwestern university, 592 students were evaluated an average of 15.9 times. Ratings were performed at the end of clinical shifts, and students often received multiple ratings from the same rater. A completely nested G study model (occasion: rater: person) was used to analyze sampled rating data.
  
  
  RESULTS
  The variance component (VC) related to occasion was small relative to the VC associated with rater. The D study clearly demonstrates that having a preceptor rate a student on multiple occasions does not substantially enhance the reliability of a clerkship performance summary score.
  
  
  CONCLUSIONS
  Although further research is needed, it is clear that case-specific factors do not explain the low correlation between ratings and that having one or two raters repeatedly rate a student on different occasions/cases is unlikely to yield a reliable mean score. This research suggests that it may be more efficient to have a preceptor rate a student just once. However, when multiple ratings from a single preceptor are available for a student, it is recommended that a mean of the preceptor's ratings be used to calculate the student's overall mean performance score.
authors.affiliation: Department of Family Medicine, University of Iowa College of Medicine, Iowa City, IA, USA.|Office of Consultation and Research in Medical Education, University of Iowa College of Medicine, Iowa City, IA, USA; clarence-kreiter@uiowa.edu.
authors.forename: Clarence D
authors.initials: CD
authors.lastname: Kreiter
authors.affiliation: Department of Surgery, Indiana University School of Medicine, Indianapolis, IN, USA.
authors.forename: Adam B
authors.initials: AB
authors.lastname: Wilson
authors.affiliation: Office of Undergraduate Medical Education, Indiana University School of Medicine, Indianapolis, IN, USA.
authors.forename: Aloysius J
authors.initials: AJ
authors.lastname: Humbert
authors.affiliation: Office for Mentoring and Student Development, Medical Student Affairs, Indiana University School of Medicine, Indianapolis, IN, USA.
authors.forename: Patricia A
authors.initials: PA
authors.lastname: Wade
mesh_terms: D002982:Clinical Clerkship / Q000592:standards*; D002983:Clinical Competence; D004521:Educational Measurement / Q000379:methods* / Q000592:standards*; D004635:Emergency Medicine / Q000193:education; D006801:Humans; D015588:Observer Variation; D015203:Reproducibility of Results
keywords: clinical ratings; clinical skills; generalizability theory