title: Effects of a rater training on rating accuracy in a physical examination skills assessment.
abstract: BACKGROUND
  The accuracy and reproducibility of medical skills assessment is generally low. Rater training has little or no effect. Our knowledge in this field, however, relies on studies involving video ratings of overall clinical performances. We hypothesised that a rater training focussing on the frame of reference could improve accuracy in grading the curricular assessment of a highly standardised physical head-to-toe examination.
  
  
  METHODS
  Twenty-one raters assessed the performance of 242 third-year medical students. Eleven raters had been randomly assigned to undergo a brief frame-of-reference training a few days before the assessment. 218 encounters were successfully recorded on video and re-assessed independently by three additional observers. Accuracy was defined as the concordance between the raters' grade and the median of the observers' grade. After the assessment, both students and raters filled in a questionnaire about their views on the assessment.
  
  
  RESULTS
  Rater training did not have a measurable influence on accuracy. However, trained raters rated significantly more stringently than untrained raters, and their overall stringency was closer to the stringency of the observers. The questionnaire indicated a higher awareness of the halo effect in the trained raters group. Although the self-assessment of the students mirrored the assessment of the raters in both groups, the students assessed by trained raters felt more discontent with their grade.
  
  
  CONCLUSIONS
  While training had some marginal effects, it failed to have an impact on the individual accuracy. These results in real-life encounters are consistent with previous studies on rater training using video assessments of clinical performances. The high degree of standardisation in this study was not suitable to harmonize the trained raters' grading. The data support the notion that the process of appraising medical performance is highly individual. A frame-of-reference training as applied does not effectively adjust the physicians' judgement on medical students in real-live assessments.
authors.affiliation: Universitätsklinikum Schleswig-Holstein, Campus Lübeck, Medizinische Klinik I, Lübeck, Deutschland.
authors.forename: Gunther
authors.initials: G
authors.lastname: Weitz
authors.affiliation: Institut für Qualitätsentwicklung an Schulen Schleswig-Holstein, Kronshagen, Deutschland.
authors.forename: Christian
authors.initials: C
authors.lastname: Vinzentius
authors.affiliation: Universitätsklinikum Schleswig-Holstein, Campus Lübeck, Medizinische Klinik I, Lübeck, Deutschland.
authors.forename: Christoph
authors.initials: C
authors.lastname: Twesten
authors.affiliation: Universitätsklinikum Schleswig-Holstein, Campus Lübeck, Medizinische Klinik I, Lübeck, Deutschland.
authors.forename: Hendrik
authors.initials: H
authors.lastname: Lehnert
authors.affiliation: Universitätsklinikum Schlesweig-Holstein, Campus Kiel, Medizinische Klinik III, Kiel, Deutschland.
authors.forename: Hendrik
authors.initials: H
authors.lastname: Bonnemeier
authors.affiliation: Universität zu Lübeck, Institut für Medizinische Biometrie und Statistik, Lübeck, Deutschland.
authors.forename: Inke R
authors.initials: IR
authors.lastname: König
mesh_terms: D002983:Clinical Competence / Q000592:standards*; D003479:Curriculum; D004501:Education, Medical; D004521:Educational Measurement / Q000592:standards; D005180:Faculty, Medical; D005858:Germany; D006801:Humans; D007318:Inservice Training; D015588:Observer Variation; D010808:Physical Examination / Q000592:standards*; D012647:Self-Assessment; D013337:Students, Medical / Q000523:psychology; D000070260:Teacher Training; D014741:Video Recording
keywords: physical examination skills; randomised controlled trial; rater training; rating accuracy; skills assessment