title: Relatively speaking: contrast effects influence assessors' scores and narrative feedback.
abstract: BACKGROUND
  In prior research, the scores assessors assign can be biased away from the standard of preceding performances (i.e. 'contrast effects' occur).
  
  
  OBJECTIVE
  This study examines the mechanism and robustness of these findings to advance understanding of assessor cognition. We test the influence of the immediately preceding performance relative to that of a series of prior performances. Further, we examine whether assessors' narrative comments are similarly influenced by contrast effects.
  
  
  METHODS
  Clinicians (n = 61) were randomised to three groups in a blinded, Internet-based experiment. Participants viewed identical videos of good, borderline and poor performances by first-year doctors in varied orders. They provided scores and written feedback after each video. Narrative comments were blindly content-analysed to generate measures of valence and content. Variability of narrative comments and scores was compared between groups.
  
  
  RESULTS
  Comparisons indicated contrast effects after a single performance. When a good performance was preceded by a poor performance, ratings were higher (mean 5.01, 95% confidence interval [CI] 4.79-5.24) than when observation of the good performance was unbiased (mean 4.36, 95% CI 4.14-4.60; p < 0.05, d = 1.3). Similarly, borderline performance was rated lower when preceded by good performance (mean 2.96, 95% CI 2.56-3.37) than when viewed without preceding bias (mean 3.55, 95% CI 3.17-3.92; p < 0.05, d = 0.7). The series of ratings participants assigned suggested that the magnitude of contrast effects is determined by an averaging of recent experiences. The valence (but not content) of narrative comments showed contrast effects similar to those found in numerical scores.
  
  
  CONCLUSIONS
  These findings are consistent with research from behavioural economics and psychology that suggests judgement tends to be relative in nature. Observing that the valence of narrative comments is similarly influenced suggests these effects represent more than difficulty in translating impressions into a number. The extent to which such factors impact upon assessment in practice remains to be determined as the influence is likely to depend on context.
authors: Centre for Respiratory Medicine and Allergy, Institute of Inflammation and Repair, University of Manchester, Manchester, UK.
authors: Peter
authors: P
authors: Yeates
authors: Royal Bolton Hospital, Bolton NHS Foundation Trust, Bolton, Lancashire, UK.
authors: Jenna
authors: J
authors: Cardell
authors: Health Education North West, Health Education England, Manchester, UK.
authors: Gerard
authors: G
authors: Byrne
authors: Centre for Health Education Scholarship, Division of Medicine, University of British Columbia, Vancouver, BC, Canada.
authors: Kevin W
authors: KW
authors: Eva
mesh_terms: D002983:Clinical Competence / Q000592:standards; D004521:Educational Measurement / Q000379:methods / Q000592:standards*; D005246:Feedback; D006801:Humans; D007600:Judgment; D033262:Narration; D015588:Observer Variation; D010820:Physicians / Q000523:psychology; D015203:Reproducibility of Results