title: The influence of first impressions on subsequent ratings within an OSCE station.
abstract: Competency-based assessment is placing increasing emphasis on the direct observation of learners. For this process to produce valid results, it is important that raters provide quality judgments that are accurate. Unfortunately, the quality of these judgments is variable and the roles of factors that influence the accuracy of those judgments are not clearly understood. One such factor is first impressions: that is, judgments about people we do not know, made quickly and based on very little information. This study explores the influence of first impressions in an OSCE. Specifically, the purpose is to begin to examine the accuracy of a first impression and its influence on subsequent ratings. We created six videotapes of history-taking performance. Each video was scripted from a real performance by six examinee residents within a single OSCE station. Each performance was re-enacted with six different actors playing the role of the examinees and one actor playing the role of the patient and videotaped. A total of 23 raters (i.e., physician examiners) reviewed each video and were asked to make a global judgment of the examinee's clinical abilities after 60 s (First Impression GR) by providing a rating on a six-point global rating scale and then to rate their confidence in the accuracy of that judgment by providing a rating on a five-point rating scale (Confidence GR). After making these ratings, raters then watched the remainder of the examinee's performance and made another global rating of performance (Final GR) before moving on to the next video. First impression ratings of ability varied across examinees and were moderately correlated to expert ratings (r = .59, 95% CI [-.13, .90]). There were significant differences in mean ratings for three examinees. Correlations ranged from .05 to .56 but were only significant for three examinees. Rater confidence in their first impression was not related to the likelihood of a rater changing their rating between the first impression and a subsequent rating. The findings suggest that first impressions could play a role in explaining variability in judgments, but their importance was determined by the videotaped performance of the examinees. More work is needed to clarify conditions that support or discourage the use of first impressions.
authors.affiliation: Department of Innovation in Medical Education, Faculty of Medicine, University of Ottawa, RGN 2206, 451 Smyth Road, Ottawa, ON, K1H 8M5, Canada. twood@uottawa.ca.
authors.forename: Timothy J
authors.initials: TJ
authors.lastname: Wood
authors.affiliation: Department of Medicine, University of Ottawa, 501 Smyth Road, Ottawa, ON, K1H 8L6, Canada.
authors.forename: James
authors.initials: J
authors.lastname: Chan
authors.affiliation: Department of Medicine, University of Ottawa, 501 Smyth Road, Ottawa, ON, K1H 8L6, Canada.
authors.forename: Susan
authors.initials: S
authors.lastname: Humphrey-Murto
authors.affiliation: Department of Medicine, University of Ottawa, 501 Smyth Road, Ottawa, ON, K1H 8L6, Canada.
authors.forename: Debra
authors.initials: D
authors.lastname: Pugh
authors.affiliation: Department of Medicine, University of Ottawa, 501 Smyth Road, Ottawa, ON, K1H 8L6, Canada.
authors.forename: Claire
authors.initials: C
authors.lastname: Touchie
mesh_terms: D002983:Clinical Competence / Q000592:standards; D004501:Education, Medical / Q000379:methods* / Q000592:standards; D004521:Educational Measurement / Q000379:methods* / Q000592:standards*; D005180:Faculty, Medical / Q000523:psychology* / Q000592:standards; D006801:Humans; D008487:Medical History Taking / Q000592:standards; D015588:Observer Variation; D015203:Reproducibility of Results; D014743:Videotape Recording
keywords: First impression; OSCE; Rater cognition; Rater-based assessment