title: Collecting validity evidence for an assessment of professionalism: findings from think-aloud interviews.
abstract: BACKGROUND
  This study investigated whether participants' subjective reports of how they assigned ratings on a multisource feedback instrument provide evidence to support interpreting the resulting scores as objective, accurate measures of professional behavior.
  
  
  METHODS
  Twenty-six participants completed think-aloud interviews while rating students, residents, or faculty members they had worked with previously. The items rated included 15 behavioral items and one global item.
  
  
  RESULTS
  Participants referred to generalized behaviors and global impressions six times as often as specific behaviors, rated observees in the absence of information necessary to do so, relied on indirect evidence about performance, and varied in how they interpreted items.
  
  
  CONCLUSIONS
  Behavioral change becomes difficult to address if it is unclear what behaviors raters considered when providing feedback. These findings highlight the importance of explicitly stating and empirically investigating the assumptions that underlie the use of an observational assessment tool.
authors.affiliation: Kathleen.mazor@umassmed.edu
authors.forename: Kathleen M
authors.initials: KM
authors.lastname: Mazor
authors.forename: Colleen
authors.initials: C
authors.lastname: Canavan
authors.forename: Margaret
authors.initials: M
authors.lastname: Farrell
authors.forename: Melissa J
authors.initials: MJ
authors.lastname: Margolis
authors.forename: Brian E
authors.initials: BE
authors.lastname: Clauser
mesh_terms: D030141:Feedback, Psychological; D006801:Humans; D007396:Internship and Residency; D007407:Interviews as Topic; D007721:Knowledge of Results, Psychological; D015588:Observer Variation; D010372:Pediatrics / Q000193:education*; D011361:Professional Competence; D036301:Qualitative Research; D015203:Reproducibility of Results; D012919:Social Behavior