title: Does what we write matter? Determining the features of high- and low-quality summative written comments of students on the internal medicine clerkship using pile-sort and consensus analysis: a mixed-methods study.
abstract: BACKGROUND
  Written comments by medical student supervisors provide written foundation for grade narratives and deans' letters and play an important role in student's professional development. Written comments are widely used but little has been published about the quality of written comments. We hypothesized that medical students share an understanding of qualities inherent to a high-quality and a low-quality narrative comment and we aimed to determine the features that define high- and low-quality comments.
  
  
  METHODS
  Using the well-established anthropological pile-sort method, medical students sorted written comments into 'helpful' and 'unhelpful' piles, then were interviewed to determine how they evaluated comments. We used multidimensional scaling and cluster analysis to analyze data, revealing how written comments were sorted across student participants. We calculated the degree of shared knowledge to determine the level of internal validity in the data. We transcribed and coded data elicited during the structured interview to contextualize the student's answers. Length of comment was compared using one-way analysis of variance; valence and frequency comments were thought of as helpful were analyzed by chi-square.
  
  
  RESULTS
  Analysis of written comments revealed four distinct clusters. Cluster A comments reinforced good behaviors or gave constructive criticism for how changes could be made. Cluster B comments exhorted students to continue non-specific behaviors already exhibited. Cluster C comments used grading rubric terms without giving student-specific examples. Cluster D comments used sentence fragments lacking verbs and punctuation. Student data exhibited a strong fit to the consensus model, demonstrating that medical students share a robust model of attributes of helpful and unhelpful comments. There was no correlation between valence of comment and perceived helpfulness.
  
  
  CONCLUSIONS
  Students find comments demonstrating knowledge of the student and providing specific examples of appropriate behavior to be reinforced or inappropriate behavior to be eliminated helpful, and comments that are non-actionable and non-specific to be least helpful. Our research and analysis allow us to make recommendations helpful for faculty development around written feedback.
authors.affiliation: School of Social Work, The University of Texas, Austin, TX, USA.
authors.forename: Lauren
authors.initials: L
authors.lastname: Gulbas
authors.affiliation: Geisel School of Medicine at Dartmouth, Hanover, NH, USA.
authors.forename: William
authors.initials: W
authors.lastname: Guerin
authors.affiliation: Geisel School of Medicine at Dartmouth, Hanover, NH, USA. Hilary.F.Ryder@hitchcock.org.|Department of Medicine, Dartmouth-Hitchcock Medical Center, One Medical Center Drive, Lebanon, NH, 03784, USA. Hilary.F.Ryder@hitchcock.org.
authors.forename: Hilary F
authors.initials: HF
authors.lastname: Ryder
mesh_terms: D000328:Adult; D002982:Clinical Clerkship; D016000:Cluster Analysis; D032921:Consensus; D005260:Female; D000068156:Formative Feedback; D006379:Helping Behavior; D006801:Humans; D007388:Internal Medicine / Q000193:education*; D008297:Male; D014956:Writing; D055815:Young Adult
keywords: Clinical medical education; Cultural consensus; Medical education; Written comments