title: Considerations in the use of reflective writing for student assessment: issues of reliability and validity.
abstract: BACKGROUND
  Reflective writing is a popular tool to support the growth of reflective capacity in undergraduate medical learners. Its popularity stems from research suggesting that reflective capacity may lead to improvements in skills such as empathy, communication, collaboration and professionalism. This has led to assumptions that reflective writing can also serve as a tool for student assessment. However, evidence to support the reliability and validity of reflective writing as a meaningful assessment strategy is lacking.
  
  
  METHODS
  Using a published instrument for measuring 'reflective capacity' (the Reflection Evaluation for Learners' Enhanced Competencies Tool [REFLECT]), four trained raters independently scored four samples of writing from each of 107 undergraduate medical students to determine the reliability of reflective writing scores. REFLECT scores were then correlated with scores on a Year 4 objective structured clinical examination (OSCE) and Year 2 multiple-choice question (MCQ) examinations to examine, respectively, convergent and divergent validity.
  
  
  RESULTS
  Across four writing samples, four-rater Cronbach's α-values ranged from 0.72 to 0.82, demonstrating reasonable inter-rater reliability with four raters using the REFLECT rubric. However, inter-sample reliability was fairly low (four-sample Cronbach's α = 0.54, single-sample intraclass correlation coefficient: 0.23), which suggests that performance on one reflective writing sample was not strongly indicative of performance on the next. Approximately 14 writing samples are required to achieve reasonable inter-sample reliability. The study found weak, non-significant correlations between reflective writing scores and both OSCE global scores (r = 0.13) and MCQ examination scores (r = 0.10), demonstrating a lack of relationship between reflective writing and these measures of performance.
  
  
  CONCLUSIONS
  Our findings suggest that to draw meaningful conclusions about reflective capacity as a stable construct in individuals requires 14 writing samples per student, each assessed by four or five raters. This calls into question the feasibility and utility of using reflective writing rigorously as an assessment tool in undergraduate medical education.
authors: Department of Communication Studies, Mount Saint Vincent University, Halifax, Nova Scotia, Canada.
authors: Tracy
authors: T
authors: Moniz
authors: Department of Obstetrics and Gynaecology, Western University, London, Ontario, Canada.
authors: Shannon
authors: S
authors: Arntfield
authors: Department of Health and Rehabilitation Sciences, Western University, London, Ontario, Canada.
authors: Kristina
authors: K
authors: Miller
authors: Centre for Education Research and Innovation, Western University, London, Ontario, Canada.
authors: Lorelei
authors: L
authors: Lingard
authors: Centre for Education Research and Innovation, Western University, London, Ontario, Canada.
authors: Chris
authors: C
authors: Watling
authors: Centre for Health Education Scholarship, University of British Columbia, Vancouver, British Columbia, Canada.
authors: Glenn
authors: G
authors: Regehr
mesh_terms: D002170:Canada; D003142:Communication; D004504:Education, Medical, Undergraduate; D004521:Educational Measurement / Q000379:methods*; D004645:Empathy; D006801:Humans; D015203:Reproducibility of Results; D013337:Students, Medical / Q000523:psychology; D014956:Writing