title: Assessment of medical student clinical reasoning by "lay" vs physician raters: inter-rater reliability using a scoring guide in a multidisciplinary objective structured clinical examination.
abstract: BACKGROUND
  To determine whether a "lay" rater could assess clinical reasoning, interrater reliability was measured between physician and lay raters of patient notes written by medical students as part of an 8-station objective structured clinical examination.
  
  
  METHODS
  Seventy-five notes were rated on core elements of clinical reasoning by physician and lay raters independently, using a scoring guide developed by physician consensus. Twenty-five notes were rerated by a 2nd physician rater as an expert control. Kappa statistics and simple percentage agreement were calculated in 3 areas: evidence for and against each diagnosis and diagnostic workup.
  
  
  RESULTS
  Agreement between physician and lay raters for the top diagnosis was as follows: supporting evidence, 89% (κ = .72); evidence against, 89% (κ = .81); and diagnostic workup, 79% (κ = .58). Physician rater agreement was 83% (κ = .59), 92% (κ = .87), and 96% (κ = .87), respectively.
  
  
  CONCLUSIONS
  Using a comprehensive scoring guide, interrater reliability for physician and lay raters was comparable with reliability between 2 expert physician raters.
authors.affiliation: Department of Surgery, New York University School of Medicine, New York, NY, USA.
authors.forename: Alexandra J
authors.initials: AJ
authors.lastname: Berger
authors.forename: Colleen C
authors.initials: CC
authors.lastname: Gillespie
authors.forename: Linda R
authors.initials: LR
authors.lastname: Tewksbury
authors.forename: Ivey M
authors.initials: IM
authors.lastname: Overstreet
authors.forename: Ming C
authors.initials: MC
authors.lastname: Tsai
authors.forename: Adina L
authors.initials: AL
authors.lastname: Kalet
authors.forename: Jennifer B
authors.initials: JB
authors.lastname: Ogilvie
mesh_terms: D015746:Abdominal Pain / Q000175:diagnosis*; D002983:Clinical Competence; D003479:Curriculum; D004501:Education, Medical; D004521:Educational Measurement / Q000592:standards*; D005260:Female; D006801:Humans; D008297:Male; D015203:Reproducibility of Results; D013337:Students, Medical / Q000523:psychology*; D013647:Task Performance and Analysis; D013850:Thinking