title: Effects of item and rater characteristics on checklist recording: what should we look for?
abstract: OBJECTIVE
  Examinations based on using standardised patients (SPs) commonly use checklist recordings to evaluate students' clinical performance. This paper examines whether and to what extent item and rater characteristics affect the reliability of history checklist recording in an SP-based assessment.
  
  
  METHODS
  Checklist items were reviewed for the presence or absence of 5 item characteristics and a 2-point versus 3-point scoring scale. Agreement between checklist recordings obtained from SPs and clinician-examiners (CEs) were compared by item characteristics, scoring scale and CEs' level of involvement in the assessment.
  
  
  RESULTS
  Based on 3179 pairs of recordings, the overall percentage of agreement between SPs and CEs was 83% (kappa = 0.64). Agreement was significantly higher for items scored on a 2-point than on a 3-point scale, and when the CE was also the author and the trainer of the station. After controlling for other factors, item characteristics were only marginally associated with level of interrater agreement.
  
  
  CONCLUSIONS
  This study suggests that attention should be paid to specific aspects of checklist development and checklist recording training when an SP or CE is used as recorder.
authors.affiliation: Unit of Development and Research in Medical Education, Faculty of Medicine, University of Geneva, Geneva, Switzerland. philippe.huber@medecine.unige.ch
authors.forename: Philippe
authors.initials: P
authors.lastname: Huber
authors.forename: Anne
authors.initials: A
authors.lastname: Baroffio
authors.forename: Eric
authors.initials: E
authors.lastname: Chamot
authors.forename: François
authors.initials: F
authors.lastname: Herrmann
authors.forename: Mathieu R
authors.initials: MR
authors.lastname: Nendaz
authors.forename: Nu V
authors.initials: NV
authors.lastname: Vu
mesh_terms: D002983:Clinical Competence / Q000592:standards*; D004504:Education, Medical, Undergraduate; D004521:Educational Measurement / Q000379:methods*; D006801:Humans; D008487:Medical History Taking / Q000592:standards; D010808:Physical Examination / Q000592:standards; D013337:Students, Medical; D013663:Teaching / Q000379:methods