title: Reliability of consultation skills assessments using standardised versus real patients.
abstract: OBJECTIVE
  Training in and assessment of consultation skills are high on the agenda of vocational training institutes for postgraduate training. There is a need to establish valid and reliable instruments to assess consultation skills in authentic settings. We investigated the number of assessors and observations needed to achieve reliable assessments of the consultation skills of general practice trainees (GPTs) using a communication instrument (MAAS-Global) and either standardised patient (SP) encounters or videotaped real patient (RP) encounters.
  
  
  METHODS
  Eight teachers at the Vrije Universiteit (VU) University Medical Centre in Amsterdam attended a training course on the use of the MAAS-Global instrument, which they subsequently used to assess the consultation skills of 53 GPTs in 176 videotaped consultations (102 with SPs, 74 with RPs). All consultations were randomly allocated and assessed by two teachers independently. The reliability of the ratings was estimated using generalisability theory.
  
  
  RESULTS
  It was easier to obtain acceptable reliability using RP consultations than SP consultations. Two assessors and five consultations were required to achieve minimal reliability (generalisability coefficient 0.7) with RPs, whereas three assessors and 30 consultations were needed to achieve minimal reliability with SPs.
  
  
  CONCLUSIONS
  Inter-observer and context variability in the assessment of the consultation skills of GPTs remains high. To achieve acceptable levels of reliability, large samples of observations are required in both formats, but, interestingly, RP encounters require a smaller sample than SP encounters.
authors.affiliation: EMGO Institute for Health and Care Research, Vrije Universiteit University Medical Centre, Amsterdam, The Netherlands. marcel.reinders@vumc.nl
authors.forename: Marcel E
authors.initials: ME
authors.lastname: Reinders
authors.forename: Annette H
authors.initials: AH
authors.lastname: Blankenstein
authors.forename: Harm W J
authors.initials: HW
authors.lastname: van Marwijk
authors.forename: Dirk L
authors.initials: DL
authors.lastname: Knol
authors.forename: Paul
authors.initials: P
authors.lastname: Ram
authors.forename: Henriette E
authors.initials: HE
authors.lastname: van der Horst
authors.forename: Henrica C W
authors.initials: HC
authors.lastname: de Vet
authors.forename: Cees P M
authors.initials: CP
authors.lastname: van der Vleuten
mesh_terms: D002983:Clinical Competence / Q000592:standards*; D003142:Communication; D004503:Education, Medical, Graduate / Q000379:methods* / Q000592:standards; D004521:Educational Measurement / Q000379:methods / Q000592:standards; D006801:Humans; D015588:Observer Variation; D016544:Patient Simulation; D010817:Physician-Patient Relations; D012017:Referral and Consultation / Q000592:standards*; D013223:Statistics as Topic; D014743:Videotape Recording / Q000706:statistics & numerical data