title: Estimating and comparing the reliability of a suite of workplace-based assessments: an obstetrics and gynaecology setting.
abstract: This paper reports on a study that compares estimates of the reliability of a suite of workplace based assessment forms as employed to formatively assess the progress of trainee obstetricians and gynaecologists. The use of such forms of assessment is growing nationally and internationally in many specialties, but there is little research evidence on comparisons by procedure/competency and form-type across an entire specialty. Generalisability theory combined with a multilevel modelling approach is used to estimate variance components, G-coefficients and standard errors of measurement across 13 procedures and three form-types (mini-CEX, OSATS and CbD). The main finding is that there are wide variations in the estimates of reliability across the forms, and that therefore the guidance on assessment within the specialty does not always allow for enough forms per trainee to ensure that the levels of reliability of the process is adequate. There is, however, little evidence that reliability varies systematically by form-type. Methodologically, the problems of accurately estimating reliability in these contexts through the calculation of variance components and, crucially, their associated standard errors are considered. The importance of the use of appropriate methods in such calculations is emphasised, and the unavoidable limitations of research in naturalistic settings are discussed.
authors.affiliation: School of Medicine, University of Leeds, Leeds, LS2 9JT, UK. m.s.homer@leeds.ac.uk
authors.forename: Matt
authors.initials: M
authors.lastname: Homer
authors.forename: Zeryab
authors.initials: Z
authors.lastname: Setna
authors.forename: Vikram
authors.initials: V
authors.lastname: Jha
authors.forename: Jenny
authors.initials: J
authors.lastname: Higham
authors.forename: Trudie
authors.initials: T
authors.lastname: Roberts
authors.forename: Katherine
authors.initials: K
authors.lastname: Boursicot
mesh_terms: D002983:Clinical Competence; D003142:Communication; D003657:Decision Making; D004503:Education, Medical, Graduate / Q000379:methods; D004521:Educational Measurement / Q000379:methods* / Q000592:standards; D006176:Gynecology / Q000193:education* / Q000592:standards; D006801:Humans; D008487:Medical History Taking; D009774:Obstetrics / Q000193:education* / Q000592:standards; D010808:Physical Examination; D015203:Reproducibility of Results; D012189:Retrospective Studies; D017132:Workplace