title: Composite undergraduate clinical examinations: how should the components be combined to maximize reliability?
abstract: BACKGROUND
  Clinical examinations increasingly consist of composite tests to assess all aspects of the curriculum recommended by the General Medical Council.
  
  
  METHODS
  A final undergraduate medical school examination for 214 students.
  
  
  OBJECTIVE
  To estimate the overall reliability of a composite examination, the correlations between the tests, and the effect of differences in test length, number of items and weighting of the results on the reliability.
  
  
  METHODS
  The examination consisted of four written and two clinical tests: multiple-choice questions (MCQ) test, extended matching questions (EMQ), short-answer questions (SAQ), essays, an objective structured clinical examination (OSCE) and history-taking long cases. Multivariate generalizability theory was used to estimate the composite reliability of the examination and the effects of item weighting and test length.
  
  
  RESULTS
  The composite reliability of the examination was 0.77, if all tests contributed equally. Correlations between examination components varied, suggesting that different theoretically interpretable parameters of competence were being tested. Weighting tests according to items per test or total test time gave improved reliabilities of 0.93 and 0.81, respectively. Double weighting of the clinical component marginally affected the reliability (0.76).
  
  
  CONCLUSIONS
  This composite final examination achieved an overall reliability sufficient for high-stakes decisions on student clinical competence. However, examination structure must be carefully planned and results combined with caution. Weighting according to number of items or test length significantly affected reliability. The components testing different aspects of knowledge and clinical skills must be carefully balanced to ensure both content validity and parity between items and test length.
authors.affiliation: Department of General Practice and Primary Care, Guy's, King's and St Thomas' School of Medicine, London, UK.
authors.forename: V
authors.initials: V
authors.lastname: Wass
authors.forename: D
authors.initials: D
authors.lastname: McGibbon
authors.forename: C
authors.initials: C
authors.lastname: Van der Vleuten
mesh_terms: D002983:Clinical Competence / Q000592:standards*; D003479:Curriculum; D004504:Education, Medical, Undergraduate / Q000592:standards*; D004521:Educational Measurement / Q000592:standards; D006801:Humans; D015203:Reproducibility of Results