title: Investigating disparity between global grades and checklist scores in OSCEs.
abstract: BACKGROUND
  When measuring assessment quality, increasing focus is placed on the value of station-level metrics in the detection and remediation of problems in the assessment.
  
  
  OBJECTIVE
  This article investigates how disparity between checklist scores and global grades in an Objective Structured Clinical Examination (OSCE) can provide powerful new insights at the station level whenever such disparities occur and develops metrics to indicate when this is a problem.
  
  
  METHODS
  This retrospective study uses OSCE data from multiple examinations to investigate the extent to which these new measurements of disparity complement existing station-level metrics.
  
  
  RESULTS
  In stations where existing metrics are poor, the new metrics provide greater understanding of the underlying sources of error. Equally importantly, stations of apparently satisfactory "quality" based on traditional metrics are shown to sometimes have problems of their own - with a tendency for checklist score "performance" to be judged stronger than would be expected from the global grades awarded.
  
  
  CONCLUSIONS
  There is an ongoing tension in OSCE assessment between global holistic judgements and the necessarily more reductionist, but arguably more objective, checklist scores. This article develops methods to quantify the disparity between these judgements and illustrates how such analyses can inform ongoing improvement in station quality.
authors: a University of Leeds , UK.
authors: Godfrey
authors: G
authors: Pell
authors: a University of Leeds , UK.
authors: Matt
authors: M
authors: Homer
authors: a University of Leeds , UK.
authors: Richard
authors: R
authors: Fuller
mesh_terms: D000328:Adult; D002983:Clinical Competence; D003657:Decision Making; D004504:Education, Medical, Undergraduate; D004521:Educational Measurement / Q000379:methods*; D005260:Female; D006801:Humans; D008297:Male; D011340:Problem Solving; D015203:Reproducibility of Results; D012189:Retrospective Studies; D012577:Schools, Medical; D013337:Students, Medical / Q000523:psychology / Q000706:statistics & numerical data; D006113:United Kingdom; D055815:Young Adult