title: Benchmarking by cross-institutional comparison of student achievement in a progress test.
abstract: OBJECTIVE
  To determine the effectiveness of single-point benchmarking and longitudinal benchmarking for inter-school educational evaluation.
  
  
  METHODS
  We carried out a mixed, longitudinal, cross-sectional study using data from 24 annual measurement moments (4 tests x 6 year groups) over 4 years for 4 annual progress tests assessing the graduation-level knowledge of all students from 3 co-operating medical schools. Participants included undergraduate medical students (about 5000) from 3 medical schools. The main outcome measures involved between-school comparisons of progress test results based on different benchmarking methods.
  
  
  RESULTS
  Variations in relative school performance across different tests and year groups indicate instability and low reliability of single-point benchmarking, which is subject to distortions as a result of school-test and year group-test interaction effects. Deviations of school means from the overall mean follow an irregular, noisy pattern obscuring systematic between-school differences. The longitudinal benchmarking method results in suppression of noise and revelation of systematic differences. The pattern of a school's cumulative deviations per year group gives a credible reflection of the relative performance of year groups.
  
  
  CONCLUSIONS
  Even with highly comparable curricula, single-point benchmarking can result in distortion of the results of comparisons. If longitudinal data are available, the information contained in a school's cumulative deviations from the overall mean can be used. In such a case, the mean test score across schools is a useful benchmark for cross-institutional comparison.
authors.affiliation: Department of Educational Development and Research, Faculty of Medicine, Maastricht University, Maastricht, The Netherlands. a.muijtjens@educ.unimaas.nl
authors.forename: Arno M M
authors.initials: AM
authors.lastname: Muijtjens
authors.forename: Lambert W T
authors.initials: LW
authors.lastname: Schuwirth
authors.forename: Janke
authors.initials: J
authors.lastname: Cohen-Schotanus
authors.forename: Arnold J N M
authors.initials: AJ
authors.lastname: Thoben
authors.forename: Cees P M
authors.initials: CP
authors.lastname: van der Vleuten
mesh_terms: D019985:Benchmarking / Q000458:organization & administration / Q000592:standards*; D003479:Curriculum; D004504:Education, Medical, Undergraduate / Q000379:methods / Q000592:standards*; D004521:Educational Measurement / Q000592:standards*; D007373:Interinstitutional Relations; D009426:Netherlands; D012189:Retrospective Studies; D012577:Schools, Medical