Development of a competency-based formative progress test with student-generated MCQs: Results from a multi-centre pilot study.
BACKGROUND
Progress tests provide students feedback on their level of proficiency over the course of their medical studies. Peer-assisted learning and competency-based education have become increasingly important in medical education. Although progress tests have been proven to be useful as a longitudinal feedback instrument, there are currently no progress tests that have been created in cooperation with students or that focus on competency in medical education. In this study, we investigated the extent to which students can be included in the development of a progress test and demonstrated that aspects of knowledge related to competency can be represented on a competency-based progress test.


METHODS
A two-dimensional blueprint for 144 multiple-choice questions (MCQs) covering groups of medical subjects and groups of competency areas was generated by three expert groups for developing the competency-based progress test. A total of 31 students from seven medical schools in Germany actively participated in this exercise. After completing an intensive and comprehensive training programme, the students generated and reviewed the test questions for the competency-based progress test using a separate platform of the ItemManagementSystem (IMS). This test was administered as a formative test to 469 students in a pilot study in November 2013 at eight medical schools in Germany. The scores were analysed for the overall test and differentiated according to the subject groups and competency areas.


RESULTS
A pool of more than 200 MCQs was compiled by the students for pilot use, of which 118 student-generated MCQs were used in the progress test. University instructors supplemented this pool with 26 MCQs, which primarily addressed the area of scientific skills. The post-review showed that student-generated MCQs were of high quality with regard to test statistic criteria and content. Overall, the progress test displayed a very high reliability. When the academic years were compared, the progress test mapped out over the course of study not only by the overall test but also in terms of the subject groups and competency areas.


CONCLUSIONS
Further development in cooperation with students will be continued. Focus will be on compiling additional questions and test formats that can represent competency at a higher skill level, such as key feature questions, situational judgement test questions and OSCE. In addition, the feedback formats will be successively expanded. The intention is also to offer the formative competency-based progress test online.
University of Heidelberg, Faculty of Medicine, Heidelberg, Germany.
Stefan
S
Wagener
University of Heidelberg, Center of Excellence for Assessment in Medicine - Baden-Wuerttemberg, Heidelberg, Germany.
Andreas
A
Möltner
University of Heidelberg, Faculty of Medicine, Heidelberg, Germany.
Sevgi
S
Tımbıl
University of Heidelberg, Center of Excellence for Assessment in Medicine - Baden-Wuerttemberg, Heidelberg, Germany.
Maryna
M
Gornostayeva
University of Heidelberg, Faculty of Medicine, Heidelberg, Germany.
Jobst-Hendrik
JH
Schultz
Albert-Ludwigs-University Freiburg, Freiburg Competence Center for Evaluation of Teaching in Medicine - Baden-Wuerttemberg, Freiburg, Germany.
Peter
P
Brüstle
University of Tübingen, Faculty of Medicine, Tübingen, Germany.
Daniela
D
Mohr
University of Ulm, Faculty of Medicine, Ulm, Germany.
Anna
A
Vander Beken
University of Marburg, Faculty of Medicine, Marburg, Germany.
Julian
J
Better
University of Marburg, Faculty of Medicine, Marburg, Germany.
Martin
M
Fries
University of Magdeburg, Faculty of Medicine, Magdeburg, Germany.
Marc
M
Gottschalk
University of Freiburg, Faculty of Medicine, Freiburg, Germany.
Janine
J
Günther
University of Freiburg, Faculty of Medicine, Freiburg, Germany.
Laura
L
Herrmann
University of Marburg, Faculty of Medicine, Marburg, Germany.
Christian
C
Kreisel
University of Witten/Herdecke, Faculty of Medicine, Witten, Germany.
Tobias
T
Moczko
University of Heidelberg, Faculty of Medicine, Heidelberg, Germany.
Claudius
C
Illg
University of Heidelberg, Faculty of Medicine, Heidelberg, Germany.
Adam
A
Jassowicz
University of Heidelberg, Faculty of Medicine, Heidelberg, Germany.
Andreas
A
Müller
University of Heidelberg, Faculty of Medicine, Heidelberg, Germany.
Moritz
M
Niesert
University of Heidelberg, Faculty of Medicine, Heidelberg, Germany.
Felix
F
Strübing
University of Heidelberg, Center of Excellence for Assessment in Medicine - Baden-Wuerttemberg, Heidelberg, Germany.
Jana
J
Jünger
D002983:Clinical Competence / Q000592:standards*; D003162:Competency-Based Education / Q000592:standards*; D003479:Curriculum / Q000592:standards; D004504:Education, Medical, Undergraduate / Q000592:standards*; D004521:Educational Measurement / Q000592:standards*; D005246:Feedback; D006801:Humans; D008137:Longitudinal Studies; D010865:Pilot Projects; D058028:Research Report / Q000592:standards*; D013337:Students, Medical
competency-based; medical education; medical students; multiple-choice questions; progress test