A Comprehensive, Multi-modal Evaluation of the Assessment System of an Undergraduate Research Methodology Course: Translating Theory into Practice.
OBJECTIVE
To evaluate assessment system of the 'Research Methodology Course' using utility criteria (i.e. validity, reliability, acceptability, educational impact, and cost-effectiveness). This study demonstrates comprehensive evaluation of assessment system and suggests a framework for similar courses.


METHODS
Qualitative and quantitative methods used for evaluation of the course assessment components (50 MCQ, 3 Short Answer Questions (SAQ) and research project) using the utility criteria. RESULTS of multiple evaluation methods for all the assessment components were collected and interpreted together to arrive at holistic judgments, rather than judgments based on individual methods or individual assessment.


RESULTS
Face validity, evaluated using a self-administered questionnaire (response rate-88.7%) disclosed that the students perceived that there was an imbalance in the contents covered by the assessment. This was confirmed by the assessment blueprint. Construct validity was affected by the low correlation between MCQ and SAQ scores (r=0.326). There was a higher correlation between the project and MCQ (r=0.466)/SAQ (r=0.463) scores. Construct validity was also affected by the presence of recall type of MCQs (70%; 35/50), item construction flaws and non-functioning distractors. High discriminating indices (>0.35) were found in MCQs with moderate difficulty indices (0.3-0.7). Reliability of the MCQs was 0.75 which could be improved up to 0.8 by increasing the number of MCQs to at least 70. A positive educational impact was found in the form of the research project assessment driving students to present/publish their work in conferences/peer reviewed journals. Cost per student to complete the course was US$164.50.


CONCLUSIONS
The multi-modal evaluation of an assessment system is feasible and provides thorough and diagnostic information. Utility of the assessment system could be further improved by modifying the psychometrically inappropriate assessment items.
Hamza Mohammad Abdulghani, FRCGP, Associate Professor, College of Medicine, King Saud University, P.O. Box: 230155, Riyadh 11321, Kingdom of Saudi Arabia.
Hamza
H
Mohammad Abdulghani
Gominda G. Ponnamperuma, PhD, College of Medicine, University of Colombo, Colombo, Sri Lanka.
Gominda
G
G Ponnamperuma
Farah Ahmad, PhD, Department of Medical Education, College of Medicine, King Saud University.
Farah
F
Ahmad
Zubair Amin, MD, Associate Professor, Department of Pediatrics, College of Medicine, National University of Singapore, Singapore.
Zubair
Z
Amin
Assessment; Evaluation; Research Course; Utility Criteria