Using a sampling strategy to address psychometric challenges in tutorial-based assessments.
BACKGROUND
Tutorial-based assessment, despite providing a good match with the philosophy adopted by educational programmes that emphasize small group learning, remains one of the greatest challenges for educators working in this context. The current study was performed in an attempt to assess the psychometric characteristics of tutorial-based evaluation upon adopting a multiple sampling approach that requires minimal recording of observations.


METHODS
After reviewing the literature, a simple 3-item evaluation form was created. The items were "Professional Behaviour," "Contribution to Group Process," and "Contribution to Group Content." Explicit definition of these items was provided on an evaluation form. Twenty five tutors in five different programmes were asked to use the form to evaluate their students (N=169) after every tutorial over the course of an academic unit. Each item was rated using a 10-point scale.


RESULTS
Cronbach's alpha revealed an appropriate internal consistency in all five programmes. Test-retest reliability of any single rating was low, but the reliability of the average rating was at least 0.75 in all cases. The construct validity of the tool was supported by the observation of increasing ratings over the course of the academic unit and by the finding that more senior students received higher ratings than more junior students.


CONCLUSIONS
Consistent with the context specificity phenomenon, the adoption of a "minimal observations often" approach to tutorial-based assessment appears to maintain better psychometric characteristics than do attempts to assess tutorial performance using more comprehensive measurement tools.
Eva
Kevin W
KW
Department of Clinical Epidemiology and Biostatistics, Program for Educational Research and Development, MDCL 3510, McMaster University, L8N 3Z5, Hamilton, ON, Canada. evakw@mcmaster.ca
Solomon
Patty
P
Neville
Alan J
AJ
Ladouceur
Michael
M
Kaufman
Karyn
K
Walsh
Allyn
A
Norman
Geoffrey R
GR
D003479:Curriculum; D004521:Educational Measurement / Q000379:methods*; D006126:Group Processes; D006801:Humans; D007858:Learning; D015397:Program Evaluation / Q000379:methods*; D011594:Psychometrics / Q000295:instrumentation*; D013336:Students, Health Occupations