Reliability of the evaluation of students' answers to essay-type questions.
BACKGROUND
This paper seeks to quantify the reliability of the assessment of students' answers to essay-type questions, in an attempt to define the role of such questions in University examinations.


METHODS
The marks awarded for essay-type questions during three consecutive final undergraduate examinations in surgery were analyzed. The mean scores, 95% confidence intervals and the standard error of the mean were calculated to determine the distribution of the marks. Statistical analysis was used to determine the correlation of the marks awarded for the same answer by different markers and deduce the dependability of this method of testing.


RESULTS
The marks awarded to 233 answer papers were available for analysis. The marks awarded by each pair of examiners for student answers to individual questions coincided on only 46.3% of occasions, but varied within just +/- 5% on 90.7% of occasions. Use of the kappa index to determine the agreement between markers produced a value of just 0.385, well short of the ideal of 1.0. Assessment of the overall reliability of this type of examination by Cronbach's reliability coefficent gave a value of 0.672.


CONCLUSIONS
There was a significant variation among markers in the evaluation of answers to essay-type questions. However; the overall test reliability was acceptable enough to justify continuation of this type of assessment as a supplement to other methods.
Anatol
T
T
Department of Clinical Surgical Sciences, Faculty of Medical Sciences, Eric Williams Medical Sciences Complex, Trinidad and Tobago. trevana@wow.net
Hariharan
S
S
D004504:Education, Medical, Undergraduate; D004521:Educational Measurement / Q000379:methods*; D005260:Female; D006801:Humans; D008297:Male; D015203:Reproducibility of Results; D012577:Schools, Medical; D014900:West Indies; D014956:Writing; D055815:Young Adult