Comparison between inter-rater reliability and inter-rater agreement in performance assessment.
BACKGROUND
Over the years, performance assessment (PA) has been widely employed in medical education, Objective Structured Clinical Examination (OSCE) being an excellent example. Typically, performance assessment involves multiple raters, and therefore, consistency among the scores provided by the auditors is a precondition to ensure the accuracy of the assessment. Inter-rater agreement and inter-rater reliability are two indices that are used to ensure such scoring consistency. This research primarily examined the relationship between inter-rater agreement and inter-rater reliability.


METHODS
This study used 3 sets of simulated data that was based on raters' evaluation of student performance to examine the relationship between inter-rater agreement and inter-rater reliability.


RESULTS
Data set 1 had high inter-rater agreement but low inter-rater reliability, data set 2 had high inter-rater reliability but low inter-rater agreement, and data set 3 had high inter-rater agreement and high inter-rater reliability.


CONCLUSIONS
Inter-rater agreement and inter-rater reliability can but do not necessarily coexist. The presence of one does not guarantee that of the other. Inter-rater agreement and inter-rater reliability are both important for PA. The former shows stability of scores a student receives from different raters, while the latter shows the consistence of scores across different students from different raters.
Liao
Shih Chieh
SC
School of Medicine, College of Medicine, China Medical University, Taichung, Taiwan.
Hunt
Elizabeth A
EA
Chen
Walter
W
D002983:Clinical Competence; D003479:Curriculum; D004501:Education, Medical; D006801:Humans; D011594:Psychometrics; D015203:Reproducibility of Results; D013223:Statistics as Topic; D018709:Statistics, Nonparametric; D013337:Students, Medical; D013647:Task Performance and Analysis; D054928:Validation Studies as Topic