title: Agreement in Scoring Preclinical Dental Procedures: Impact on Grades and Instructor-Related Determinants.
abstract: The aims of this study were to assess agreement among instructors in scoring preclinical dental procedures, the effect of this agreement on grades, and factors associated with agreement and grades differences. Twenty-six instructors from five clinical specialties participated in a scoring exercise at the College of Dentistry, University of Dammam, Saudi Arabia, in January 2015. Instructors blindly and independently scored 19-20 anonymous students' procedures, filling forms with rating scales as used in preclinical lab sessions. Their scores were compared to those of gold standard instructors in the same specialty. Percent agreement and having one letter grade difference from gold standard instructors were assessed, and associated factors were investigated after controlling for the effect of specialty/student. Agreement between instructors and their specialty gold standards occurred in 54.3% of decisions, with only 12.8% similar scores and 52.8% of scores representing underestimation in relation to gold standards. Instructor-related variables accounted for 40-50% of the variation in percent score as well as percent agreement and percent difference between instructors and gold standards. Educational background and teaching experience were significantly associated with less agreement among instructors and odds of one letter grade difference. Modest agreement existed among instructors in scoring various preclinical procedures, and there was a considerable percentage of one letter grade difference from gold standard instructors. Some factors associated with these outcomes are non-modifiable, but corrective measures are needed to increase agreement in scoring among instructors.
authors.affiliation: Dr. AlHumaid is Assistant Professor, Department of Preventive Dental Sciences, College of Dentistry, University of Dammam, Saudi Arabia; Dr. El Tantawi is Associate Professor, Department of Preventive Dental Sciences, College of Dentistry, University of Dammam, Saudi Arabia; Dr. Al-Ansari is Associate Professor, Department of Preventive Dental Sciences, College of Dentistry, University of Dammam, Saudi Arabia; and Dr. Al-Harbi is Associate Professor, Department of Substitutive Dental Sciences, College of Dentistry, University of Dammam, Saudi Arabia.
authors.forename: Jehan
authors.initials: J
authors.lastname: AlHumaid
authors.affiliation: Dr. AlHumaid is Assistant Professor, Department of Preventive Dental Sciences, College of Dentistry, University of Dammam, Saudi Arabia; Dr. El Tantawi is Associate Professor, Department of Preventive Dental Sciences, College of Dentistry, University of Dammam, Saudi Arabia; Dr. Al-Ansari is Associate Professor, Department of Preventive Dental Sciences, College of Dentistry, University of Dammam, Saudi Arabia; and Dr. Al-Harbi is Associate Professor, Department of Substitutive Dental Sciences, College of Dentistry, University of Dammam, Saudi Arabia. maha_tantawy@hotmail.com.
authors.forename: Maha El
authors.initials: ME
authors.lastname: Tantawi
authors.affiliation: Dr. AlHumaid is Assistant Professor, Department of Preventive Dental Sciences, College of Dentistry, University of Dammam, Saudi Arabia; Dr. El Tantawi is Associate Professor, Department of Preventive Dental Sciences, College of Dentistry, University of Dammam, Saudi Arabia; Dr. Al-Ansari is Associate Professor, Department of Preventive Dental Sciences, College of Dentistry, University of Dammam, Saudi Arabia; and Dr. Al-Harbi is Associate Professor, Department of Substitutive Dental Sciences, College of Dentistry, University of Dammam, Saudi Arabia.
authors.forename: Asim A
authors.initials: AA
authors.lastname: Al-Ansari
authors.affiliation: Dr. AlHumaid is Assistant Professor, Department of Preventive Dental Sciences, College of Dentistry, University of Dammam, Saudi Arabia; Dr. El Tantawi is Associate Professor, Department of Preventive Dental Sciences, College of Dentistry, University of Dammam, Saudi Arabia; Dr. Al-Ansari is Associate Professor, Department of Preventive Dental Sciences, College of Dentistry, University of Dammam, Saudi Arabia; and Dr. Al-Harbi is Associate Professor, Department of Substitutive Dental Sciences, College of Dentistry, University of Dammam, Saudi Arabia.
authors.forename: Fahad A
authors.initials: FA
authors.lastname: Al-Harbi
mesh_terms: D002983:Clinical Competence; D004497:Education, Dental / Q000592:standards; D005179:Faculty, Dental; D005260:Female; D006801:Humans; D008297:Male
keywords: assessment; calibration; dental education; educational measurement; inter-examiner reliability