title: Lucky guess or knowledge: a cross-sectional study using the Bland and Altman analysis to compare confidence-based testing of pharmacological knowledge in 3rd and 5th year medical students.
abstract: Multiple-choice-questions are common in medical examinations, but guessing biases assessment results. Confidence-based-testing (CBT) integrates indicated confidence levels. It has been suggested that correctness of and confidence in an answer together indicate knowledge levels thus determining the quality of a resulting decision. We used a CBT approach to investigate whether decision quality improves during undergraduate medical education. 3rd- and 5th-year students attended formative multiple-choice exams on pharmacological issues. Students were asked to indicate their confidence in a given answer. Correctness of answers was scored binary (1-correct; 0-wrong) and confidence levels were transformed to an ordinal scale (guess: 0; rather unsure: 0.33; rather sure: 0.66; very sure: 1). 5th-year students gave more correct answers (73 ± 16 vs. 49 ± 13 %, p < 0.05) and were on average more confident regarding the correctness of their answers (0.61 ± 0.18 vs. 0.46 ± 0.13, p < 0.05). Correlation of these parameters was stronger for 5th-year students (r = 0.81 vs. r = 0.52), but agreement of confidence and correctness ('centration') was lower. By combining the Bland-and-Altman approach with categories of decision-quality we found that 5th-year students were more likely to be 'well-informed' (41 vs. 5 %), while more 3rd-students were 'uninformed' (24 vs. 76 %). Despite a good correlation of exam results and confidence in given answers increased knowledge might be accompanied by a more critical view at the own abilities. Combining the statistical Bland-and-Altman analysis with a theoretical approach to decision-quality, more advanced students are expected to apply correct beliefs, while their younger fellows are rather at risk to hesitate or to act amiss.
authors.affiliation: Department of Pharmacology, University of Cologne, Priv.-Doz. Dr. med. Jan Matthes, Gleueler Strasse 24, 50931, Cologne, Germany, daniela.kampmeyer@gmail.com.
authors.forename: Daniela
authors.initials: D
authors.lastname: Kampmeyer
authors.forename: Jan
authors.initials: J
authors.lastname: Matthes
authors.forename: Stefan
authors.initials: S
authors.lastname: Herzig
mesh_terms: D003430:Cross-Sectional Studies; D003657:Decision Making; D004504:Education, Medical, Undergraduate / Q000379:methods*; D004521:Educational Measurement / Q000379:methods*; D006801:Humans; D019359:Knowledge; D010600:Pharmacology; D011336:Probability