title: Application of a utility analysis to evaluate a novel assessment tool for clinically oriented physiology and pharmacology.
abstract: Multiple-choice questions are a gold-standard tool in medical school for assessment of knowledge and are the mainstay of licensing examinations. However, multiple-choice questions items can be criticized for lacking the ability to test higher-order learning or integrative thinking across multiple disciplines. Our objective was to develop a novel assessment that would address understanding of pathophysiology and pharmacology, evaluate learning at the levels of application, evaluation and synthesis, and allow students to demonstrate clinical reasoning. The rubric assesses student writeups of clinical case problems. The method is based on the physician's traditional postencounter Subjective, Objective, Assessment and Plan note. Students were required to correctly identify subjective and objective findings in authentic clinical case problems, to ascribe pathophysiological as well as pharmacological mechanisms to these findings, and to justify a list of differential diagnoses. A utility analysis was undertaken to evaluate the new assessment tool by appraising its reliability, validity, feasibility, cost effectiveness, acceptability, and educational impact using a mixed-method approach. The Subjective, Objective, Assessment and Plan assessment tool scored highly in terms of validity and educational impact and had acceptable levels of statistical reliability but was limited in terms of acceptance, feasibility, and cost effectiveness due to high time demands on expert graders and workload concerns from students. We conclude by making suggestions for improving the tool and recommend deployment of the instrument for low-stakes summative assessment or formative assessment.
authors.affiliation: College of Medicine, University of Central Florida, Orlando, Florida.
authors.forename: Nicholas
authors.initials: N
authors.lastname: Cramer
authors.affiliation: College of Medicine, University of Central Florida, Orlando, Florida.
authors.forename: Abdo
authors.initials: A
authors.lastname: Asmar
authors.affiliation: College of Medicine, University of Central Florida, Orlando, Florida.
authors.forename: Laurel
authors.initials: L
authors.lastname: Gorman
authors.affiliation: College of Medicine, University of Central Florida, Orlando, Florida.
authors.forename: Bernard
authors.initials: B
authors.lastname: Gros
authors.affiliation: College of Medicine, University of Central Florida, Orlando, Florida.
authors.forename: David
authors.initials: D
authors.lastname: Harris
authors.affiliation: College of Medicine, University of Central Florida, Orlando, Florida.
authors.forename: Thomas
authors.initials: T
authors.lastname: Howard
authors.affiliation: College of Medicine, University of Central Florida, Orlando, Florida.
authors.forename: Mujtaba
authors.initials: M
authors.lastname: Hussain
authors.affiliation: College of Medicine, University of Central Florida, Orlando, Florida.
authors.forename: Sergio
authors.initials: S
authors.lastname: Salazar
authors.affiliation: College of Medicine, University of Central Florida, Orlando, Florida jonathan.kibble@ucf.edu.
authors.forename: Jonathan D
authors.initials: JD
authors.lastname: Kibble
mesh_terms: D000328:Adult; D002983:Clinical Competence / Q000592:standards; D015331:Cohort Studies; D004504:Education, Medical, Undergraduate / Q000379:methods* / Q000592:standards; D004521:Educational Measurement / Q000379:methods* / Q000592:standards; D005260:Female; D017144:Focus Groups / Q000379:methods; D006801:Humans; D008297:Male; D010600:Pharmacology / Q000193:education*; D010827:Physiology / Q000193:education*; D015203:Reproducibility of Results; D013337:Students, Medical; D055815:Young Adult
keywords: Bloom's taxonomy; assessment; curriculum integration; evaluation; medical education; multiple-choice questions; physiology education