title: A Method for Investigating "Instructional Familiarity" and Discerning Authentic Learning.
abstract: BACKGROUND
  Presently, most medical educators rely exclusively on item difficulty and discrimination indices to investigate an item's psychometric quality and functioning. We argue "instructional familiarity" effects should also be of primary concern for persons attempting to discern the quality and meaning of a set of test scores.
  
  
  OBJECTIVE
  There were four primary objectives of this study: (1) Revisit Haladyna and Roid's conceptualization of "instructional sensitivity" within the context of criterion-referenced assessments, (2) provide an overview of "instructional familiarity" and its importance, (3) reframe the concept for a modern audience concerned with medical school assessments, and (4) conduct an empirical evaluation of a medical school examination in which we attempt to investigate the instructional effects on person and item measures.
  
  
  METHODS
  This study involved a medical school course instructor providing ratings of instructional familiarity (IF) for each mid-term examination item, and a series of psychometric analyses to investigate the effects of IF on students' scores and item statistics. The methodology used in this study is based primarily on a mixed-method, "action research" design for a medical school course focusing on endocrinology. Rasch measurement model; correlation analysis.
  
  
  RESULTS
  The methodology presented in this article was evidenced to better discern authentic learning than traditional approaches that ignore valuable contextual information about students' familiarity with exam items.
  
  
  CONCLUSIONS
  The authors encourage other medical educators to adopt this straightforward methodology so as to increase the likelihood of making valid inferences about learning.
authors.affiliation: Department of Clinical Sciences, North Carolina State University, Raleigh, North Carolina, USA.
authors.forename: K D
authors.initials: KD
authors.lastname: Royal
authors.affiliation: Office of Veterinary Medical Education, North Carolina State University, Raleigh, North Carolina, USA.
authors.forename: M W
authors.initials: MW
authors.lastname: Hedgpeth
authors.affiliation: Office of Medical Education, University of North Carolina, Chapel Hill, North Carolina, USA.
authors.forename: K W
authors.initials: KW
authors.lastname: Smith
authors.affiliation: Department of Medicine, University of North Carolina, Chapel Hill, North Carolina, USA.
authors.forename: D
authors.initials: D
authors.lastname: Kirk
keywords: Action research; Assessment; Medical education; Psychometrics; Testing