title: Hidden item variance in multiple mini-interview scores.
abstract: The extant literature has largely ignored a potentially significant source of variance in multiple mini-interview (MMI) scores by "hiding" the variance attributable to the sample of attributes used on an evaluation form. This potential source of hidden variance can be defined as rating items, which typically comprise an MMI evaluation form. Due to its multi-faceted, repeated measures format, reliability for the MMI has been primarily evaluated using generalizability (G) theory. A key assumption of G theory is that G studies model the most important sources of variance to which a researcher plans to generalize. Because G studies can only attribute variance to the facets that are modeled in a G study, failure to model potentially substantial sources of variation in MMI scores can result in biased estimates of variance components. This study demonstrates the implications of hiding the item facet in MMI studies when true item-level effects exist. An extensive Monte Carlo simulation study was conducted to examine whether a commonly used hidden item, person-by-station (p × s|i) G study design results in biased estimated variance components. Estimates from this hidden item model were compared with estimates from a more complete person-by-station-by-item (p × s × i) model. Results suggest that when true item-level effects exist, the hidden item model (p × s|i) will result in biased variance components which can bias reliability estimates; therefore, researchers should consider using the more complete person-by-station-by-item model (p × s × i) when evaluating generalizability of MMI scores.
authors.affiliation: 5117 Taubman Health Sciences Library, Office of Medical Student Education, University of Michigan Medical School, Ann Arbor, MI, 48109-5726, USA. bibler@med.umich.edu.
authors.forename: Nikki L Bibler
authors.initials: NLB
authors.lastname: Zaidi
authors.affiliation: University of Cincinnati College of Education, Cincinnati, OH, USA.
authors.forename: Christopher M
authors.initials: CM
authors.lastname: Swoboda
authors.affiliation: University of Cincinnati College of Education, Cincinnati, OH, USA.
authors.forename: Benjamin M
authors.initials: BM
authors.lastname: Kelcey
authors.affiliation: University of Mississippi School of Medicine, Jackson, MS, USA.
authors.forename: R Stephen
authors.initials: RS
authors.lastname: Manuel
mesh_terms: D003142:Communication; D006801:Humans; D007407:Interviews as Topic / Q000379:methods* / Q000592:standards*; D009010:Monte Carlo Method; D015203:Reproducibility of Results; D012570:School Admission Criteria; D012577:Schools, Medical / Q000592:standards*
keywords: Estimated variance components; Generalizability (G) theory, medical school admissions; Monte Carlo simulation; Multiple mini-interview (MMI)