title: Classification Accuracy of Mixed Format Tests: A Bi-Factor Item Response Theory Approach.
abstract: Mixed format tests (e.g., a test consisting of multiple-choice [MC] items and constructed response [CR] items) have become increasingly popular. However, the latent structure of item pools consisting of the two formats is still equivocal. Moreover, the implications of this latent structure are unclear: For example, do constructed response items tap reasoning skills that cannot be assessed with multiple choice items? This study explored the dimensionality of mixed format tests by applying bi-factor models to 10 tests of various subjects from the College Board's Advanced Placement (AP) Program and compared the accuracy of scores based on the bi-factor analysis with scores derived from a unidimensional analysis. More importantly, this study focused on a practical and important question-classification accuracy of the overall grade on a mixed format test. Our findings revealed that the degree of multidimensionality resulting from the mixed item format varied from subject to subject, depending on the disattenuated correlation between scores from MC and CR subtests. Moreover, remarkably small decrements in classification accuracy were found for the unidimensional analysis when the disattenuated correlations exceeded 0.90.
authors.affiliation: Department of Psychology, University of Central Florida Orlando, FL, USA.
authors.forename: Wei
authors.initials: W
authors.lastname: Wang
authors.affiliation: Department of Psychology and School of Labor and Employment Relations, University of Illinois at Urbana-Champaign Champaign, IL, USA.
authors.forename: Fritz
authors.initials: F
authors.lastname: Drasgow
authors.affiliation: Department of Psychology, University of Illinois at Urbana-ChampaignChampaign, IL, USA; American Institutes for ResearchWashington, DC, USA.
authors.forename: Liwen
authors.initials: L
authors.lastname: Liu
keywords: bi-factor model; classification accuracy; constructed response items; item response theory; mixed format test