title: Limited-information goodness-of-fit testing of diagnostic classification item response models.
abstract: Despite the growing popularity of diagnostic classification models (e.g., Rupp et al., 2010, Diagnostic measurement: theory, methods, and applications, Guilford Press, New York, NY) in educational and psychological measurement, methods for testing their absolute goodness of fit to real data remain relatively underdeveloped. For tests of reasonable length and for realistic sample size, full-information test statistics such as Pearson's X2and the likelihood ratio statistic G2suffer from sparseness in the underlying contingency table from which they are computed. Recently, limited-information fit statistics such as Maydeu-Olivares and Joe's (2006, Psychometrika, 71, 713) M2have been found to be quite useful in testing the overall goodness of fit of item response theory models. In this study, we applied Maydeu-Olivares and Joe's (2006, Psychometrika, 71, 713) M2statistic to diagnostic classification models. Through a series of simulation studies, we found that M2is well calibrated across a wide range of diagnostic model structures and was sensitive to certain misspecifications of the item model (e.g., fitting disjunctive models to data generated according to a conjunctive model), errors in the Q-matrix (adding or omitting paths, omitting a latent variable), and violations of local item independence due to unmodelled testlet effects. On the other hand, M2was largely insensitive to misspecifications in the distribution of higher-order latent dimensions and to the specification of an extraneous attribute. To complement the analyses of the overall model goodness of fit using M2, we investigated the utility of the Chen and Thissen (1997, J. Educ. Behav. Stat., 22, 265) local dependence statistic XLD2 for characterizing sources of misfit, an important aspect of model appraisal often overlooked in favour of overall statements. The XLD2 statistic was found to be slightly conservative (with Type I error rates consistently below the nominal level) but still useful in pinpointing the sources of misfit. Patterns of local dependence arising due to specific model misspecifications are illustrated. Finally, we used the M2and XLD2 statistics to evaluate a diagnostic model fit to data from the Trends in Mathematics and Science Study, drawing upon analyses previously conducted by Lee et al., (2011, IJT, 11, 144).
authors.affiliation: University of California, Los Angeles, California, USA.
authors.forename: Mark
authors.initials: M
authors.lastname: Hansen
authors.affiliation: University of California, Los Angeles, California, USA. lcai@ucla.edu.
authors.forename: Li
authors.initials: L
authors.lastname: Cai
authors.affiliation: University of California, Los Angeles, California, USA.
authors.forename: Scott
authors.initials: S
authors.lastname: Monroe
authors.affiliation: University of California, Los Angeles, California, USA.
authors.forename: Zhen
authors.initials: Z
authors.lastname: Li
mesh_terms: D000465:Algorithms; D003198:Computer Simulation; D003627:Data Interpretation, Statistical; D003936:Diagnosis, Computer-Assisted / Q000379:methods*; D016013:Likelihood Functions; D015233:Models, Statistical; D009716:Numerical Analysis, Computer-Assisted; D017063:Outcome Assessment, Health Care / Q000379:methods* / Q000706:statistics & numerical data; D015203:Reproducibility of Results; D018401:Sample Size; D012680:Sensitivity and Specificity
keywords: diagnostic classification models; item response models; limited-information goodness of fit; local item independence