Analysis of residuals in contingency tables: another nail in the coffin of conditional approaches to significance testing.
Omnibus tests of significance in contingency tables use statistics of the chi-square type. When the null is rejected, residual analyses are conducted to identify cells in which observed frequencies differ significantly from expected frequencies. Residual analyses are thus conditioned on a significant omnibus test. Conditional approaches have been shown to substantially alter type I error rates in cases involving t tests conditional on the results of a test of equality of variances, or tests of regression coefficients conditional on the results of tests of heteroscedasticity. We show that residual analyses conditional on a significant omnibus test are also affected by this problem, yielding type I error rates that can be up to 6 times larger than nominal rates, depending on the size of the table and the form of the marginal distributions. We explored several unconditional approaches in search for a method that maintains the nominal type I error rate and found out that a bootstrap correction for multiple testing achieved this goal. The validity of this approach is documented for two-way contingency tables in the contexts of tests of independence, tests of homogeneity, and fitting psychometric functions. Computer code in MATLAB and R to conduct these analyses is provided as Supplementary Material.
Departamento de Metodología, Facultad de Psicología, Universidad Complutense, Campus de Somosaguas, 28223, Madrid, Spain, miguel@psi.ucm.es.
Miguel A
MA
García-Pérez
Vicente
V
Núñez-Antón
Rocío
R
Alcalá-Quintana
D001699:Biometry; D016009:Chi-Square Distribution; D003205:Computing Methodologies; D006801:Humans; D015999:Multivariate Analysis; D011594:Psychometrics / Q000379:methods*; D015203:Reproducibility of Results; D013597:Systems Analysis