Assessing individual differences in categorical data.
In cognitive modeling, data are often categorical observations taken over participants and items. Usually subsets of these observations are pooled and analyzed by a cognitive model assuming the category counts come from a multinomial distribution with the same model parameters underlying all observations. It is well known that if there are individual differences in participants and/or items, a model analysis of the pooled data may be quite misleading, and in such cases it may be appropriate to augment the cognitive model with parametric random effects assumptions. On the other hand, if random effects are incorporated into a cognitive model that is not needed, the resulting model may be more flexible than the multinomial model that assumes no heterogeneity, and this may lead to overfitting. This article presents Monte Carlo statistical tests for directly detecting individual participant and/or item heterogeneity that depend only on the data structure itself. These tests are based on the fact that heterogeneity in participants and/or items results in overdispersion of certain category count statistics. It is argued that the methods developed in the article should be applied to any set of participant x item categorical data prior to cognitive model-based analyses.
Smith
Jared B
JB
University of California, Irvine, California 92697-5100, USA.
Batchelder
William H
WH
D001288:Attention; D016009:Chi-Square Distribution; D003071:Cognition; D003198:Computer Simulation; D003625:Data Collection / Q000706:statistics & numerical data*; D003627:Data Interpretation, Statistical; D006801:Humans; D007206:Individuality; D011939:Mental Recall; D009010:Monte Carlo Method; D010153:Paired-Associate Learning; D011932:Reading; D021641:Recognition, Psychology; D012559:Schizophrenia / Q000175:diagnosis; D012565:Schizophrenic Psychology; D012691:Serial Learning; D014706:Verbal Learning
Biometrics. 1973 Dec;29(4):637-48
4785230
Psychon Bull Rev. 2004 Aug;11(4):579-615
15581115
J Exp Psychol Learn Mem Cogn. 2006 Mar;32(2):316-32
16569149
J Exp Psychol Learn Mem Cogn. 2002 Mar;28(2):392-406
11911396
Psychon Bull Rev. 2005 Aug;12(4):605-21
16447375
Psychon Bull Rev. 2005 Aug;12(4):573-604
16447374
Percept Psychophys. 2001 Nov;63(8):1421-55
11800466
Psychol Rep. 1995 Dec;77(3 Pt 2):1379-90
8643807
Psychol Rev. 2001 Jan;108(1):33-56
11212631
Psychol Bull. 1956 Mar;53(2):134-40
13297917
Psychon Bull Rev. 2005 Jun;12(3):409-41
16235626
Stat Med. 1999 Feb 15;18(3):287-305
10070675
Psychol Rev. 1993 Jan;100(1):143-8; discussion 149-53
8426879
Psychon Bull Rev. 2000 Jun;7(2):185-207
10909131
J Vis. 2005 May 27;5(5):478-92
16097878
Psychol Assess. 2002 Jun;14(2):184-201
12056081
J Exp Psychol Learn Mem Cogn. 1995 May;21(3):531-47
7602260
Psychon Bull Rev. 1999 Mar;6(1):57-86
12199315
J Gerontol B Psychol Sci Soc Sci. 1997 Sep;52(5):P206-15
9310089
Psychol Rev. 2002 Oct;109(4):710-21
12374325