Model evaluation using grouped or individual data.
Analyzing the data of individuals has several advantages over analyzing the data combined across the individuals (the latter we term group analysis): Grouping can distort the form of data, and different individuals might perform the task using different processes and parameters. These factors notwithstanding, we demonstrate conditions in which group analysis outperforms individual analysis. Such conditions include those in which there are relatively few trials per subject per condition, a situation that sometimes introduces distortions and biases when models are fit and parameters are estimated. We employed a simulation technique in which data were generated from each of two known models, each with parameter variation across simulated individuals. We examined how well the generating model and its competitor each fared in fitting (both sets of) the data, using both individual and group analysis. We examined the accuracy of model selection (the probability that the correct model would be selected by the analysis method). Trials per condition and individuals per experiment were varied systematically. Three pairs of cognitive models were compared: exponential versus power models of forgetting, generalized context versus prototype models of categorization, and the fuzzy logical model of perception versus the linear integration model of information integration. We show that there are situations in which small numbers of trials per condition cause group analysis to outperform individual analysis. Additional tables and figures may be downloaded from the Psychonomic Society Archive of Norms, Stimuli, and Data, www.psychonomic.org/archive.
Cohen
Andrew L
AL
Department of Psychology, University of Massachusetts, Amherst, Massachusetts 01003, USA. acohen@psych.umass.edu
Sanborn
Adam N
AN
Shiffrin
Richard M
RM
D015982:Bias; D003196:Computer Graphics; D003198:Computer Simulation; D003625:Data Collection / Q000706:statistics & numerical data; D003627:Data Interpretation, Statistical; D006801:Humans; D007206:Individuality; D015233:Models, Statistical; D012153:Retention, Psychology; D014706:Verbal Learning
J Math Psychol. 2000 Mar;44(1):171-189
10733863
Psychon Bull Rev. 2005 Aug;12(4):605-21
16447375
J Exp Psychol Learn Mem Cogn. 2002 Mar;28(2):275-92
11911384
J Exp Psychol Learn Mem Cogn. 2002 Sep;28(5):924-40
12219799
Cogn Psychol. 2004 Aug;49(1):47-84
15193972
Psychol Rev. 1978 May;85(3):172-91
663005
Psychol Bull. 1952 May;49(3):263-9
14930162
J Math Psychol. 2000 Mar;44(1):108-132
10733860
J Math Psychol. 2000 Mar;44(1):133-152
10733861
Psychon Bull Rev. 2005 Jun;12(3):403-8
16235625
Science. 1987 Sep 11;237(4820):1317-23
3629243
Mem Cognit. 2000 Jul;28(5):832-40
10983457
Psychol Rev. 1953 Jul;60(4):269-75
13089005
Mem Cognit. 1997 Sep;25(5):724-30
9337590
J Exp Psychol Gen. 1986 Mar;115(1):39-61
2937873
Psychol Bull. 1956 Mar;53(2):134-40
13297917
J Exp Psychol Learn Mem Cogn. 1988 Jan;14(1):33-53
2963894