title: Model evaluation using grouped or individual data.
abstract: Analyzing the data of individuals has several advantages over analyzing the data combined across the individuals (the latter we term group analysis): Grouping can distort the form of data, and different individuals might perform the task using different processes and parameters. These factors notwithstanding, we demonstrate conditions in which group analysis outperforms individual analysis. Such conditions include those in which there are relatively few trials per subject per condition, a situation that sometimes introduces distortions and biases when models are fit and parameters are estimated. We employed a simulation technique in which data were generated from each of two known models, each with parameter variation across simulated individuals. We examined how well the generating model and its competitor each fared in fitting (both sets of) the data, using both individual and group analysis. We examined the accuracy of model selection (the probability that the correct model would be selected by the analysis method). Trials per condition and individuals per experiment were varied systematically. Three pairs of cognitive models were compared: exponential versus power models of forgetting, generalized context versus prototype models of categorization, and the fuzzy logical model of perception versus the linear integration model of information integration. We show that there are situations in which small numbers of trials per condition cause group analysis to outperform individual analysis. Additional tables and figures may be downloaded from the Psychonomic Society Archive of Norms, Stimuli, and Data, www.psychonomic.org/archive.
authors.affiliation: Department of Psychology, University of Massachusetts, Amherst, Massachusetts 01003, USA. acohen@psych.umass.edu
authors.forename: Andrew L
authors.initials: AL
authors.lastname: Cohen
authors.forename: Adam N
authors.initials: AN
authors.lastname: Sanborn
authors.forename: Richard M
authors.initials: RM
authors.lastname: Shiffrin
mesh_terms: D015982:Bias; D003196:Computer Graphics; D003198:Computer Simulation; D003625:Data Collection / Q000706:statistics & numerical data; D003627:Data Interpretation, Statistical; D006801:Humans; D007206:Individuality; D015233:Models, Statistical; D012153:Retention, Psychology; D014706:Verbal Learning
references.citation: J Math Psychol. 2000 Mar;44(1):171-189
references.pmid: 10733863
references.citation: Psychon Bull Rev. 2005 Aug;12(4):605-21
references.pmid: 16447375
references.citation: J Exp Psychol Learn Mem Cogn. 2002 Mar;28(2):275-92
references.pmid: 11911384
references.citation: J Exp Psychol Learn Mem Cogn. 2002 Sep;28(5):924-40
references.pmid: 12219799
references.citation: Cogn Psychol. 2004 Aug;49(1):47-84
references.pmid: 15193972
references.citation: Psychol Rev. 1978 May;85(3):172-91
references.pmid: 663005
references.citation: Psychol Bull. 1952 May;49(3):263-9
references.pmid: 14930162
references.citation: J Math Psychol. 2000 Mar;44(1):108-132
references.pmid: 10733860
references.citation: J Math Psychol. 2000 Mar;44(1):133-152
references.pmid: 10733861
references.citation: Psychon Bull Rev. 2005 Jun;12(3):403-8
references.pmid: 16235625
references.citation: Science. 1987 Sep 11;237(4820):1317-23
references.pmid: 3629243
references.citation: Mem Cognit. 2000 Jul;28(5):832-40
references.pmid: 10983457
references.citation: Psychol Rev. 1953 Jul;60(4):269-75
references.pmid: 13089005
references.citation: Mem Cognit. 1997 Sep;25(5):724-30
references.pmid: 9337590
references.citation: J Exp Psychol Gen. 1986 Mar;115(1):39-61
references.pmid: 2937873
references.citation: Psychol Bull. 1956 Mar;53(2):134-40
references.pmid: 13297917
references.citation: J Exp Psychol Learn Mem Cogn. 1988 Jan;14(1):33-53
references.pmid: 2963894