title: Limitations of true score variance to measure discriminating power: psychometric simulation study.
abstract: Demonstrating a specific cognitive deficit usually involves comparing patients' performance on 2 or more tests. The psychometric confound occurs if the psychometric properties of these tests lead patients to show greater cognitive deficits in 1 domain. One way to avoid the psychometric confound is to use tests with a similar level of discriminating power, which is a test's ability to index true individual differences in classic psychometric theory. One suggested way to measure discriminating power is to calculate true score variance (L. J. Chapman & J. P. Chapman, 1978). Despite the centrality of these formulations, there is no systematic examination of the relationship between the observable property of true score variance and the latent property of discriminating power. The authors simulated administrations of free response tests and forced choice tests by creating different replicable ability scores for 2 groups, across a wide range of various psychometric properties (i.e., difficulty, reliability, observed variance, and number of items), and computing an ideal index of discriminating power. Simulation results indicated that true score variance had only limited ability to predict discriminating power (explained about 10% of variance in replicable ability scores). Furthermore, the ability varied across tests with wide ranges of psychometric variables, such as difficulty, observed variance, reliability, and number of items. Discriminating power depends on a complicated interaction of psychometric properties that is not well estimated solely by a test's true score variance.
authors.affiliation: Department of Psychology, Department of Psychiatry, University of Minnesota, Veterans Affairs Medical Center, Minneapolis, Minnesota, USA.
authors.forename: Seung Suk
authors.initials: SS
authors.lastname: Kang
authors.forename: Angus W
authors.initials: AW
authors.lastname: MacDonald
mesh_terms: D003072:Cognition Disorders / Q000175:diagnosis* / Q000523:psychology; D003198:Computer Simulation; D004192:Discrimination, Psychological; D008960:Models, Psychological; D009483:Neuropsychological Tests; D011594:Psychometrics / Q000379:methods*; D011795:Surveys and Questionnaires
references.citation: Arch Gen Psychiatry. 2007 May;64(5):532-42
references.pmid: 17485605
references.citation: Nebr Symp Motiv. 2007;52:1-83
references.pmid: 17682331
references.citation: J Abnorm Psychol. 2001 Feb;110(1):15-30
references.pmid: 11261389
references.citation: J Abnorm Psychol. 2001 Feb;110(1):6-14
references.pmid: 11261401
references.citation: J Psychiatr Res. 1978;14(1-4):303-11
references.pmid: 722633
references.citation: Psychol Bull. 1973 Jun;79(6):380-5
references.pmid: 4707457
references.citation: J Nerv Ment Dis. 1984 Jun;172(6):373-4
references.pmid: 6544811
references.citation: Schizophr Bull. 2008 Jul;34(4):645-55
references.pmid: 18468987
references.citation: Psychometrika. 1949 Jun;14(2):117-9
references.pmid: 18145837