title: Development of an itemwise efficiency scoring method: Concurrent, convergent, discriminant, and neuroimaging-based predictive validity assessed in a large community sample.
abstract: Traditional "paper-and-pencil" testing is imprecise in measuring speed and hence limited in assessing performance efficiency, but computerized testing permits precision in measuring itemwise response time. We present a method of scoring performance efficiency (combining information from accuracy and speed) at the item level. Using a community sample of 9,498 youths age 8-21, we calculated item-level efficiency scores on 4 neurocognitive tests, and compared the concurrent, convergent, discriminant, and predictive validity of these scores with simple averaging of standardized speed and accuracy-summed scores. Concurrent validity was measured by the scores' abilities to distinguish men from women and their correlations with age; convergent and discriminant validity were measured by correlations with other scores inside and outside of their neurocognitive domains; predictive validity was measured by correlations with brain volume in regions associated with the specific neurocognitive abilities. Results provide support for the ability of itemwise efficiency scoring to detect signals as strong as those detected by standard efficiency scoring methods. We find no evidence of superior validity of the itemwise scores over traditional scores, but point out several advantages of the former. The itemwise efficiency scoring method shows promise as an alternative to standard efficiency scoring methods, with overall moderate support from tests of 4 different types of validity. This method allows the use of existing item analysis methods and provides the convenient ability to adjust the overall emphasis of accuracy versus speed in the efficiency score, thus adjusting the scoring to the real-world demands the test is aiming to fulfill. (PsycINFO Database Record
authors: Department of Psychiatry, University of Pennsylvania.
authors: Tyler M
authors: TM
authors: Moore
authors: Department of Psychology, University of California.
authors: Steven P
authors: SP
authors: Reise
authors: Department of Psychiatry, University of Pennsylvania.
authors: David R
authors: DR
authors: Roalf
authors: Department of Psychiatry, University of Pennsylvania.
authors: Theodore D
authors: TD
authors: Satterthwaite
authors: Section of Biomedical Image Analysis, University of Pennsylvania.
authors: Christos
authors: C
authors: Davatzikos
authors: Department of Biostatistics and Epidemiology, University of Pennsylvania.
authors: Warren B
authors: WB
authors: Bilker
authors: Department of Psychiatry, University of Pennsylvania.
authors: Allison M
authors: AM
authors: Port
authors: Department of Psychiatry, University of Pennsylvania.
authors: Chad T
authors: CT
authors: Jackson
authors: Department of Psychiatry, University of Pennsylvania.
authors: Kosha
authors: K
authors: Ruparel
authors: Department of Psychiatry, University of Pennsylvania.
authors: Adam P
authors: AP
authors: Savitt
authors: Department of Psychiatry, University of Pennsylvania.
authors: Robert B
authors: RB
authors: Baron
authors: Department of Psychiatry, University of Pennsylvania.
authors: Raquel E
authors: RE
authors: Gur
authors: Department of Psychiatry, University of Pennsylvania.
authors: Ruben C
authors: RC
authors: Gur
mesh_terms: D000293:Adolescent; D000465:Algorithms; D002648:Child; D003201:Computers; D004526:Efficiency; D005260:Female; D006801:Humans; D008297:Male; D059906:Neuroimaging; D009483:Neuropsychological Tests; D011446:Prospective Studies; D011594:Psychometrics; D011930:Reaction Time; D015203:Reproducibility of Results; D055815:Young Adult