title: Initial clinical validation of an embedded performance validity measure within the automated neuropsychological metrics (ANAM).
abstract: The measurement of effort and performance validity is essential for computerized testing where less direct supervision is needed. The clinical validation of an Automated Neuropsychological Metrics-Performance Validity Index (ANAM-PVI) was examined by converting ANAM test scores into a common metric based on their relative infrequency in an outpatient clinic sample with presumed good effort. Optimal ANAM-PVI cut-points were determined using receiver operator characteristic (ROC) curve analyses and an a priori specificity of 90%. Sensitivity/specificity was examined in available validation samples (controls, simulators, and neurorehabilitation patients). ANAM-PVI scores differed between groups with simulators scoring the highest. ROC curve analysis indicated excellent discriminability of ANAM-PVI scores ≥5 to detect simulators versus controls (area under the curve = 0.858; odds ratio for detecting suboptimal performance = 15.6), but resulted in a 27% false-positive rate in the clinical sample. When specificity in the clinical sample was set at 90%, sensitivity decreased (68%), but was consistent with other embedded effort measures. Results support the ANAM-PVI as an embedded effort measure and demonstrate the value of sample-specific cut-points in groups with cognitive impairment. Examination of different cut-points indicates that clinicians should choose sample-specific cut-points based on sensitivity and specificity rates that are most appropriate for their patient population with higher cut-points for those expected to have severe cognitive impairment (e.g., dementia or severe acquired brain injury).
authors.affiliation: Cognitive Science Research Center, University of Oklahoma, Norman, OK, USA.
authors.forename: Tresa M
authors.initials: TM
authors.lastname: Roebuck-Spencer
authors.forename: Andrea S
authors.initials: AS
authors.lastname: Vincent
authors.forename: Kirby
authors.initials: K
authors.lastname: Gilliland
authors.forename: Dan R
authors.initials: DR
authors.lastname: Johnson
authors.forename: Douglas B
authors.initials: DB
authors.lastname: Cooper
mesh_terms: D000293:Adolescent; D000328:Adult; D000368:Aged; D003072:Cognition Disorders / Q000175:diagnosis* / Q000523:psychology; D005260:Female; D006801:Humans; D008297:Male; D008875:Middle Aged; D009483:Neuropsychological Tests; D012372:ROC Curve; D012680:Sensitivity and Specificity
keywords: ANAM; Computer-based testing; Neuropsychology; Performance Validity Testing; Simulators