title: The test-retest reliability of the latent construct of executive function depends on whether tasks are represented as formative or reflective indicators.
abstract: This study investigates the test-retest reliability of a battery of executive function (EF) tasks with a specific interest in testing whether the method that is used to create a battery-wide score would result in differences in the apparent test-retest reliability of children's performance. A total of 188 4-year-olds completed a battery of computerized EF tasks twice across a period of approximately two weeks. Two different approaches were used to create a score that indexed children's overall performance on the battery-i.e., (1) the mean score of all completed tasks and (2) a factor score estimate which used confirmatory factor analysis (CFA). Pearson and intra-class correlations were used to investigate the test-retest reliability of individual EF tasks, as well as an overall battery score. Consistent with previous studies, the test-retest reliability of individual tasks was modest (rs ≈ .60). The test-retest reliability of the overall battery scores differed depending on the scoring approach (rmean= .72; rfactor_score= .99). It is concluded that the children's performance on individual EF tasks exhibit modest levels of test-retest reliability. This underscores the importance of administering multiple tasks and aggregating performance across these tasks in order to improve precision of measurement. However, the specific strategy that is used has a large impact on the apparent test-retest reliability of the overall score. These results replicate our earlier findings and provide additional cautionary evidence against the routine use of factor analytic approaches for representing individual performance across a battery of EF tasks.
authors.affiliation: a Education & Workforce Development, RTI International , Research Triangle Park , Durham , NC , USA.
authors.forename: Michael T
authors.initials: MT
authors.lastname: Willoughby
authors.affiliation: b FPG Child Development Institute , University of North Carolina , Chapel Hill , NC , USA.
authors.forename: Laura J
authors.initials: LJ
authors.lastname: Kuhn
authors.affiliation: c Department of Applied Psychology , New York University , New York , NY , USA.
authors.forename: Clancy B
authors.initials: CB
authors.lastname: Blair
authors.affiliation: d Center for Economic & Social Research , University of Southern California , Los Angeles , CA , USA.
authors.forename: Anya
authors.initials: A
authors.lastname: Samek
authors.affiliation: e Department of Economics , University of Chicago , Chicago , IL , USA.
authors.forename: John A
authors.initials: JA
authors.lastname: List
mesh_terms: D001288:Attention / Q000502:physiology*; D002657:Child Development / Q000502:physiology*; D002675:Child, Preschool; D056344:Executive Function / Q000502:physiology*; D005163:Factor Analysis, Statistical; D005260:Female; D006801:Humans; D008297:Male; D009483:Neuropsychological Tests; D011594:Psychometrics; D011597:Psychomotor Performance / Q000502:physiology*; D011930:Reaction Time; D015203:Reproducibility of Results
keywords: Confirmatory factor analysis; Early childhood; Executive function; Formative measurement; Test–retest reliability