title: Implicit Value Updating Explains Transitive Inference Performance: The Betasort Model.
abstract: Transitive inference (the ability to infer that B > D given that B > C and C > D) is a widespread characteristic of serial learning, observed in dozens of species. Despite these robust behavioral effects, reinforcement learning models reliant on reward prediction error or associative strength routinely fail to perform these inferences. We propose an algorithm called betasort, inspired by cognitive processes, which performs transitive inference at low computational cost. This is accomplished by (1) representing stimulus positions along a unit span using beta distributions, (2) treating positive and negative feedback asymmetrically, and (3) updating the position of every stimulus during every trial, whether that stimulus was visible or not. Performance was compared for rhesus macaques, humans, and the betasort algorithm, as well as Q-learning, an established reward-prediction error (RPE) model. Of these, only Q-learning failed to respond above chance during critical test trials. Betasort's success (when compared to RPE models) and its computational efficiency (when compared to full Markov decision process implementations) suggests that the study of reinforcement learning in organisms will be best served by a feature-driven approach to comparing formal models.
authors.affiliation: Department of Neuroscience, Columbia University, New York, New York, United States of America; Department of Psychology, Columbia University, New York, New York, United States of America.
authors.forename: Greg
authors.initials: G
authors.lastname: Jensen
authors.affiliation: Department of Neuroscience, Columbia University, New York, New York, United States of America.
authors.forename: Fabian
authors.initials: F
authors.lastname: Muñoz
authors.affiliation: Department of Neuroscience, Columbia University, New York, New York, United States of America.
authors.forename: Yelda
authors.initials: Y
authors.lastname: Alkan
authors.affiliation: Department of Neuroscience, Columbia University, New York, New York, United States of America; Department of Psychiatry, Columbia University, New York, New York, United States of America.
authors.forename: Vincent P
authors.initials: VP
authors.lastname: Ferrera
authors.affiliation: Department of Psychology, Columbia University, New York, New York, United States of America.
authors.forename: Herbert S
authors.initials: HS
authors.lastname: Terrace
mesh_terms: D000465:Algorithms; D000818:Animals; D019295:Computational Biology; D006801:Humans; D007858:Learning / Q000502:physiology*; D008253:Macaca mulatta; D000069550:Machine Learning; D008297:Male; D008959:Models, Neurological; D015233:Models, Statistical; D012201:Reward