title: Reinforcement Learning Explains Conditional Cooperation and Its Moody Cousin.
abstract: Direct reciprocity, or repeated interaction, is a main mechanism to sustain cooperation under social dilemmas involving two individuals. For larger groups and networks, which are probably more relevant to understanding and engineering our society, experiments employing repeated multiplayer social dilemma games have suggested that humans often show conditional cooperation behavior and its moody variant. Mechanisms underlying these behaviors largely remain unclear. Here we provide a proximate account for this behavior by showing that individuals adopting a type of reinforcement learning, called aspiration learning, phenomenologically behave as conditional cooperator. By definition, individuals are satisfied if and only if the obtained payoff is larger than a fixed aspiration level. They reinforce actions that have resulted in satisfactory outcomes and anti-reinforce those yielding unsatisfactory outcomes. The results obtained in the present study are general in that they explain extant experimental results obtained for both so-called moody and non-moody conditional cooperation, prisoner's dilemma and public goods games, and well-mixed groups and networks. Different from the previous theory, individuals are assumed to have no access to information about what other individuals are doing such that they cannot explicitly use conditional cooperation rules. In this sense, myopic aspiration learning in which the unconditional propensity of cooperation is modulated in every discrete time step explains conditional behavior of humans. Aspiration learners showing (moody) conditional cooperation obeyed a noisy GRIM-like strategy. This is different from the Pavlov, a reinforcement learning strategy promoting mutual cooperation in two-player situations.
authors.affiliation: Research Center for Advanced Science and Technology, The University of Tokyo, Meguro-ku, Tokyo, Japan.|Japan Society for the Promotion of Science, Kojimachi, Chiyoda-ku, Tokyo, Japan.|National Institute of Informatics, Hitotsubashi, Chiyoda-ku, Tokyo, Japan.|JST, ERATO, Kawarabayashi Large Graph Project, c/o Global Research Center for Big Data Mathematics, NII, Chiyoda-ku, Tokyo, Japan.
authors.forename: Takahiro
authors.initials: T
authors.lastname: Ezaki
authors.affiliation: National Institute of Informatics, Hitotsubashi, Chiyoda-ku, Tokyo, Japan.|JST, ERATO, Kawarabayashi Large Graph Project, c/o Global Research Center for Big Data Mathematics, NII, Chiyoda-ku, Tokyo, Japan.
authors.forename: Yutaka
authors.initials: Y
authors.lastname: Horita
authors.affiliation: Department of Behavioral Science, Hokkaido University, Kita-ku, Sapporo, Japan.|Center for Experimental Research in Social Sciences, Hokkaido University, Kita-ku, Sapporo, Japan.
authors.forename: Masanori
authors.identifier: 0000-0003-4103-0591
authors.initials: M
authors.lastname: Takezawa
authors.affiliation: Department of Engineering Mathematics, University of Bristol, Clifton, Bristol, United Kingdom.
authors.forename: Naoki
authors.initials: N
authors.lastname: Masuda
mesh_terms: D000339:Affect; D019295:Computational Biology; D003299:Cooperative Behavior; D005716:Game Theory; D006801:Humans; D007858:Learning / Q000502:physiology*; D008960:Models, Psychological; D012054:Reinforcement, Psychology; D012919:Social Behavior