title: Learning to soar in turbulent environments.
abstract: Birds and gliders exploit warm, rising atmospheric currents (thermals) to reach heights comparable to low-lying clouds with a reduced expenditure of energy. This strategy of flight (thermal soaring) is frequently used by migratory birds. Soaring provides a remarkable instance of complex decision making in biology and requires a long-term strategy to effectively use the ascending thermals. Furthermore, the problem is technologically relevant to extend the flying range of autonomous gliders. Thermal soaring is commonly observed in the atmospheric convective boundary layer on warm, sunny days. The formation of thermals unavoidably generates strong turbulent fluctuations, which constitute an essential element of soaring. Here, we approach soaring flight as a problem of learning to navigate complex, highly fluctuating turbulent environments. We simulate the atmospheric boundary layer by numerical models of turbulent convective flow and combine them with model-free, experience-based, reinforcement learning algorithms to train the gliders. For the learned policies in the regimes of moderate and strong turbulence levels, the glider adopts an increasingly conservative policy as turbulence levels increase, quantifying the degree of risk affordable in turbulent environments. Reinforcement learning uncovers those sensorimotor cues that permit effective control over soaring in turbulent environments.
authors.affiliation: Department of Physics, University of California, San Diego, La Jolla, CA 92093;
authors.forename: Gautam
authors.initials: G
authors.lastname: Reddy
authors.affiliation: The Abdus Salam International Center for Theoretical Physics, I-34014 Trieste, Italy;
authors.forename: Antonio
authors.initials: A
authors.lastname: Celani
authors.affiliation: Howard Hughes Medical Institute, The Salk Institute for Biological Studies, La Jolla, CA 92037; Division of Biological Sciences, University of California, San Diego, La Jolla, CA 92093 terry@salk.edu.
authors.forename: Terrence J
authors.initials: TJ
authors.lastname: Sejnowski
authors.affiliation: Department of Physics, University of California, San Diego, La Jolla, CA 92093;
authors.forename: Massimo
authors.initials: M
authors.lastname: Vergassola
mesh_terms: D000465:Algorithms; D001696:Biomechanical Phenomena; D003463:Cues; D005426:Flight, Animal / Q000502:physiology*; D007858:Learning; D012054:Reinforcement, Psychology; D012201:Reward
keywords: navigation; reinforcement learning; thermal soaring; turbulence