title: Reinforcement learning solution for HJB equation arising in constrained optimal control problem.
abstract: The constrained optimal control problem depends on the solution of the complicated Hamilton-Jacobi-Bellman equation (HJBE). In this paper, a data-based off-policy reinforcement learning (RL) method is proposed, which learns the solution of the HJBE and the optimal control policy from real system data. One important feature of the off-policy RL is that its policy evaluation can be realized with data generated by other behavior policies, not necessarily the target policy, which solves the insufficient exploration problem. The convergence of the off-policy RL is proved by demonstrating its equivalence to the successive approximation approach. Its implementation procedure is based on the actor-critic neural networks structure, where the function approximation is conducted with linearly independent basis functions. Subsequently, the convergence of the implementation procedure with function approximation is also proved. Finally, its effectiveness is verified through computer simulations.
authors.affiliation: The State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China. Electronic address: biao.luo@hotmail.com.
authors.forename: Biao
authors.initials: B
authors.lastname: Luo
authors.affiliation: Science and Technology on Aircraft Control Laboratory, Beihang University (Beijing University of Aeronautics and Astronautics), Beijing 100191, China. Electronic address: whn@buaa.edu.cn.
authors.forename: Huai-Ning
authors.initials: HN
authors.lastname: Wu
authors.affiliation: Texas A&M University at Qatar, PO Box 23874, Doha, Qatar. Electronic address: tingwen.huang@qatar.tamu.edu.
authors.forename: Tingwen
authors.initials: T
authors.lastname: Huang
authors.affiliation: School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China. Electronic address: derong@ustb.edu.cn.
authors.forename: Derong
authors.initials: D
authors.lastname: Liu
mesh_terms: D000465:Algorithms; D003198:Computer Simulation; D000069550:Machine Learning; D008962:Models, Theoretical; D016571:Neural Networks, Computer; D017711:Nonlinear Dynamics; D011340:Problem Solving; D012054:Reinforcement, Psychology
keywords: Constrained optimal control; Data-based; Hamilton–Jacobi–Bellman equation; Off-policy reinforcement learning; The method of weighted residuals