title: A new Q-learning algorithm based on the metropolis criterion.
abstract: The balance between exploration and exploitation is one of the key problems of action selection in Q-learning. Pure exploitation causes the agent to reach the locally optimal policies quickly, whereas excessive exploration degrades the performance of the Q-learning algorithm even if it may accelerate the learning process and allow avoiding the locally optimal policies. In this paper, finding the optimum policy in Q-learning is described as search for the optimum solution in combinatorial optimization. The Metropolis criterion of simulated annealing algorithm is introduced in order to balance exploration and exploitation of Q-learning, and the modified Q-learning algorithm based on this criterion, SA-Q-learning, is presented. Experiments show that SA-Q-learning converges more quickly than Q-learning or Boltzmann exploration, and that the search does not suffer of performance degradation due to excessive exploration.
authors.forename: Maozu
authors.initials: M
authors.lastname: Guo
authors.forename: Yang
authors.initials: Y
authors.lastname: Liu
authors.forename: Jacek
authors.initials: J
authors.lastname: Malec
mesh_terms: D000465:Algorithms; D001185:Artificial Intelligence; D003198:Computer Simulation; D016247:Information Storage and Retrieval / Q000379:methods*; D008962:Models, Theoretical