Partially observable Markov decision processes and performance sensitivity analysis.
The sensitivity-based optimization of Markov systems has become an increasingly important area. From the perspective of performance sensitivity analysis, policy-iteration algorithms and gradient estimation methods can be directly obtained for Markov decision processes (MDPs). In this correspondence, the sensitivity-based optimization is extended to average reward partially observable MDPs (POMDPs). We derive the performance-difference and performance-derivative formulas of POMDPs. On the basis of the performance-derivative formula, we present a new method to estimate the performance gradients. From the performance-difference formula, we obtain a sufficient optimality condition without the discounted reward formulation. We also propose a policy-iteration algorithm to obtain a nearly optimal finite-state-controller policy.
Li
Yanjie
Y
Department of Automation, University of Science and Technology of China, Hefei 230026, China. whylyj@ustc.edu
Yin
Baoqun
B
Xi
Hongsheng
H
D000465:Algorithms; D001185:Artificial Intelligence; D003198:Computer Simulation; D003657:Decision Making; D003661:Decision Support Techniques; D008390:Markov Chains; D015233:Models, Statistical; D010363:Pattern Recognition, Automated / Q000379:methods*; D012680:Sensitivity and Specificity