Approximate robust policy iteration for discounted infinite-horizon Markov decision processes with uncertain stationary parametric transition matrices

Baohua Li, Jennie Si

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We consider Markov decision processes with finite states, finite actions, and discounted infinite-horizon cost in the deterministic policy space. State transition matrices are uncertain but with stationary parameterization. The uncertainty in transition matrices signifies realistic considerations that an accurate system model is not available for the controller design due to limitations in estimation methods and model deficiencies. Based on the quadratic total value function formulation, two approximate robust policy iterations are developed, the performance errors of which are guaranteed to be within an arbitrarily small error bound. The two approximations make use of iterative aggregation and multilayer perceptron, respectively. It is proved that the robust policy iteration based on approximation with iterative aggregation converges surely to a stationary optimal or near-optimal policy, and also that under some conditions the robust policy iteration based on approximation with multilayer perceptron converges in a probability sense to a stationary near-optimal policy. Furthermore, under some assumptions, the stationary solutions are guaranteed to be near-optimal in the deterministic policy space.

Original languageEnglish (US)
Title of host publicationThe 2007 International Joint Conference on Neural Networks, IJCNN 2007 Conference Proceedings
Pages2052-2057
Number of pages6
DOIs
StatePublished - Dec 1 2007
Event2007 International Joint Conference on Neural Networks, IJCNN 2007 - Orlando, FL, United States
Duration: Aug 12 2007Aug 17 2007

Publication series

NameIEEE International Conference on Neural Networks - Conference Proceedings
ISSN (Print)1098-7576

Other

Other2007 International Joint Conference on Neural Networks, IJCNN 2007
CountryUnited States
CityOrlando, FL
Period8/12/078/17/07

ASJC Scopus subject areas

  • Software

Fingerprint Dive into the research topics of 'Approximate robust policy iteration for discounted infinite-horizon Markov decision processes with uncertain stationary parametric transition matrices'. Together they form a unique fingerprint.

Cite this