New value iteration and Q-learning methods for the average cost dynamic programming problem

Research output: Contribution to journalConference articlepeer-review

Abstract

We propose a new value iteration method for the classical average cost Markovian Decision problem, under the assumption that all stationary policies are unichain and furthermore there exists a state that is recurrent under all stationary policies. This method is motivated by a relation between the average cost problem and an associated stochastic shortest path problem. Contrary to the standard relative value iteration, our method involves a weighted sup norm contraction and for this reason it admits a Gauss-Seidel and an asynchronous implementation. Computational tests indicate that the Gauss-Seidel version of the new method substantially outperforms the standard method for difficult problems. The contraction property also makes the method a suitable basis for the development of asynchronous Q-learning methods.

Original languageEnglish (US)
Pages (from-to)2692-2697
Number of pages6
JournalProceedings of the IEEE Conference on Decision and Control
Volume3
StatePublished - 1998
Externally publishedYes
EventProceedings of the 1998 37th IEEE Conference on Decision and Control (CDC) - Tampa, FL, USA
Duration: Dec 16 1998Dec 18 1998

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Modeling and Simulation
  • Control and Optimization

Fingerprint Dive into the research topics of 'New value iteration and Q-learning methods for the average cost dynamic programming problem'. Together they form a unique fingerprint.

Cite this