Q-learning and policy iteration algorithms for stochastic shortest path problems

Research output: Contribution to journalArticlepeer-review

16 Scopus citations

Abstract

We consider the stochastic shortest path problem, a classical finite-state Markovian decision problem with a termination state, and we propose new convergent Q-learning algorithms that combine elements of policy iteration and classical Q-learning/value iteration. These algorithms are related to the ones introduced by the authors for discounted problems in Bertsekas and Yu (Math. Oper. Res. 37(1):66-94, 2012). The main difference from the standard policy iteration approach is in the policy evaluation phase: instead of solving a linear system of equations, our algorithm solves an optimal stopping problem inexactly with a finite number of value iterations. The main advantage over the standard Q-learning approach is lower overhead: most iterations do not require a minimization over all controls, in the spirit of modified policy iteration. We prove the convergence of asynchronous deterministic and stochastic lookup table implementations of our method for undiscounted, total cost stochastic shortest path problems. These implementations overcome some of the traditional convergence difficulties of asynchronous modified policy iteration, and provide policy iteration-like alternative Q-learning schemes with as reliable convergence as classical Q-learning. We also discuss methods that use basis function approximations of Q-factors and we give an associated error bound.

Original languageEnglish (US)
Pages (from-to)95-132
Number of pages38
JournalAnnals of Operations Research
Volume208
Issue number1
DOIs
StatePublished - Sep 2013
Externally publishedYes

Keywords

  • Approximate dynamic programming
  • Markov decision processes
  • Policy iteration
  • Q-learning
  • Stochastic approximation
  • Stochastic shortest paths
  • Value iteration

ASJC Scopus subject areas

  • Decision Sciences(all)
  • Management Science and Operations Research

Fingerprint Dive into the research topics of 'Q-learning and policy iteration algorithms for stochastic shortest path problems'. Together they form a unique fingerprint.

Cite this