Markov chain sparsification with independent sets for approximate value iteration

Eduardo Pavez, Nicolo Michelusi, Aamir Anis, Urbashi Mitra, Antonio Ortega

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The ever-increasing size of wireless networks poses a significant computational challenge for policy optimization schemes. In this paper, we propose a technique to reduce the dimensionality of the value iteration problem, and thereby reduce computational complexity, by exploiting certain structural properties of the logical state transition network. Specifically, our method involves approximating the original Markov chain by a simplified one whose state transition graph contains an independent set of a prespecified size, thus resulting in a sparsification of the transition probability matrix. As a result, value iteration needs to be performed only on the vertex cover of the network, from which the value function on the independent set can be obtained in a one-step process via interpolation. The Markov chain approximation process presented in this paper, for a given choice of independent set, involves minimizing matrix distance defined in terms of Frobenius norm or the Kullback-Leibler distance. This minimum distance then helps us to define a cost that can be minimized through an iterative greedy algorithm to obtain an approximately optimal independent set. Our method provides a tradeoff between accuracy and complexity that one can exploit by choosing the size of the independent set. Numerical results show that for a class of collision networks the value function approximation is accurate, even with a large independent set.

Original languageEnglish (US)
Title of host publication2015 53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1399-1405
Number of pages7
ISBN (Electronic)9781509018239
DOIs
StatePublished - Apr 4 2016
Externally publishedYes
Event53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015 - Monticello, United States
Duration: Sep 29 2015Oct 2 2015

Publication series

Name2015 53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015

Other

Other53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015
CountryUnited States
CityMonticello
Period9/29/1510/2/15

Keywords

  • approximate value function
  • independent set
  • Markov chain
  • Markov decision processes
  • wireless networks

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications
  • Control and Systems Engineering

Fingerprint Dive into the research topics of 'Markov chain sparsification with independent sets for approximate value iteration'. Together they form a unique fingerprint.

Cite this