TY - GEN
T1 - Markov chain sparsification with independent sets for approximate value iteration
AU - Pavez, Eduardo
AU - Michelusi, Nicolo
AU - Anis, Aamir
AU - Mitra, Urbashi
AU - Ortega, Antonio
N1 - Publisher Copyright:
© 2015 IEEE.
PY - 2016/4/4
Y1 - 2016/4/4
N2 - The ever-increasing size of wireless networks poses a significant computational challenge for policy optimization schemes. In this paper, we propose a technique to reduce the dimensionality of the value iteration problem, and thereby reduce computational complexity, by exploiting certain structural properties of the logical state transition network. Specifically, our method involves approximating the original Markov chain by a simplified one whose state transition graph contains an independent set of a prespecified size, thus resulting in a sparsification of the transition probability matrix. As a result, value iteration needs to be performed only on the vertex cover of the network, from which the value function on the independent set can be obtained in a one-step process via interpolation. The Markov chain approximation process presented in this paper, for a given choice of independent set, involves minimizing matrix distance defined in terms of Frobenius norm or the Kullback-Leibler distance. This minimum distance then helps us to define a cost that can be minimized through an iterative greedy algorithm to obtain an approximately optimal independent set. Our method provides a tradeoff between accuracy and complexity that one can exploit by choosing the size of the independent set. Numerical results show that for a class of collision networks the value function approximation is accurate, even with a large independent set.
AB - The ever-increasing size of wireless networks poses a significant computational challenge for policy optimization schemes. In this paper, we propose a technique to reduce the dimensionality of the value iteration problem, and thereby reduce computational complexity, by exploiting certain structural properties of the logical state transition network. Specifically, our method involves approximating the original Markov chain by a simplified one whose state transition graph contains an independent set of a prespecified size, thus resulting in a sparsification of the transition probability matrix. As a result, value iteration needs to be performed only on the vertex cover of the network, from which the value function on the independent set can be obtained in a one-step process via interpolation. The Markov chain approximation process presented in this paper, for a given choice of independent set, involves minimizing matrix distance defined in terms of Frobenius norm or the Kullback-Leibler distance. This minimum distance then helps us to define a cost that can be minimized through an iterative greedy algorithm to obtain an approximately optimal independent set. Our method provides a tradeoff between accuracy and complexity that one can exploit by choosing the size of the independent set. Numerical results show that for a class of collision networks the value function approximation is accurate, even with a large independent set.
KW - approximate value function
KW - independent set
KW - Markov chain
KW - Markov decision processes
KW - wireless networks
UR - http://www.scopus.com/inward/record.url?scp=84969869878&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84969869878&partnerID=8YFLogxK
U2 - 10.1109/ALLERTON.2015.7447172
DO - 10.1109/ALLERTON.2015.7447172
M3 - Conference contribution
AN - SCOPUS:84969869878
T3 - 2015 53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015
SP - 1399
EP - 1405
BT - 2015 53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015
Y2 - 29 September 2015 through 2 October 2015
ER -