A performance gradient perspective on approximate dynamic programming and its application to partially observable markov decision processes

James Dankert; Lei Yang; Jennie Si

doi:10.1109/ISIC.2006.285595

A performance gradient perspective on approximate dynamic programming and its application to partially observable markov decision processes

James Dankert, Lei Yang, Jennie Si

IAFSE-ECEE: Electrical Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

8 Scopus citations

Abstract

This paper shows an approach to integrating common approximate dynamic programming (ADP) algorithms into a theoretical framework to address both analytical characteristics and algorithmic features. Several important insights are gained from this analysis, including new approaches to the creation of algorithms. Built on this paradigm, ADP learning algorithms are further developed to address a broader class of problems: optimization with partial observability. This framework is based on an average cost formulation which makes use of the concepts of differential costs and performance gradients to describe learning and optimization algorithms. Numerical simulations are conducted including a queueing problem and a maze problem to illustrate and verify features of the proposed algorithms. Pathways for applying this analysis to adaptive critics are also shown.

Original language	English (US)
Title of host publication	Proceedings of the 2006 IEEE International Symposium on Intelligent Control, ISIC
Pages	458-463
Number of pages	6
DOIs	https://doi.org/10.1109/ISIC.2006.285595
State	Published - Dec 1 2006
Event	Joint 2006 IEEE Conference on Control Applications (CCA), Computer-Aided Control Systems Design Symposium (CACSD) and International Symposium on Intelligent Control (ISIC) - Munich, Germany Duration: Oct 4 2006 → Oct 6 2006

Publication series

Name	IEEE International Symposium on Intelligent Control - Proceedings

Other

Other	Joint 2006 IEEE Conference on Control Applications (CCA), Computer-Aided Control Systems Design Symposium (CACSD) and International Symposium on Intelligent Control (ISIC)
Country	Germany
City	Munich
Period	10/4/06 → 10/6/06

ASJC Scopus subject areas

Control and Systems Engineering
Modeling and Simulation
Computer Science Applications
Electrical and Electronic Engineering

Access to Document

10.1109/ISIC.2006.285595

Fingerprint Dive into the research topics of 'A performance gradient perspective on approximate dynamic programming and its application to partially observable markov decision processes'. Together they form a unique fingerprint.

Cite this

Dankert, J., Yang, L., & Si, J. (2006). A performance gradient perspective on approximate dynamic programming and its application to partially observable markov decision processes. In Proceedings of the 2006 IEEE International Symposium on Intelligent Control, ISIC (pp. 458-463). [4064920] (IEEE International Symposium on Intelligent Control - Proceedings). https://doi.org/10.1109/ISIC.2006.285595

A performance gradient perspective on approximate dynamic programming and its application to partially observable markov decision processes. / Dankert, James; Yang, Lei; Si, Jennie.

Proceedings of the 2006 IEEE International Symposium on Intelligent Control, ISIC. 2006. p. 458-463 4064920 (IEEE International Symposium on Intelligent Control - Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Dankert, J, Yang, L & Si, J 2006, A performance gradient perspective on approximate dynamic programming and its application to partially observable markov decision processes. in Proceedings of the 2006 IEEE International Symposium on Intelligent Control, ISIC., 4064920, IEEE International Symposium on Intelligent Control - Proceedings, pp. 458-463, Joint 2006 IEEE Conference on Control Applications (CCA), Computer-Aided Control Systems Design Symposium (CACSD) and International Symposium on Intelligent Control (ISIC), Munich, Germany, 10/4/06. https://doi.org/10.1109/ISIC.2006.285595

Dankert J, Yang L, Si J. A performance gradient perspective on approximate dynamic programming and its application to partially observable markov decision processes. In Proceedings of the 2006 IEEE International Symposium on Intelligent Control, ISIC. 2006. p. 458-463. 4064920. (IEEE International Symposium on Intelligent Control - Proceedings). https://doi.org/10.1109/ISIC.2006.285595

@inproceedings{122ef57fe92940f3ba9375e57e670239,

title = "A performance gradient perspective on approximate dynamic programming and its application to partially observable markov decision processes",

abstract = "This paper shows an approach to integrating common approximate dynamic programming (ADP) algorithms into a theoretical framework to address both analytical characteristics and algorithmic features. Several important insights are gained from this analysis, including new approaches to the creation of algorithms. Built on this paradigm, ADP learning algorithms are further developed to address a broader class of problems: optimization with partial observability. This framework is based on an average cost formulation which makes use of the concepts of differential costs and performance gradients to describe learning and optimization algorithms. Numerical simulations are conducted including a queueing problem and a maze problem to illustrate and verify features of the proposed algorithms. Pathways for applying this analysis to adaptive critics are also shown.",

author = "James Dankert and Lei Yang and Jennie Si",

year = "2006",

month = dec,

day = "1",

doi = "10.1109/ISIC.2006.285595",

language = "English (US)",

isbn = "0780397983",

series = "IEEE International Symposium on Intelligent Control - Proceedings",

pages = "458--463",

booktitle = "Proceedings of the 2006 IEEE International Symposium on Intelligent Control, ISIC",

note = "Joint 2006 IEEE Conference on Control Applications (CCA), Computer-Aided Control Systems Design Symposium (CACSD) and International Symposium on Intelligent Control (ISIC) ; Conference date: 04-10-2006 Through 06-10-2006",

}

TY - GEN

T1 - A performance gradient perspective on approximate dynamic programming and its application to partially observable markov decision processes

AU - Dankert, James

AU - Yang, Lei

AU - Si, Jennie

PY - 2006/12/1

Y1 - 2006/12/1

N2 - This paper shows an approach to integrating common approximate dynamic programming (ADP) algorithms into a theoretical framework to address both analytical characteristics and algorithmic features. Several important insights are gained from this analysis, including new approaches to the creation of algorithms. Built on this paradigm, ADP learning algorithms are further developed to address a broader class of problems: optimization with partial observability. This framework is based on an average cost formulation which makes use of the concepts of differential costs and performance gradients to describe learning and optimization algorithms. Numerical simulations are conducted including a queueing problem and a maze problem to illustrate and verify features of the proposed algorithms. Pathways for applying this analysis to adaptive critics are also shown.

AB - This paper shows an approach to integrating common approximate dynamic programming (ADP) algorithms into a theoretical framework to address both analytical characteristics and algorithmic features. Several important insights are gained from this analysis, including new approaches to the creation of algorithms. Built on this paradigm, ADP learning algorithms are further developed to address a broader class of problems: optimization with partial observability. This framework is based on an average cost formulation which makes use of the concepts of differential costs and performance gradients to describe learning and optimization algorithms. Numerical simulations are conducted including a queueing problem and a maze problem to illustrate and verify features of the proposed algorithms. Pathways for applying this analysis to adaptive critics are also shown.

UR - http://www.scopus.com/inward/record.url?scp=61849156138&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=61849156138&partnerID=8YFLogxK

U2 - 10.1109/ISIC.2006.285595

DO - 10.1109/ISIC.2006.285595

M3 - Conference contribution

AN - SCOPUS:61849156138

SN - 0780397983

SN - 9780780397989

T3 - IEEE International Symposium on Intelligent Control - Proceedings

SP - 458

EP - 463

BT - Proceedings of the 2006 IEEE International Symposium on Intelligent Control, ISIC

T2 - Joint 2006 IEEE Conference on Control Applications (CCA), Computer-Aided Control Systems Design Symposium (CACSD) and International Symposium on Intelligent Control (ISIC)

Y2 - 4 October 2006 through 6 October 2006

ER -