Improved Exploration through Latent Trajectory Optimization in Deep Deterministic Policy Gradient

Kevin Sebastian Luck, Mel Vecerik, Simon Stepputtis, Heni Ben Amor, Jonathan Scholz

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Scopus citations

Abstract

Model-free reinforcement learning algorithms such as Deep Deterministic Policy Gradient (DDPG) often require additional exploration strategies, especially if the actor is of deterministic nature. This work evaluates the use of model-based trajectory optimization methods used for exploration in Deep Deterministic Policy Gradient when trained on a latent image embedding. In addition, an extension of DDPG is derived using a value function as critic, making use of a learned deep dynamics model to compute the policy gradient. This approach leads to a symbiotic relationship between the deep reinforcement learning algorithm and the latent trajectory optimizer. The trajectory optimizer benefits from the critic learned by the RL algorithm and the latter from the enhanced exploration generated by the planner. The developed methods are evaluated on two continuous control tasks, one in simulation and one in the real world. In particular, a Baxter robot is trained to perform an insertion task, while only receiving sparse rewards and images as observations from the environment.

Original languageEnglish (US)
Title of host publication2019 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages3704-3711
Number of pages8
ISBN (Electronic)9781728140049
DOIs
StatePublished - Nov 2019
Event2019 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2019 - Macau, China
Duration: Nov 3 2019Nov 8 2019

Publication series

NameIEEE International Conference on Intelligent Robots and Systems
ISSN (Print)2153-0858
ISSN (Electronic)2153-0866

Conference

Conference2019 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2019
CountryChina
CityMacau
Period11/3/1911/8/19

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Software
  • Computer Vision and Pattern Recognition
  • Computer Science Applications

Fingerprint Dive into the research topics of 'Improved Exploration through Latent Trajectory Optimization in Deep Deterministic Policy Gradient'. Together they form a unique fingerprint.

Cite this