Analyzing and Enhancing Direct NDP Designs Using a Control-Theoretic Approach

Direct NDP is in the family of approximate dynamic programming designs aiming at using learning and approximation methods to solve dynamic optimization problems formulated in dynamic programming, and to overcome the curse of dimensionality. Due to the statistical learning nature of the approaches, researchers usually make use of statistical measures to evaluate the design performance of the learning system such as the learning speed and the variation from one learning experience to the other. However, there are no systematic studies to date that address closed loop system performance from an input-output functional perspective. This paper analyzes direct NDP designs using classic control-theoretic sensitivity arguments. By using the benchmark cart-pole problem, it is shown that direct NDP uses an LQR with desired closed-loop properties as a learning guide, it is more likely for direct NDP to generate better designs than a direct NDP learning from scratch. Although the approach and results are illustrated using a simple nonlinear cart-pole system, it is clear that they are readily extended to more complex dynamical systems.