Neuro-dynamic programming based on self-organized patterns

This paper introduces a real-time learning control mechanism, as a robust and efficient scheme of neuro-dynamic programming. The objective of the learning controller is to optimize a certain performance measure by learning to create appropriate control actions through interacting with the environment. The controller is set out to learn to perform better over time starting with no prior knowledge about the system. The system under consideration does not render a complete system model describing its behaviors either. Instead, real-time sampled measurements are available to the designer. The feedback signal from the environment about the system is less descriptive in the sense that only indicative signals, such as binary-valued signals, are available at the end of a task signifying either a success or a failure. The current study proposes pattern-based neuro-dynamic programming techniques. The state measurements are first analyzed by similarity and organized by proximity. Control actions are then generated in relevance to the state patterns. A critic network serves the purpose of `monitoring' the performance of the controller to achieve a given optimality. In the present paper, we provide detailed implementation, and performance evaluations of this learning controller in a cart-pole balancing problem.