TY - GEN
T1 - Offline policy iteration based reinforcement learning controller for online robotic knee prosthesis parameter tuning
AU - Li, Minhan
AU - Gao, Xiang
AU - Wen, Yue
AU - Si, Jennie
AU - Huang, He Helen
N1 - Funding Information:
∗This work was partly supported by National Science Foundation #1563454, #1563921, #1808752 and #1808898. (Minhan Li and Xiang Gao are co-first authors. Corresponding authors: He (Helen) Huang; Jennie Si.) M. Li, Y. Wen, and H. Huang are with the NCSU/UNC Department of Biomedical Engineering, NC State University, Raleigh, NC, 27695-7115; University of North Carolina at Chapel Hill, Chapel Hill, NC 27599 USA.
Publisher Copyright:
© 2019 IEEE.
PY - 2019/5
Y1 - 2019/5
N2 - This paper aims to develop an optimal controller that can automatically provide personalized control of robotic knee prosthesis in order to best support gait of individual prosthesis wearers. We introduced a new reinforcement learning (RL) controller for this purpose based on the promising ability of RL controllers to solve optimal control problems through interactions with the environment without requiring an explicit system model. However, collecting data from a human-prosthesis system is expensive and thus the design of a RL controller has to take into account data and time efficiency. We therefore propose an offline policy iteration based reinforcement learning approach. Our solution is built on the finite state machine (FSM) impedance control framework, which is the most used prosthesis control method in commercial and prototypic robotic prosthesis. Under such a framework, we designed an approximate policy iteration algorithm to devise impedance parameter update rules for 12 prosthesis control parameters in order to meet individual users' needs. The goal of the reinforcement learning-based control was to reproduce near-normal knee kinematics during gait. We tested the RL controller obtained from offline learning in real time experiment involving the same able-bodied human subject wearing a robotic lower limb prosthesis. Our results showed that the RL control resulted in good convergent behavior in kinematic states, and the offline learning control policy successfully adjusted the prosthesis control parameters to produce near-normal knee kinematics in 10 updates of the impedance control parameters.
AB - This paper aims to develop an optimal controller that can automatically provide personalized control of robotic knee prosthesis in order to best support gait of individual prosthesis wearers. We introduced a new reinforcement learning (RL) controller for this purpose based on the promising ability of RL controllers to solve optimal control problems through interactions with the environment without requiring an explicit system model. However, collecting data from a human-prosthesis system is expensive and thus the design of a RL controller has to take into account data and time efficiency. We therefore propose an offline policy iteration based reinforcement learning approach. Our solution is built on the finite state machine (FSM) impedance control framework, which is the most used prosthesis control method in commercial and prototypic robotic prosthesis. Under such a framework, we designed an approximate policy iteration algorithm to devise impedance parameter update rules for 12 prosthesis control parameters in order to meet individual users' needs. The goal of the reinforcement learning-based control was to reproduce near-normal knee kinematics during gait. We tested the RL controller obtained from offline learning in real time experiment involving the same able-bodied human subject wearing a robotic lower limb prosthesis. Our results showed that the RL control resulted in good convergent behavior in kinematic states, and the offline learning control policy successfully adjusted the prosthesis control parameters to produce near-normal knee kinematics in 10 updates of the impedance control parameters.
UR - http://www.scopus.com/inward/record.url?scp=85071426593&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85071426593&partnerID=8YFLogxK
U2 - 10.1109/ICRA.2019.8794212
DO - 10.1109/ICRA.2019.8794212
M3 - Conference contribution
AN - SCOPUS:85071426593
T3 - Proceedings - IEEE International Conference on Robotics and Automation
SP - 2831
EP - 2837
BT - 2019 International Conference on Robotics and Automation, ICRA 2019
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 2019 International Conference on Robotics and Automation, ICRA 2019
Y2 - 20 May 2019 through 24 May 2019
ER -