Offline policy iteration based reinforcement learning controller for online robotic knee prosthesis parameter tuning

Minhan Li, Xiang Gao, Yue Wen, Jennie Si, He Helen Huang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

This paper aims to develop an optimal controller that can automatically provide personalized control of robotic knee prosthesis in order to best support gait of individual prosthesis wearers. We introduced a new reinforcement learning (RL) controller for this purpose based on the promising ability of RL controllers to solve optimal control problems through interactions with the environment without requiring an explicit system model. However, collecting data from a human-prosthesis system is expensive and thus the design of a RL controller has to take into account data and time efficiency. We therefore propose an offline policy iteration based reinforcement learning approach. Our solution is built on the finite state machine (FSM) impedance control framework, which is the most used prosthesis control method in commercial and prototypic robotic prosthesis. Under such a framework, we designed an approximate policy iteration algorithm to devise impedance parameter update rules for 12 prosthesis control parameters in order to meet individual users' needs. The goal of the reinforcement learning-based control was to reproduce near-normal knee kinematics during gait. We tested the RL controller obtained from offline learning in real time experiment involving the same able-bodied human subject wearing a robotic lower limb prosthesis. Our results showed that the RL control resulted in good convergent behavior in kinematic states, and the offline learning control policy successfully adjusted the prosthesis control parameters to produce near-normal knee kinematics in 10 updates of the impedance control parameters.

Original languageEnglish (US)
Title of host publication2019 International Conference on Robotics and Automation, ICRA 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages2831-2837
Number of pages7
ISBN (Electronic)9781538660263
DOIs
StatePublished - May 1 2019
Event2019 International Conference on Robotics and Automation, ICRA 2019 - Montreal, Canada
Duration: May 20 2019May 24 2019

Publication series

NameProceedings - IEEE International Conference on Robotics and Automation
Volume2019-May
ISSN (Print)1050-4729

Conference

Conference2019 International Conference on Robotics and Automation, ICRA 2019
CountryCanada
CityMontreal
Period5/20/195/24/19

Fingerprint

Knee prostheses
Reinforcement learning
Robotics
Tuning
Controllers
Kinematics
Finite automata
Prostheses and Implants

ASJC Scopus subject areas

  • Software
  • Control and Systems Engineering
  • Artificial Intelligence
  • Electrical and Electronic Engineering

Cite this

Li, M., Gao, X., Wen, Y., Si, J., & Huang, H. H. (2019). Offline policy iteration based reinforcement learning controller for online robotic knee prosthesis parameter tuning. In 2019 International Conference on Robotics and Automation, ICRA 2019 (pp. 2831-2837). [8794212] (Proceedings - IEEE International Conference on Robotics and Automation; Vol. 2019-May). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICRA.2019.8794212

Offline policy iteration based reinforcement learning controller for online robotic knee prosthesis parameter tuning. / Li, Minhan; Gao, Xiang; Wen, Yue; Si, Jennie; Huang, He Helen.

2019 International Conference on Robotics and Automation, ICRA 2019. Institute of Electrical and Electronics Engineers Inc., 2019. p. 2831-2837 8794212 (Proceedings - IEEE International Conference on Robotics and Automation; Vol. 2019-May).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Li, M, Gao, X, Wen, Y, Si, J & Huang, HH 2019, Offline policy iteration based reinforcement learning controller for online robotic knee prosthesis parameter tuning. in 2019 International Conference on Robotics and Automation, ICRA 2019., 8794212, Proceedings - IEEE International Conference on Robotics and Automation, vol. 2019-May, Institute of Electrical and Electronics Engineers Inc., pp. 2831-2837, 2019 International Conference on Robotics and Automation, ICRA 2019, Montreal, Canada, 5/20/19. https://doi.org/10.1109/ICRA.2019.8794212
Li M, Gao X, Wen Y, Si J, Huang HH. Offline policy iteration based reinforcement learning controller for online robotic knee prosthesis parameter tuning. In 2019 International Conference on Robotics and Automation, ICRA 2019. Institute of Electrical and Electronics Engineers Inc. 2019. p. 2831-2837. 8794212. (Proceedings - IEEE International Conference on Robotics and Automation). https://doi.org/10.1109/ICRA.2019.8794212
Li, Minhan ; Gao, Xiang ; Wen, Yue ; Si, Jennie ; Huang, He Helen. / Offline policy iteration based reinforcement learning controller for online robotic knee prosthesis parameter tuning. 2019 International Conference on Robotics and Automation, ICRA 2019. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 2831-2837 (Proceedings - IEEE International Conference on Robotics and Automation).
@inproceedings{b5e37afd373b4d41866fc894f416e016,
title = "Offline policy iteration based reinforcement learning controller for online robotic knee prosthesis parameter tuning",
abstract = "This paper aims to develop an optimal controller that can automatically provide personalized control of robotic knee prosthesis in order to best support gait of individual prosthesis wearers. We introduced a new reinforcement learning (RL) controller for this purpose based on the promising ability of RL controllers to solve optimal control problems through interactions with the environment without requiring an explicit system model. However, collecting data from a human-prosthesis system is expensive and thus the design of a RL controller has to take into account data and time efficiency. We therefore propose an offline policy iteration based reinforcement learning approach. Our solution is built on the finite state machine (FSM) impedance control framework, which is the most used prosthesis control method in commercial and prototypic robotic prosthesis. Under such a framework, we designed an approximate policy iteration algorithm to devise impedance parameter update rules for 12 prosthesis control parameters in order to meet individual users' needs. The goal of the reinforcement learning-based control was to reproduce near-normal knee kinematics during gait. We tested the RL controller obtained from offline learning in real time experiment involving the same able-bodied human subject wearing a robotic lower limb prosthesis. Our results showed that the RL control resulted in good convergent behavior in kinematic states, and the offline learning control policy successfully adjusted the prosthesis control parameters to produce near-normal knee kinematics in 10 updates of the impedance control parameters.",
author = "Minhan Li and Xiang Gao and Yue Wen and Jennie Si and Huang, {He Helen}",
year = "2019",
month = "5",
day = "1",
doi = "10.1109/ICRA.2019.8794212",
language = "English (US)",
series = "Proceedings - IEEE International Conference on Robotics and Automation",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "2831--2837",
booktitle = "2019 International Conference on Robotics and Automation, ICRA 2019",

}

TY - GEN

T1 - Offline policy iteration based reinforcement learning controller for online robotic knee prosthesis parameter tuning

AU - Li, Minhan

AU - Gao, Xiang

AU - Wen, Yue

AU - Si, Jennie

AU - Huang, He Helen

PY - 2019/5/1

Y1 - 2019/5/1

N2 - This paper aims to develop an optimal controller that can automatically provide personalized control of robotic knee prosthesis in order to best support gait of individual prosthesis wearers. We introduced a new reinforcement learning (RL) controller for this purpose based on the promising ability of RL controllers to solve optimal control problems through interactions with the environment without requiring an explicit system model. However, collecting data from a human-prosthesis system is expensive and thus the design of a RL controller has to take into account data and time efficiency. We therefore propose an offline policy iteration based reinforcement learning approach. Our solution is built on the finite state machine (FSM) impedance control framework, which is the most used prosthesis control method in commercial and prototypic robotic prosthesis. Under such a framework, we designed an approximate policy iteration algorithm to devise impedance parameter update rules for 12 prosthesis control parameters in order to meet individual users' needs. The goal of the reinforcement learning-based control was to reproduce near-normal knee kinematics during gait. We tested the RL controller obtained from offline learning in real time experiment involving the same able-bodied human subject wearing a robotic lower limb prosthesis. Our results showed that the RL control resulted in good convergent behavior in kinematic states, and the offline learning control policy successfully adjusted the prosthesis control parameters to produce near-normal knee kinematics in 10 updates of the impedance control parameters.

AB - This paper aims to develop an optimal controller that can automatically provide personalized control of robotic knee prosthesis in order to best support gait of individual prosthesis wearers. We introduced a new reinforcement learning (RL) controller for this purpose based on the promising ability of RL controllers to solve optimal control problems through interactions with the environment without requiring an explicit system model. However, collecting data from a human-prosthesis system is expensive and thus the design of a RL controller has to take into account data and time efficiency. We therefore propose an offline policy iteration based reinforcement learning approach. Our solution is built on the finite state machine (FSM) impedance control framework, which is the most used prosthesis control method in commercial and prototypic robotic prosthesis. Under such a framework, we designed an approximate policy iteration algorithm to devise impedance parameter update rules for 12 prosthesis control parameters in order to meet individual users' needs. The goal of the reinforcement learning-based control was to reproduce near-normal knee kinematics during gait. We tested the RL controller obtained from offline learning in real time experiment involving the same able-bodied human subject wearing a robotic lower limb prosthesis. Our results showed that the RL control resulted in good convergent behavior in kinematic states, and the offline learning control policy successfully adjusted the prosthesis control parameters to produce near-normal knee kinematics in 10 updates of the impedance control parameters.

UR - http://www.scopus.com/inward/record.url?scp=85071426593&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85071426593&partnerID=8YFLogxK

U2 - 10.1109/ICRA.2019.8794212

DO - 10.1109/ICRA.2019.8794212

M3 - Conference contribution

AN - SCOPUS:85071426593

T3 - Proceedings - IEEE International Conference on Robotics and Automation

SP - 2831

EP - 2837

BT - 2019 International Conference on Robotics and Automation, ICRA 2019

PB - Institute of Electrical and Electronics Engineers Inc.

ER -