Robotic Knee Parameter Tuning Using Approximate Policy Iteration

Xiang Gao, Yue Wen, Minhan Li, Jennie Si, He (Helen) Huang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

This paper presents an online model-free reinforcement learning based controller realized by approximate dynamic programming for a robotic knee as part of a human-machine system. Traditionally, prosthesis wearers’ gait performance is improved by manually tuning the impedance parameters. In this paper, we show that the parameter tuning problem can be formulated as an optimal control problem and thus solved by dynamic programming. Toward this goal, we constructed an quadratic instantaneous cost, which resulted in a value function that could be approximated by a neural network. The control policy is then solved by the least-squared method iteratively, a framework of which we refer to as approximate policy iteration. We performed extensive simulations based on prosthetic kinetics and human performance data extracted from real human subjects. Our results show that the proposed parameter tuning algorithm can be readily used for adaptive optimal tuning of prosthetic knee control parameters and the tuning process is time and sample efficient.

Original languageEnglish (US)
Title of host publicationCognitive Systems and Signal Processing - 4th International Conference, ICCSIP 2018, Revised Selected Papers
EditorsFuchun Sun, Huaping Liu, Dewen Hu
PublisherSpringer Verlag
Pages554-563
Number of pages10
ISBN (Print)9789811379826
DOIs
StatePublished - Jan 1 2019
Event4th International Conference on Cognitive Systems and Information Processing, ICCSIP 2018 - Beijing, China
Duration: Nov 29 2018Dec 1 2018

Publication series

NameCommunications in Computer and Information Science
Volume1005
ISSN (Print)1865-0929

Conference

Conference4th International Conference on Cognitive Systems and Information Processing, ICCSIP 2018
CountryChina
CityBeijing
Period11/29/1812/1/18

Fingerprint

Policy Iteration
Parameter Tuning
Robotics
Tuning
Approximate Dynamic Programming
Human Performance
Prosthetics
Dynamic programming
Gait
Control Policy
Reinforcement Learning
Value Function
Control Parameter
Impedance
Instantaneous
Dynamic Programming
Optimal Control Problem
Man machine systems
Kinetics
Neural Networks

Keywords

  • Approximate dynamic programming (ADP)
  • Lower limb prosthesis
  • Policy iteration
  • Sample efficient learning

ASJC Scopus subject areas

  • Computer Science(all)
  • Mathematics(all)

Cite this

Gao, X., Wen, Y., Li, M., Si, J., & Huang, H. H. (2019). Robotic Knee Parameter Tuning Using Approximate Policy Iteration. In F. Sun, H. Liu, & D. Hu (Eds.), Cognitive Systems and Signal Processing - 4th International Conference, ICCSIP 2018, Revised Selected Papers (pp. 554-563). (Communications in Computer and Information Science; Vol. 1005). Springer Verlag. https://doi.org/10.1007/978-981-13-7983-3_49

Robotic Knee Parameter Tuning Using Approximate Policy Iteration. / Gao, Xiang; Wen, Yue; Li, Minhan; Si, Jennie; Huang, He (Helen).

Cognitive Systems and Signal Processing - 4th International Conference, ICCSIP 2018, Revised Selected Papers. ed. / Fuchun Sun; Huaping Liu; Dewen Hu. Springer Verlag, 2019. p. 554-563 (Communications in Computer and Information Science; Vol. 1005).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Gao, X, Wen, Y, Li, M, Si, J & Huang, HH 2019, Robotic Knee Parameter Tuning Using Approximate Policy Iteration. in F Sun, H Liu & D Hu (eds), Cognitive Systems and Signal Processing - 4th International Conference, ICCSIP 2018, Revised Selected Papers. Communications in Computer and Information Science, vol. 1005, Springer Verlag, pp. 554-563, 4th International Conference on Cognitive Systems and Information Processing, ICCSIP 2018, Beijing, China, 11/29/18. https://doi.org/10.1007/978-981-13-7983-3_49
Gao X, Wen Y, Li M, Si J, Huang HH. Robotic Knee Parameter Tuning Using Approximate Policy Iteration. In Sun F, Liu H, Hu D, editors, Cognitive Systems and Signal Processing - 4th International Conference, ICCSIP 2018, Revised Selected Papers. Springer Verlag. 2019. p. 554-563. (Communications in Computer and Information Science). https://doi.org/10.1007/978-981-13-7983-3_49
Gao, Xiang ; Wen, Yue ; Li, Minhan ; Si, Jennie ; Huang, He (Helen). / Robotic Knee Parameter Tuning Using Approximate Policy Iteration. Cognitive Systems and Signal Processing - 4th International Conference, ICCSIP 2018, Revised Selected Papers. editor / Fuchun Sun ; Huaping Liu ; Dewen Hu. Springer Verlag, 2019. pp. 554-563 (Communications in Computer and Information Science).
@inproceedings{f108e25462ee42109b3513483671d3cc,
title = "Robotic Knee Parameter Tuning Using Approximate Policy Iteration",
abstract = "This paper presents an online model-free reinforcement learning based controller realized by approximate dynamic programming for a robotic knee as part of a human-machine system. Traditionally, prosthesis wearers’ gait performance is improved by manually tuning the impedance parameters. In this paper, we show that the parameter tuning problem can be formulated as an optimal control problem and thus solved by dynamic programming. Toward this goal, we constructed an quadratic instantaneous cost, which resulted in a value function that could be approximated by a neural network. The control policy is then solved by the least-squared method iteratively, a framework of which we refer to as approximate policy iteration. We performed extensive simulations based on prosthetic kinetics and human performance data extracted from real human subjects. Our results show that the proposed parameter tuning algorithm can be readily used for adaptive optimal tuning of prosthetic knee control parameters and the tuning process is time and sample efficient.",
keywords = "Approximate dynamic programming (ADP), Lower limb prosthesis, Policy iteration, Sample efficient learning",
author = "Xiang Gao and Yue Wen and Minhan Li and Jennie Si and Huang, {He (Helen)}",
year = "2019",
month = "1",
day = "1",
doi = "10.1007/978-981-13-7983-3_49",
language = "English (US)",
isbn = "9789811379826",
series = "Communications in Computer and Information Science",
publisher = "Springer Verlag",
pages = "554--563",
editor = "Fuchun Sun and Huaping Liu and Dewen Hu",
booktitle = "Cognitive Systems and Signal Processing - 4th International Conference, ICCSIP 2018, Revised Selected Papers",

}

TY - GEN

T1 - Robotic Knee Parameter Tuning Using Approximate Policy Iteration

AU - Gao, Xiang

AU - Wen, Yue

AU - Li, Minhan

AU - Si, Jennie

AU - Huang, He (Helen)

PY - 2019/1/1

Y1 - 2019/1/1

N2 - This paper presents an online model-free reinforcement learning based controller realized by approximate dynamic programming for a robotic knee as part of a human-machine system. Traditionally, prosthesis wearers’ gait performance is improved by manually tuning the impedance parameters. In this paper, we show that the parameter tuning problem can be formulated as an optimal control problem and thus solved by dynamic programming. Toward this goal, we constructed an quadratic instantaneous cost, which resulted in a value function that could be approximated by a neural network. The control policy is then solved by the least-squared method iteratively, a framework of which we refer to as approximate policy iteration. We performed extensive simulations based on prosthetic kinetics and human performance data extracted from real human subjects. Our results show that the proposed parameter tuning algorithm can be readily used for adaptive optimal tuning of prosthetic knee control parameters and the tuning process is time and sample efficient.

AB - This paper presents an online model-free reinforcement learning based controller realized by approximate dynamic programming for a robotic knee as part of a human-machine system. Traditionally, prosthesis wearers’ gait performance is improved by manually tuning the impedance parameters. In this paper, we show that the parameter tuning problem can be formulated as an optimal control problem and thus solved by dynamic programming. Toward this goal, we constructed an quadratic instantaneous cost, which resulted in a value function that could be approximated by a neural network. The control policy is then solved by the least-squared method iteratively, a framework of which we refer to as approximate policy iteration. We performed extensive simulations based on prosthetic kinetics and human performance data extracted from real human subjects. Our results show that the proposed parameter tuning algorithm can be readily used for adaptive optimal tuning of prosthetic knee control parameters and the tuning process is time and sample efficient.

KW - Approximate dynamic programming (ADP)

KW - Lower limb prosthesis

KW - Policy iteration

KW - Sample efficient learning

UR - http://www.scopus.com/inward/record.url?scp=85065763048&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85065763048&partnerID=8YFLogxK

U2 - 10.1007/978-981-13-7983-3_49

DO - 10.1007/978-981-13-7983-3_49

M3 - Conference contribution

AN - SCOPUS:85065763048

SN - 9789811379826

T3 - Communications in Computer and Information Science

SP - 554

EP - 563

BT - Cognitive Systems and Signal Processing - 4th International Conference, ICCSIP 2018, Revised Selected Papers

A2 - Sun, Fuchun

A2 - Liu, Huaping

A2 - Hu, Dewen

PB - Springer Verlag

ER -