Manipulation action tree bank: A knowledge resource for humanoids

Yezhou Yang, Anupam Guha, Cornelia Fermüller, Yiannis Aloimonos

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Citations (Scopus)

Abstract

Our premise is that actions of manipulation are represented at multiple levels of abstraction. At the high level a grammatical structure represents symbolic information (objects, actions, tools, body parts) and their interaction in a temporal sequence, and at lower levels the symbolic quantities are grounded in perception. In this paper we create symbolic high-level representations in the form of manipulation action tree banks, which are parsed from annotated action corpora. A context free grammar provides the grammatical description for the creation of the semantic trees. Experiments conducted on the tree banks show that they allow to 1) generate so-called visual semantic graphs (VSGs), 2) compare the semantic distance between steps of activities and 3) discover the underlying semantic space of an activity. We believe that tree banks are an effective and practical way to organize semantic structures of manipulation actions for humanoids applications. They could be used as basis for 1) automatic manipulation action understanding and execution and 2) reasoning and prediction during both observation and execution. The knowledge resource follows the widely used Penn Tree Bank format.

Original languageEnglish (US)
Title of host publication2014 IEEE-RAS International Conference on Humanoid Robots, Humanoids 2014
PublisherIEEE Computer Society
Pages987-992
Number of pages6
Volume2015-February
ISBN (Electronic)9781479971749
DOIs
StatePublished - Feb 12 2015
Externally publishedYes
Event2014 14th IEEE-RAS International Conference on Humanoid Robots, Humanoids 2014 - Madrid, Spain
Duration: Nov 18 2014Nov 20 2014

Other

Other2014 14th IEEE-RAS International Conference on Humanoid Robots, Humanoids 2014
CountrySpain
CityMadrid
Period11/18/1411/20/14

Fingerprint

Semantics
Context free grammars
Experiments

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Vision and Pattern Recognition
  • Hardware and Architecture
  • Human-Computer Interaction
  • Electrical and Electronic Engineering

Cite this

Yang, Y., Guha, A., Fermüller, C., & Aloimonos, Y. (2015). Manipulation action tree bank: A knowledge resource for humanoids. In 2014 IEEE-RAS International Conference on Humanoid Robots, Humanoids 2014 (Vol. 2015-February, pp. 987-992). [7041483] IEEE Computer Society. https://doi.org/10.1109/HUMANOIDS.2014.7041483

Manipulation action tree bank : A knowledge resource for humanoids. / Yang, Yezhou; Guha, Anupam; Fermüller, Cornelia; Aloimonos, Yiannis.

2014 IEEE-RAS International Conference on Humanoid Robots, Humanoids 2014. Vol. 2015-February IEEE Computer Society, 2015. p. 987-992 7041483.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Yang, Y, Guha, A, Fermüller, C & Aloimonos, Y 2015, Manipulation action tree bank: A knowledge resource for humanoids. in 2014 IEEE-RAS International Conference on Humanoid Robots, Humanoids 2014. vol. 2015-February, 7041483, IEEE Computer Society, pp. 987-992, 2014 14th IEEE-RAS International Conference on Humanoid Robots, Humanoids 2014, Madrid, Spain, 11/18/14. https://doi.org/10.1109/HUMANOIDS.2014.7041483
Yang Y, Guha A, Fermüller C, Aloimonos Y. Manipulation action tree bank: A knowledge resource for humanoids. In 2014 IEEE-RAS International Conference on Humanoid Robots, Humanoids 2014. Vol. 2015-February. IEEE Computer Society. 2015. p. 987-992. 7041483 https://doi.org/10.1109/HUMANOIDS.2014.7041483
Yang, Yezhou ; Guha, Anupam ; Fermüller, Cornelia ; Aloimonos, Yiannis. / Manipulation action tree bank : A knowledge resource for humanoids. 2014 IEEE-RAS International Conference on Humanoid Robots, Humanoids 2014. Vol. 2015-February IEEE Computer Society, 2015. pp. 987-992
@inproceedings{471969fc5ccf4b178f953226bc3dafde,
title = "Manipulation action tree bank: A knowledge resource for humanoids",
abstract = "Our premise is that actions of manipulation are represented at multiple levels of abstraction. At the high level a grammatical structure represents symbolic information (objects, actions, tools, body parts) and their interaction in a temporal sequence, and at lower levels the symbolic quantities are grounded in perception. In this paper we create symbolic high-level representations in the form of manipulation action tree banks, which are parsed from annotated action corpora. A context free grammar provides the grammatical description for the creation of the semantic trees. Experiments conducted on the tree banks show that they allow to 1) generate so-called visual semantic graphs (VSGs), 2) compare the semantic distance between steps of activities and 3) discover the underlying semantic space of an activity. We believe that tree banks are an effective and practical way to organize semantic structures of manipulation actions for humanoids applications. They could be used as basis for 1) automatic manipulation action understanding and execution and 2) reasoning and prediction during both observation and execution. The knowledge resource follows the widely used Penn Tree Bank format.",
author = "Yezhou Yang and Anupam Guha and Cornelia Ferm{\"u}ller and Yiannis Aloimonos",
year = "2015",
month = "2",
day = "12",
doi = "10.1109/HUMANOIDS.2014.7041483",
language = "English (US)",
volume = "2015-February",
pages = "987--992",
booktitle = "2014 IEEE-RAS International Conference on Humanoid Robots, Humanoids 2014",
publisher = "IEEE Computer Society",
address = "United States",

}

TY - GEN

T1 - Manipulation action tree bank

T2 - A knowledge resource for humanoids

AU - Yang, Yezhou

AU - Guha, Anupam

AU - Fermüller, Cornelia

AU - Aloimonos, Yiannis

PY - 2015/2/12

Y1 - 2015/2/12

N2 - Our premise is that actions of manipulation are represented at multiple levels of abstraction. At the high level a grammatical structure represents symbolic information (objects, actions, tools, body parts) and their interaction in a temporal sequence, and at lower levels the symbolic quantities are grounded in perception. In this paper we create symbolic high-level representations in the form of manipulation action tree banks, which are parsed from annotated action corpora. A context free grammar provides the grammatical description for the creation of the semantic trees. Experiments conducted on the tree banks show that they allow to 1) generate so-called visual semantic graphs (VSGs), 2) compare the semantic distance between steps of activities and 3) discover the underlying semantic space of an activity. We believe that tree banks are an effective and practical way to organize semantic structures of manipulation actions for humanoids applications. They could be used as basis for 1) automatic manipulation action understanding and execution and 2) reasoning and prediction during both observation and execution. The knowledge resource follows the widely used Penn Tree Bank format.

AB - Our premise is that actions of manipulation are represented at multiple levels of abstraction. At the high level a grammatical structure represents symbolic information (objects, actions, tools, body parts) and their interaction in a temporal sequence, and at lower levels the symbolic quantities are grounded in perception. In this paper we create symbolic high-level representations in the form of manipulation action tree banks, which are parsed from annotated action corpora. A context free grammar provides the grammatical description for the creation of the semantic trees. Experiments conducted on the tree banks show that they allow to 1) generate so-called visual semantic graphs (VSGs), 2) compare the semantic distance between steps of activities and 3) discover the underlying semantic space of an activity. We believe that tree banks are an effective and practical way to organize semantic structures of manipulation actions for humanoids applications. They could be used as basis for 1) automatic manipulation action understanding and execution and 2) reasoning and prediction during both observation and execution. The knowledge resource follows the widely used Penn Tree Bank format.

UR - http://www.scopus.com/inward/record.url?scp=84945184673&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84945184673&partnerID=8YFLogxK

U2 - 10.1109/HUMANOIDS.2014.7041483

DO - 10.1109/HUMANOIDS.2014.7041483

M3 - Conference contribution

AN - SCOPUS:84945184673

VL - 2015-February

SP - 987

EP - 992

BT - 2014 IEEE-RAS International Conference on Humanoid Robots, Humanoids 2014

PB - IEEE Computer Society

ER -