Learning the semantics of manipulation action

Yezhou Yang, Yiannis Aloimonos, Cornelia Fermuller, Eren Erdal Aksoy

Research output: Chapter in Book/Report/Conference proceedingConference contribution

7 Citations (Scopus)

Abstract

In this paper we present a formal computational framework for modeling manipulation actions. The introduced formalism leads to semantics of manipulation action and has applications to both observing and understanding human manipulation actions as well as executing them with a robotic mechanism (e.g. a humanoid robot). It is based on a Combinatory Categorial Grammar. The goal of the introduced framework is to: (1) represent mani8pulation actions with both syntax and semantic parts, where the semantic part employs-calculus; (2) enable a probabilistic semantic parsing schema to learn the lambda-calculus representation of manipulation action from an annotated action corpus of videos; (3) use (1) and (2) to develop a system that visually observes manipulation actions and understands their meaning while it can reason beyond observations using propositional logic and axiom schemata. The experiments conducted on a public available large manipulation action dataset validate the theoretical framework and our implementation.

Original languageEnglish (US)
Title of host publicationACL-IJCNLP 2015 - 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, Proceedings of the Conference
PublisherAssociation for Computational Linguistics (ACL)
Pages676-686
Number of pages11
Volume1
ISBN (Electronic)9781941643723
StatePublished - 2015
Externally publishedYes
Event53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL-IJCNLP 2015 - Beijing, China
Duration: Jul 26 2015Jul 31 2015

Other

Other53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL-IJCNLP 2015
CountryChina
CityBeijing
Period7/26/157/31/15

Fingerprint

Semantics
Robotics
Robots
Experiments

ASJC Scopus subject areas

  • Artificial Intelligence
  • Software

Cite this

Yang, Y., Aloimonos, Y., Fermuller, C., & Aksoy, E. E. (2015). Learning the semantics of manipulation action. In ACL-IJCNLP 2015 - 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, Proceedings of the Conference (Vol. 1, pp. 676-686). Association for Computational Linguistics (ACL).

Learning the semantics of manipulation action. / Yang, Yezhou; Aloimonos, Yiannis; Fermuller, Cornelia; Aksoy, Eren Erdal.

ACL-IJCNLP 2015 - 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, Proceedings of the Conference. Vol. 1 Association for Computational Linguistics (ACL), 2015. p. 676-686.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Yang, Y, Aloimonos, Y, Fermuller, C & Aksoy, EE 2015, Learning the semantics of manipulation action. in ACL-IJCNLP 2015 - 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, Proceedings of the Conference. vol. 1, Association for Computational Linguistics (ACL), pp. 676-686, 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL-IJCNLP 2015, Beijing, China, 7/26/15.
Yang Y, Aloimonos Y, Fermuller C, Aksoy EE. Learning the semantics of manipulation action. In ACL-IJCNLP 2015 - 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, Proceedings of the Conference. Vol. 1. Association for Computational Linguistics (ACL). 2015. p. 676-686
Yang, Yezhou ; Aloimonos, Yiannis ; Fermuller, Cornelia ; Aksoy, Eren Erdal. / Learning the semantics of manipulation action. ACL-IJCNLP 2015 - 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, Proceedings of the Conference. Vol. 1 Association for Computational Linguistics (ACL), 2015. pp. 676-686
@inproceedings{006c8e8469e3461caf40078da68755fb,
title = "Learning the semantics of manipulation action",
abstract = "In this paper we present a formal computational framework for modeling manipulation actions. The introduced formalism leads to semantics of manipulation action and has applications to both observing and understanding human manipulation actions as well as executing them with a robotic mechanism (e.g. a humanoid robot). It is based on a Combinatory Categorial Grammar. The goal of the introduced framework is to: (1) represent mani8pulation actions with both syntax and semantic parts, where the semantic part employs-calculus; (2) enable a probabilistic semantic parsing schema to learn the lambda-calculus representation of manipulation action from an annotated action corpus of videos; (3) use (1) and (2) to develop a system that visually observes manipulation actions and understands their meaning while it can reason beyond observations using propositional logic and axiom schemata. The experiments conducted on a public available large manipulation action dataset validate the theoretical framework and our implementation.",
author = "Yezhou Yang and Yiannis Aloimonos and Cornelia Fermuller and Aksoy, {Eren Erdal}",
year = "2015",
language = "English (US)",
volume = "1",
pages = "676--686",
booktitle = "ACL-IJCNLP 2015 - 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, Proceedings of the Conference",
publisher = "Association for Computational Linguistics (ACL)",

}

TY - GEN

T1 - Learning the semantics of manipulation action

AU - Yang, Yezhou

AU - Aloimonos, Yiannis

AU - Fermuller, Cornelia

AU - Aksoy, Eren Erdal

PY - 2015

Y1 - 2015

N2 - In this paper we present a formal computational framework for modeling manipulation actions. The introduced formalism leads to semantics of manipulation action and has applications to both observing and understanding human manipulation actions as well as executing them with a robotic mechanism (e.g. a humanoid robot). It is based on a Combinatory Categorial Grammar. The goal of the introduced framework is to: (1) represent mani8pulation actions with both syntax and semantic parts, where the semantic part employs-calculus; (2) enable a probabilistic semantic parsing schema to learn the lambda-calculus representation of manipulation action from an annotated action corpus of videos; (3) use (1) and (2) to develop a system that visually observes manipulation actions and understands their meaning while it can reason beyond observations using propositional logic and axiom schemata. The experiments conducted on a public available large manipulation action dataset validate the theoretical framework and our implementation.

AB - In this paper we present a formal computational framework for modeling manipulation actions. The introduced formalism leads to semantics of manipulation action and has applications to both observing and understanding human manipulation actions as well as executing them with a robotic mechanism (e.g. a humanoid robot). It is based on a Combinatory Categorial Grammar. The goal of the introduced framework is to: (1) represent mani8pulation actions with both syntax and semantic parts, where the semantic part employs-calculus; (2) enable a probabilistic semantic parsing schema to learn the lambda-calculus representation of manipulation action from an annotated action corpus of videos; (3) use (1) and (2) to develop a system that visually observes manipulation actions and understands their meaning while it can reason beyond observations using propositional logic and axiom schemata. The experiments conducted on a public available large manipulation action dataset validate the theoretical framework and our implementation.

UR - http://www.scopus.com/inward/record.url?scp=84943791269&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84943791269&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:84943791269

VL - 1

SP - 676

EP - 686

BT - ACL-IJCNLP 2015 - 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, Proceedings of the Conference

PB - Association for Computational Linguistics (ACL)

ER -