Temporal Spatial Inverse Semantics for Robots Communicating with Humans

Ze Gong, Yu Zhang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Citations (Scopus)

Abstract

Effective communication between humans often embeds both temporal and spatial context. While spatial context captures the geographic settings of objects in the environment, temporal context describes their changes over time. In this paper, we propose temporal spatial inverse semantics (TeSIS) to extend the inverse semantics approach to also consider the temporal context for robots communicating with humans. Inverse semantics generates natural language requests while taking into account how well the human listeners would interpret those requests given the current spatial context. Compared to inverse semantics, our approach incorporates also temporal context by referring to spatial context information in the past. To achieve this, we extend the sentence structure in inverse semantics to generate sentences that can refer to not only the current but also previous states of the environment. A new metric based on the extended sentence structure is developed by breaking a single sentence into multiple independent sentences that refer to environment states at different times. Using this approach, we are able to generate sentences such as 'Please pick up the cup beside the oven that was on the dining table'. To evaluate our approach, we randomly generate scenarios in an experimental domain. Each scenario includes the description of the current and several immediate previous states. Natural language sentences are then generated for these scenarios using both inverse semantics that uses only the spatial context and our approach. Amazon MTurk is used to compare the sentences generated and results show that TeSIS achieves better accuracy, sometimes by a significant margin, than the baseline.

Original languageEnglish (US)
Title of host publication2018 IEEE International Conference on Robotics and Automation, ICRA 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages4451-4458
Number of pages8
ISBN (Electronic)9781538630815
DOIs
StatePublished - Sep 10 2018
Event2018 IEEE International Conference on Robotics and Automation, ICRA 2018 - Brisbane, Australia
Duration: May 21 2018May 25 2018

Publication series

NameProceedings - IEEE International Conference on Robotics and Automation
ISSN (Print)1050-4729

Conference

Conference2018 IEEE International Conference on Robotics and Automation, ICRA 2018
CountryAustralia
CityBrisbane
Period5/21/185/25/18

Fingerprint

Intelligent robots
Semantics
Ovens
Communication

ASJC Scopus subject areas

  • Software
  • Control and Systems Engineering
  • Artificial Intelligence
  • Electrical and Electronic Engineering

Cite this

Gong, Z., & Zhang, Y. (2018). Temporal Spatial Inverse Semantics for Robots Communicating with Humans. In 2018 IEEE International Conference on Robotics and Automation, ICRA 2018 (pp. 4451-4458). [8460754] (Proceedings - IEEE International Conference on Robotics and Automation). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICRA.2018.8460754

Temporal Spatial Inverse Semantics for Robots Communicating with Humans. / Gong, Ze; Zhang, Yu.

2018 IEEE International Conference on Robotics and Automation, ICRA 2018. Institute of Electrical and Electronics Engineers Inc., 2018. p. 4451-4458 8460754 (Proceedings - IEEE International Conference on Robotics and Automation).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Gong, Z & Zhang, Y 2018, Temporal Spatial Inverse Semantics for Robots Communicating with Humans. in 2018 IEEE International Conference on Robotics and Automation, ICRA 2018., 8460754, Proceedings - IEEE International Conference on Robotics and Automation, Institute of Electrical and Electronics Engineers Inc., pp. 4451-4458, 2018 IEEE International Conference on Robotics and Automation, ICRA 2018, Brisbane, Australia, 5/21/18. https://doi.org/10.1109/ICRA.2018.8460754
Gong Z, Zhang Y. Temporal Spatial Inverse Semantics for Robots Communicating with Humans. In 2018 IEEE International Conference on Robotics and Automation, ICRA 2018. Institute of Electrical and Electronics Engineers Inc. 2018. p. 4451-4458. 8460754. (Proceedings - IEEE International Conference on Robotics and Automation). https://doi.org/10.1109/ICRA.2018.8460754
Gong, Ze ; Zhang, Yu. / Temporal Spatial Inverse Semantics for Robots Communicating with Humans. 2018 IEEE International Conference on Robotics and Automation, ICRA 2018. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 4451-4458 (Proceedings - IEEE International Conference on Robotics and Automation).
@inproceedings{d02a72413c984653b274c7fe96237623,
title = "Temporal Spatial Inverse Semantics for Robots Communicating with Humans",
abstract = "Effective communication between humans often embeds both temporal and spatial context. While spatial context captures the geographic settings of objects in the environment, temporal context describes their changes over time. In this paper, we propose temporal spatial inverse semantics (TeSIS) to extend the inverse semantics approach to also consider the temporal context for robots communicating with humans. Inverse semantics generates natural language requests while taking into account how well the human listeners would interpret those requests given the current spatial context. Compared to inverse semantics, our approach incorporates also temporal context by referring to spatial context information in the past. To achieve this, we extend the sentence structure in inverse semantics to generate sentences that can refer to not only the current but also previous states of the environment. A new metric based on the extended sentence structure is developed by breaking a single sentence into multiple independent sentences that refer to environment states at different times. Using this approach, we are able to generate sentences such as 'Please pick up the cup beside the oven that was on the dining table'. To evaluate our approach, we randomly generate scenarios in an experimental domain. Each scenario includes the description of the current and several immediate previous states. Natural language sentences are then generated for these scenarios using both inverse semantics that uses only the spatial context and our approach. Amazon MTurk is used to compare the sentences generated and results show that TeSIS achieves better accuracy, sometimes by a significant margin, than the baseline.",
author = "Ze Gong and Yu Zhang",
year = "2018",
month = "9",
day = "10",
doi = "10.1109/ICRA.2018.8460754",
language = "English (US)",
series = "Proceedings - IEEE International Conference on Robotics and Automation",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "4451--4458",
booktitle = "2018 IEEE International Conference on Robotics and Automation, ICRA 2018",

}

TY - GEN

T1 - Temporal Spatial Inverse Semantics for Robots Communicating with Humans

AU - Gong, Ze

AU - Zhang, Yu

PY - 2018/9/10

Y1 - 2018/9/10

N2 - Effective communication between humans often embeds both temporal and spatial context. While spatial context captures the geographic settings of objects in the environment, temporal context describes their changes over time. In this paper, we propose temporal spatial inverse semantics (TeSIS) to extend the inverse semantics approach to also consider the temporal context for robots communicating with humans. Inverse semantics generates natural language requests while taking into account how well the human listeners would interpret those requests given the current spatial context. Compared to inverse semantics, our approach incorporates also temporal context by referring to spatial context information in the past. To achieve this, we extend the sentence structure in inverse semantics to generate sentences that can refer to not only the current but also previous states of the environment. A new metric based on the extended sentence structure is developed by breaking a single sentence into multiple independent sentences that refer to environment states at different times. Using this approach, we are able to generate sentences such as 'Please pick up the cup beside the oven that was on the dining table'. To evaluate our approach, we randomly generate scenarios in an experimental domain. Each scenario includes the description of the current and several immediate previous states. Natural language sentences are then generated for these scenarios using both inverse semantics that uses only the spatial context and our approach. Amazon MTurk is used to compare the sentences generated and results show that TeSIS achieves better accuracy, sometimes by a significant margin, than the baseline.

AB - Effective communication between humans often embeds both temporal and spatial context. While spatial context captures the geographic settings of objects in the environment, temporal context describes their changes over time. In this paper, we propose temporal spatial inverse semantics (TeSIS) to extend the inverse semantics approach to also consider the temporal context for robots communicating with humans. Inverse semantics generates natural language requests while taking into account how well the human listeners would interpret those requests given the current spatial context. Compared to inverse semantics, our approach incorporates also temporal context by referring to spatial context information in the past. To achieve this, we extend the sentence structure in inverse semantics to generate sentences that can refer to not only the current but also previous states of the environment. A new metric based on the extended sentence structure is developed by breaking a single sentence into multiple independent sentences that refer to environment states at different times. Using this approach, we are able to generate sentences such as 'Please pick up the cup beside the oven that was on the dining table'. To evaluate our approach, we randomly generate scenarios in an experimental domain. Each scenario includes the description of the current and several immediate previous states. Natural language sentences are then generated for these scenarios using both inverse semantics that uses only the spatial context and our approach. Amazon MTurk is used to compare the sentences generated and results show that TeSIS achieves better accuracy, sometimes by a significant margin, than the baseline.

UR - http://www.scopus.com/inward/record.url?scp=85058136682&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85058136682&partnerID=8YFLogxK

U2 - 10.1109/ICRA.2018.8460754

DO - 10.1109/ICRA.2018.8460754

M3 - Conference contribution

AN - SCOPUS:85058136682

T3 - Proceedings - IEEE International Conference on Robotics and Automation

SP - 4451

EP - 4458

BT - 2018 IEEE International Conference on Robotics and Automation, ICRA 2018

PB - Institute of Electrical and Electronics Engineers Inc.

ER -