Towards a Watson that sees: Language-guided action recognition for robots

Ching L. Teo, Yezhou Yang, Hal Daumé, Cornelia Fermuller, Yiannis Aloimonos

Research output: Chapter in Book/Report/Conference proceedingConference contribution

17 Scopus citations

Abstract

For robots of the future to interact seamlessly with humans, they must be able to reason about their surroundings and take actions that are appropriate to the situation. Such reasoning is only possible when the robot has knowledge of how the World functions, which must either be learned or hard-coded. In this paper, we propose an approach that exploits language as an important resource of high-level knowledge that a robot can use, akin to IBM's Watson in Jeopardy!. In particular, we show how language can be leveraged to reduce the ambiguity that arises from recognizing actions involving hand-tools from video data. Starting from the premise that tools and actions are intrinsically linked, with one explaining the existence of the other, we trained a language model over a large corpus of English newswire text so that we can extract this relationship directly. This model is then used as a prior to select the best tool and action that explains the video. We formalize the approach in the context of 1) an unsupervised recognition and 2) a supervised classification scenario by an EM formulation for the former and integrating language features for the latter. Results are validated over a new hand-tool action dataset, and comparisons with state of the art STIP features showed significantly improved results when language is used. In addition, we discuss the implications of these results and how it provides a framework for integrating language into vision on other robotic applications.

Original languageEnglish (US)
Title of host publication2012 IEEE International Conference on Robotics and Automation, ICRA 2012
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages374-381
Number of pages8
ISBN (Print)9781467314039
DOIs
StatePublished - Jan 1 2012
Externally publishedYes
Event 2012 IEEE International Conference on Robotics and Automation, ICRA 2012 - Saint Paul, MN, United States
Duration: May 14 2012May 18 2012

Publication series

NameProceedings - IEEE International Conference on Robotics and Automation
ISSN (Print)1050-4729

Conference

Conference 2012 IEEE International Conference on Robotics and Automation, ICRA 2012
CountryUnited States
CitySaint Paul, MN
Period5/14/125/18/12

ASJC Scopus subject areas

  • Software
  • Control and Systems Engineering
  • Artificial Intelligence
  • Electrical and Electronic Engineering

Fingerprint Dive into the research topics of 'Towards a Watson that sees: Language-guided action recognition for robots'. Together they form a unique fingerprint.

  • Cite this

    Teo, C. L., Yang, Y., Daumé, H., Fermuller, C., & Aloimonos, Y. (2012). Towards a Watson that sees: Language-guided action recognition for robots. In 2012 IEEE International Conference on Robotics and Automation, ICRA 2012 (pp. 374-381). [6224589] (Proceedings - IEEE International Conference on Robotics and Automation). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICRA.2012.6224589