Combining knowledge and reasoning through probabilistic soft logic for image puzzle solving

Somak Aditya, Yezhou Yang, Chitta Baral, Yiannis Aloimonos

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The uncertainty associated with human perception is often reduced by one's extensive prior experience and knowledge. Current datasets and systems do not emphasize the necessity and benefit of using such knowledge. In this work, we propose the task of solving a genre of image-puzzles ("image riddles") that require both capabilities involving visual detection (including object, activity recognition) and, knowledge-based or commonsense reasoning. Each puzzle involves a set of images and the question "what word connects these images?". We compile a dataset of over 3k riddles where each riddle consists of 4 images and a groundtruth answer. The annotations are validated using crowd-sourced evaluation. We also define an automatic evaluation metric to track future progress. Our task bears similarity with the commonly known IQ tasks such as analogy solving, sequence filling that are often used to test intelligence. We develop a Probabilistic Reasoning-based approach that utilizes commonsense knowledge about words and phrases to answer these riddles with a reasonable accuracy. Our approach achieves some promising results for these riddles and provides a strong baseline for future attempts. We make the entire dataset and related materials publicly available to the community (bit.ly/22f9Ala).

Original languageEnglish (US)
Title of host publication34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018
EditorsAmir Globerson, Amir Globerson, Ricardo Silva
PublisherAssociation For Uncertainty in Artificial Intelligence (AUAI)
Pages238-248
Number of pages11
Volume1
ISBN (Electronic)9781510871601
StatePublished - Jan 1 2018
Event34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018 - Monterey, United States
Duration: Aug 6 2018Aug 10 2018

Other

Other34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018
CountryUnited States
CityMonterey
Period8/6/188/10/18

Fingerprint

Object detection
Uncertainty

ASJC Scopus subject areas

  • Artificial Intelligence

Cite this

Aditya, S., Yang, Y., Baral, C., & Aloimonos, Y. (2018). Combining knowledge and reasoning through probabilistic soft logic for image puzzle solving. In A. Globerson, A. Globerson, & R. Silva (Eds.), 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018 (Vol. 1, pp. 238-248). Association For Uncertainty in Artificial Intelligence (AUAI).

Combining knowledge and reasoning through probabilistic soft logic for image puzzle solving. / Aditya, Somak; Yang, Yezhou; Baral, Chitta; Aloimonos, Yiannis.

34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018. ed. / Amir Globerson; Amir Globerson; Ricardo Silva. Vol. 1 Association For Uncertainty in Artificial Intelligence (AUAI), 2018. p. 238-248.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Aditya, S, Yang, Y, Baral, C & Aloimonos, Y 2018, Combining knowledge and reasoning through probabilistic soft logic for image puzzle solving. in A Globerson, A Globerson & R Silva (eds), 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018. vol. 1, Association For Uncertainty in Artificial Intelligence (AUAI), pp. 238-248, 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, Monterey, United States, 8/6/18.
Aditya S, Yang Y, Baral C, Aloimonos Y. Combining knowledge and reasoning through probabilistic soft logic for image puzzle solving. In Globerson A, Globerson A, Silva R, editors, 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018. Vol. 1. Association For Uncertainty in Artificial Intelligence (AUAI). 2018. p. 238-248
Aditya, Somak ; Yang, Yezhou ; Baral, Chitta ; Aloimonos, Yiannis. / Combining knowledge and reasoning through probabilistic soft logic for image puzzle solving. 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018. editor / Amir Globerson ; Amir Globerson ; Ricardo Silva. Vol. 1 Association For Uncertainty in Artificial Intelligence (AUAI), 2018. pp. 238-248
@inproceedings{c3b429cc83fc4da3890d17258e442fe2,
title = "Combining knowledge and reasoning through probabilistic soft logic for image puzzle solving",
abstract = "The uncertainty associated with human perception is often reduced by one's extensive prior experience and knowledge. Current datasets and systems do not emphasize the necessity and benefit of using such knowledge. In this work, we propose the task of solving a genre of image-puzzles ({"}image riddles{"}) that require both capabilities involving visual detection (including object, activity recognition) and, knowledge-based or commonsense reasoning. Each puzzle involves a set of images and the question {"}what word connects these images?{"}. We compile a dataset of over 3k riddles where each riddle consists of 4 images and a groundtruth answer. The annotations are validated using crowd-sourced evaluation. We also define an automatic evaluation metric to track future progress. Our task bears similarity with the commonly known IQ tasks such as analogy solving, sequence filling that are often used to test intelligence. We develop a Probabilistic Reasoning-based approach that utilizes commonsense knowledge about words and phrases to answer these riddles with a reasonable accuracy. Our approach achieves some promising results for these riddles and provides a strong baseline for future attempts. We make the entire dataset and related materials publicly available to the community (bit.ly/22f9Ala).",
author = "Somak Aditya and Yezhou Yang and Chitta Baral and Yiannis Aloimonos",
year = "2018",
month = "1",
day = "1",
language = "English (US)",
volume = "1",
pages = "238--248",
editor = "Amir Globerson and Amir Globerson and Ricardo Silva",
booktitle = "34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018",
publisher = "Association For Uncertainty in Artificial Intelligence (AUAI)",

}

TY - GEN

T1 - Combining knowledge and reasoning through probabilistic soft logic for image puzzle solving

AU - Aditya, Somak

AU - Yang, Yezhou

AU - Baral, Chitta

AU - Aloimonos, Yiannis

PY - 2018/1/1

Y1 - 2018/1/1

N2 - The uncertainty associated with human perception is often reduced by one's extensive prior experience and knowledge. Current datasets and systems do not emphasize the necessity and benefit of using such knowledge. In this work, we propose the task of solving a genre of image-puzzles ("image riddles") that require both capabilities involving visual detection (including object, activity recognition) and, knowledge-based or commonsense reasoning. Each puzzle involves a set of images and the question "what word connects these images?". We compile a dataset of over 3k riddles where each riddle consists of 4 images and a groundtruth answer. The annotations are validated using crowd-sourced evaluation. We also define an automatic evaluation metric to track future progress. Our task bears similarity with the commonly known IQ tasks such as analogy solving, sequence filling that are often used to test intelligence. We develop a Probabilistic Reasoning-based approach that utilizes commonsense knowledge about words and phrases to answer these riddles with a reasonable accuracy. Our approach achieves some promising results for these riddles and provides a strong baseline for future attempts. We make the entire dataset and related materials publicly available to the community (bit.ly/22f9Ala).

AB - The uncertainty associated with human perception is often reduced by one's extensive prior experience and knowledge. Current datasets and systems do not emphasize the necessity and benefit of using such knowledge. In this work, we propose the task of solving a genre of image-puzzles ("image riddles") that require both capabilities involving visual detection (including object, activity recognition) and, knowledge-based or commonsense reasoning. Each puzzle involves a set of images and the question "what word connects these images?". We compile a dataset of over 3k riddles where each riddle consists of 4 images and a groundtruth answer. The annotations are validated using crowd-sourced evaluation. We also define an automatic evaluation metric to track future progress. Our task bears similarity with the commonly known IQ tasks such as analogy solving, sequence filling that are often used to test intelligence. We develop a Probabilistic Reasoning-based approach that utilizes commonsense knowledge about words and phrases to answer these riddles with a reasonable accuracy. Our approach achieves some promising results for these riddles and provides a strong baseline for future attempts. We make the entire dataset and related materials publicly available to the community (bit.ly/22f9Ala).

UR - http://www.scopus.com/inward/record.url?scp=85059425637&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85059425637&partnerID=8YFLogxK

M3 - Conference contribution

VL - 1

SP - 238

EP - 248

BT - 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018

A2 - Globerson, Amir

A2 - Globerson, Amir

A2 - Silva, Ricardo

PB - Association For Uncertainty in Artificial Intelligence (AUAI)

ER -