A Locally Weighted Fixation Density-Based Metric for Assessing the Quality of Visual Saliency Predictions

Milind S. Gide, Lina Karam

Research output: Contribution to journalArticle

4 Citations (Scopus)

Abstract

With the increased focus on visual attention (VA) in the last decade, a large number of computational visual saliency methods have been developed over the past few years. These models are traditionally evaluated by using performance evaluation metrics that quantify the match between predicted saliency and fixation data obtained from eye-tracking experiments on human observers. Though a considerable number of such metrics have been proposed in the literature, there are notable problems in them. In this paper, we discuss shortcomings in the existing metrics through illustrative examples and propose a new metric that uses local weights based on fixation density, which overcomes these flaws. To compare the performance of our proposed metric at assessing the quality of saliency prediction with other existing metrics, we construct a ground-truth subjective database in which saliency maps obtained from 17 different VA models are evaluated by 16 human observers on a five-point categorical scale in terms of their visual resemblance with corresponding ground-truth fixation density maps obtained from eye-tracking data. The metrics are evaluated by correlating metric scores with the human subjective ratings. The correlation results show that the proposed evaluation metric outperforms all other popular existing metrics. In addition, the constructed database and corresponding subjective ratings provide an insight into which of the existing metrics and future metrics are better at estimating the quality of saliency prediction and can be used as a benchmark.

Original languageEnglish (US)
Article number7485871
Pages (from-to)3852-3861
Number of pages10
JournalIEEE Transactions on Image Processing
Volume25
Issue number8
DOIs
StatePublished - Aug 1 2016

Fingerprint

Defects
Experiments

Keywords

  • Benchmark
  • Metrics
  • Visual Attention

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design
  • Software

Cite this

A Locally Weighted Fixation Density-Based Metric for Assessing the Quality of Visual Saliency Predictions. / Gide, Milind S.; Karam, Lina.

In: IEEE Transactions on Image Processing, Vol. 25, No. 8, 7485871, 01.08.2016, p. 3852-3861.

Research output: Contribution to journalArticle

@article{79054b9186fb43e0b20827a15c7364fa,
title = "A Locally Weighted Fixation Density-Based Metric for Assessing the Quality of Visual Saliency Predictions",
abstract = "With the increased focus on visual attention (VA) in the last decade, a large number of computational visual saliency methods have been developed over the past few years. These models are traditionally evaluated by using performance evaluation metrics that quantify the match between predicted saliency and fixation data obtained from eye-tracking experiments on human observers. Though a considerable number of such metrics have been proposed in the literature, there are notable problems in them. In this paper, we discuss shortcomings in the existing metrics through illustrative examples and propose a new metric that uses local weights based on fixation density, which overcomes these flaws. To compare the performance of our proposed metric at assessing the quality of saliency prediction with other existing metrics, we construct a ground-truth subjective database in which saliency maps obtained from 17 different VA models are evaluated by 16 human observers on a five-point categorical scale in terms of their visual resemblance with corresponding ground-truth fixation density maps obtained from eye-tracking data. The metrics are evaluated by correlating metric scores with the human subjective ratings. The correlation results show that the proposed evaluation metric outperforms all other popular existing metrics. In addition, the constructed database and corresponding subjective ratings provide an insight into which of the existing metrics and future metrics are better at estimating the quality of saliency prediction and can be used as a benchmark.",
keywords = "Benchmark, Metrics, Visual Attention",
author = "Gide, {Milind S.} and Lina Karam",
year = "2016",
month = "8",
day = "1",
doi = "10.1109/TIP.2016.2577498",
language = "English (US)",
volume = "25",
pages = "3852--3861",
journal = "IEEE Transactions on Image Processing",
issn = "1057-7149",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "8",

}

TY - JOUR

T1 - A Locally Weighted Fixation Density-Based Metric for Assessing the Quality of Visual Saliency Predictions

AU - Gide, Milind S.

AU - Karam, Lina

PY - 2016/8/1

Y1 - 2016/8/1

N2 - With the increased focus on visual attention (VA) in the last decade, a large number of computational visual saliency methods have been developed over the past few years. These models are traditionally evaluated by using performance evaluation metrics that quantify the match between predicted saliency and fixation data obtained from eye-tracking experiments on human observers. Though a considerable number of such metrics have been proposed in the literature, there are notable problems in them. In this paper, we discuss shortcomings in the existing metrics through illustrative examples and propose a new metric that uses local weights based on fixation density, which overcomes these flaws. To compare the performance of our proposed metric at assessing the quality of saliency prediction with other existing metrics, we construct a ground-truth subjective database in which saliency maps obtained from 17 different VA models are evaluated by 16 human observers on a five-point categorical scale in terms of their visual resemblance with corresponding ground-truth fixation density maps obtained from eye-tracking data. The metrics are evaluated by correlating metric scores with the human subjective ratings. The correlation results show that the proposed evaluation metric outperforms all other popular existing metrics. In addition, the constructed database and corresponding subjective ratings provide an insight into which of the existing metrics and future metrics are better at estimating the quality of saliency prediction and can be used as a benchmark.

AB - With the increased focus on visual attention (VA) in the last decade, a large number of computational visual saliency methods have been developed over the past few years. These models are traditionally evaluated by using performance evaluation metrics that quantify the match between predicted saliency and fixation data obtained from eye-tracking experiments on human observers. Though a considerable number of such metrics have been proposed in the literature, there are notable problems in them. In this paper, we discuss shortcomings in the existing metrics through illustrative examples and propose a new metric that uses local weights based on fixation density, which overcomes these flaws. To compare the performance of our proposed metric at assessing the quality of saliency prediction with other existing metrics, we construct a ground-truth subjective database in which saliency maps obtained from 17 different VA models are evaluated by 16 human observers on a five-point categorical scale in terms of their visual resemblance with corresponding ground-truth fixation density maps obtained from eye-tracking data. The metrics are evaluated by correlating metric scores with the human subjective ratings. The correlation results show that the proposed evaluation metric outperforms all other popular existing metrics. In addition, the constructed database and corresponding subjective ratings provide an insight into which of the existing metrics and future metrics are better at estimating the quality of saliency prediction and can be used as a benchmark.

KW - Benchmark

KW - Metrics

KW - Visual Attention

UR - http://www.scopus.com/inward/record.url?scp=84978998422&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84978998422&partnerID=8YFLogxK

U2 - 10.1109/TIP.2016.2577498

DO - 10.1109/TIP.2016.2577498

M3 - Article

C2 - 27295671

AN - SCOPUS:84978998422

VL - 25

SP - 3852

EP - 3861

JO - IEEE Transactions on Image Processing

JF - IEEE Transactions on Image Processing

SN - 1057-7149

IS - 8

M1 - 7485871

ER -