Specifying and Evaluating Quality Metrics for Vision-based Perception Systems

Anand Balakrishnan, Aniruddh G. Puranic, Xin Qin, Adel Dokhanchi, Jyotirmoy V. Deshmukh, Hani Ben Amor, Georgios Fainekos

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Robust perception algorithms are a vital ingredient for autonomous systems such as self-driving vehicles. Checking the correctness of perception algorithms such as those based on deep convolutional neural networks (CNN) is a formidable challenge problem. In this paper, we suggest the use of Timed Quality Temporal Logic (TQTL) as a formal language to express desirable spatio-temporal properties of a perception algorithm processing a video. While perception algorithms are traditionally tested by comparing their performance to ground truth labels, we show how TQTL can be a useful tool to determine quality of perception, and offers an alternative metric that can give useful information, even in the absence of ground truth labels. We demonstrate TQTL monitoring on two popular CNNs: YOLO and SqueezeDet, and give a comparative study of the results obtained for each architecture.

Original languageEnglish (US)
Title of host publicationProceedings of the 2019 Design, Automation and Test in Europe Conference and Exhibition, DATE 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1433-1438
Number of pages6
ISBN (Electronic)9783981926323
DOIs
StatePublished - May 14 2019
Event22nd Design, Automation and Test in Europe Conference and Exhibition, DATE 2019 - Florence, Italy
Duration: Mar 25 2019Mar 29 2019

Publication series

NameProceedings of the 2019 Design, Automation and Test in Europe Conference and Exhibition, DATE 2019

Conference

Conference22nd Design, Automation and Test in Europe Conference and Exhibition, DATE 2019
CountryItaly
CityFlorence
Period3/25/193/29/19

Fingerprint

Temporal logic
Metric
Temporal Logic
Labels
Formal languages
Formal Languages
Autonomous Systems
Comparative Study
Neural networks
Correctness
Express
Perception
Vision
Monitoring
Neural Networks
Processing
Alternatives
Demonstrate
Truth

Keywords

  • Autonomous vehicles
  • Image processing
  • Monitoring
  • Perception
  • Quality Metrics
  • Temporal Logic

ASJC Scopus subject areas

  • Hardware and Architecture
  • Electrical and Electronic Engineering
  • Safety, Risk, Reliability and Quality
  • Control and Optimization

Cite this

Balakrishnan, A., Puranic, A. G., Qin, X., Dokhanchi, A., Deshmukh, J. V., Ben Amor, H., & Fainekos, G. (2019). Specifying and Evaluating Quality Metrics for Vision-based Perception Systems. In Proceedings of the 2019 Design, Automation and Test in Europe Conference and Exhibition, DATE 2019 (pp. 1433-1438). [8715114] (Proceedings of the 2019 Design, Automation and Test in Europe Conference and Exhibition, DATE 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.23919/DATE.2019.8715114

Specifying and Evaluating Quality Metrics for Vision-based Perception Systems. / Balakrishnan, Anand; Puranic, Aniruddh G.; Qin, Xin; Dokhanchi, Adel; Deshmukh, Jyotirmoy V.; Ben Amor, Hani; Fainekos, Georgios.

Proceedings of the 2019 Design, Automation and Test in Europe Conference and Exhibition, DATE 2019. Institute of Electrical and Electronics Engineers Inc., 2019. p. 1433-1438 8715114 (Proceedings of the 2019 Design, Automation and Test in Europe Conference and Exhibition, DATE 2019).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Balakrishnan, A, Puranic, AG, Qin, X, Dokhanchi, A, Deshmukh, JV, Ben Amor, H & Fainekos, G 2019, Specifying and Evaluating Quality Metrics for Vision-based Perception Systems. in Proceedings of the 2019 Design, Automation and Test in Europe Conference and Exhibition, DATE 2019., 8715114, Proceedings of the 2019 Design, Automation and Test in Europe Conference and Exhibition, DATE 2019, Institute of Electrical and Electronics Engineers Inc., pp. 1433-1438, 22nd Design, Automation and Test in Europe Conference and Exhibition, DATE 2019, Florence, Italy, 3/25/19. https://doi.org/10.23919/DATE.2019.8715114
Balakrishnan A, Puranic AG, Qin X, Dokhanchi A, Deshmukh JV, Ben Amor H et al. Specifying and Evaluating Quality Metrics for Vision-based Perception Systems. In Proceedings of the 2019 Design, Automation and Test in Europe Conference and Exhibition, DATE 2019. Institute of Electrical and Electronics Engineers Inc. 2019. p. 1433-1438. 8715114. (Proceedings of the 2019 Design, Automation and Test in Europe Conference and Exhibition, DATE 2019). https://doi.org/10.23919/DATE.2019.8715114
Balakrishnan, Anand ; Puranic, Aniruddh G. ; Qin, Xin ; Dokhanchi, Adel ; Deshmukh, Jyotirmoy V. ; Ben Amor, Hani ; Fainekos, Georgios. / Specifying and Evaluating Quality Metrics for Vision-based Perception Systems. Proceedings of the 2019 Design, Automation and Test in Europe Conference and Exhibition, DATE 2019. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 1433-1438 (Proceedings of the 2019 Design, Automation and Test in Europe Conference and Exhibition, DATE 2019).
@inproceedings{a5453b7827b4490ab255bafe2dddfc86,
title = "Specifying and Evaluating Quality Metrics for Vision-based Perception Systems",
abstract = "Robust perception algorithms are a vital ingredient for autonomous systems such as self-driving vehicles. Checking the correctness of perception algorithms such as those based on deep convolutional neural networks (CNN) is a formidable challenge problem. In this paper, we suggest the use of Timed Quality Temporal Logic (TQTL) as a formal language to express desirable spatio-temporal properties of a perception algorithm processing a video. While perception algorithms are traditionally tested by comparing their performance to ground truth labels, we show how TQTL can be a useful tool to determine quality of perception, and offers an alternative metric that can give useful information, even in the absence of ground truth labels. We demonstrate TQTL monitoring on two popular CNNs: YOLO and SqueezeDet, and give a comparative study of the results obtained for each architecture.",
keywords = "Autonomous vehicles, Image processing, Monitoring, Perception, Quality Metrics, Temporal Logic",
author = "Anand Balakrishnan and Puranic, {Aniruddh G.} and Xin Qin and Adel Dokhanchi and Deshmukh, {Jyotirmoy V.} and {Ben Amor}, Hani and Georgios Fainekos",
year = "2019",
month = "5",
day = "14",
doi = "10.23919/DATE.2019.8715114",
language = "English (US)",
series = "Proceedings of the 2019 Design, Automation and Test in Europe Conference and Exhibition, DATE 2019",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "1433--1438",
booktitle = "Proceedings of the 2019 Design, Automation and Test in Europe Conference and Exhibition, DATE 2019",

}

TY - GEN

T1 - Specifying and Evaluating Quality Metrics for Vision-based Perception Systems

AU - Balakrishnan, Anand

AU - Puranic, Aniruddh G.

AU - Qin, Xin

AU - Dokhanchi, Adel

AU - Deshmukh, Jyotirmoy V.

AU - Ben Amor, Hani

AU - Fainekos, Georgios

PY - 2019/5/14

Y1 - 2019/5/14

N2 - Robust perception algorithms are a vital ingredient for autonomous systems such as self-driving vehicles. Checking the correctness of perception algorithms such as those based on deep convolutional neural networks (CNN) is a formidable challenge problem. In this paper, we suggest the use of Timed Quality Temporal Logic (TQTL) as a formal language to express desirable spatio-temporal properties of a perception algorithm processing a video. While perception algorithms are traditionally tested by comparing their performance to ground truth labels, we show how TQTL can be a useful tool to determine quality of perception, and offers an alternative metric that can give useful information, even in the absence of ground truth labels. We demonstrate TQTL monitoring on two popular CNNs: YOLO and SqueezeDet, and give a comparative study of the results obtained for each architecture.

AB - Robust perception algorithms are a vital ingredient for autonomous systems such as self-driving vehicles. Checking the correctness of perception algorithms such as those based on deep convolutional neural networks (CNN) is a formidable challenge problem. In this paper, we suggest the use of Timed Quality Temporal Logic (TQTL) as a formal language to express desirable spatio-temporal properties of a perception algorithm processing a video. While perception algorithms are traditionally tested by comparing their performance to ground truth labels, we show how TQTL can be a useful tool to determine quality of perception, and offers an alternative metric that can give useful information, even in the absence of ground truth labels. We demonstrate TQTL monitoring on two popular CNNs: YOLO and SqueezeDet, and give a comparative study of the results obtained for each architecture.

KW - Autonomous vehicles

KW - Image processing

KW - Monitoring

KW - Perception

KW - Quality Metrics

KW - Temporal Logic

UR - http://www.scopus.com/inward/record.url?scp=85066619950&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85066619950&partnerID=8YFLogxK

U2 - 10.23919/DATE.2019.8715114

DO - 10.23919/DATE.2019.8715114

M3 - Conference contribution

T3 - Proceedings of the 2019 Design, Automation and Test in Europe Conference and Exhibition, DATE 2019

SP - 1433

EP - 1438

BT - Proceedings of the 2019 Design, Automation and Test in Europe Conference and Exhibition, DATE 2019

PB - Institute of Electrical and Electronics Engineers Inc.

ER -