TY - GEN
T1 - Specifying and Evaluating Quality Metrics for Vision-based Perception Systems
AU - Balakrishnan, Anand
AU - Puranic, Aniruddh G.
AU - Qin, Xin
AU - Dokhanchi, Adel
AU - Deshmukh, Jyotirmoy V.
AU - Ben Amor, Heni
AU - Fainekos, Georgios
N1 - Funding Information:
Acknowledgements This work was partially supported by the NSF I/UCRC Center for Embedded Systems and by NSF grants 1350420, 1361926, 1446730, and CCF 1837131.
Funding Information:
This work was partially supported by the NSF I/UCRC Center for Embedded Systems and by NSF grants 1350420, 1361926, 1446730, and CCF 1837131.
Publisher Copyright:
© 2019 EDAA.
PY - 2019/5/14
Y1 - 2019/5/14
N2 - Robust perception algorithms are a vital ingredient for autonomous systems such as self-driving vehicles. Checking the correctness of perception algorithms such as those based on deep convolutional neural networks (CNN) is a formidable challenge problem. In this paper, we suggest the use of Timed Quality Temporal Logic (TQTL) as a formal language to express desirable spatio-temporal properties of a perception algorithm processing a video. While perception algorithms are traditionally tested by comparing their performance to ground truth labels, we show how TQTL can be a useful tool to determine quality of perception, and offers an alternative metric that can give useful information, even in the absence of ground truth labels. We demonstrate TQTL monitoring on two popular CNNs: YOLO and SqueezeDet, and give a comparative study of the results obtained for each architecture.
AB - Robust perception algorithms are a vital ingredient for autonomous systems such as self-driving vehicles. Checking the correctness of perception algorithms such as those based on deep convolutional neural networks (CNN) is a formidable challenge problem. In this paper, we suggest the use of Timed Quality Temporal Logic (TQTL) as a formal language to express desirable spatio-temporal properties of a perception algorithm processing a video. While perception algorithms are traditionally tested by comparing their performance to ground truth labels, we show how TQTL can be a useful tool to determine quality of perception, and offers an alternative metric that can give useful information, even in the absence of ground truth labels. We demonstrate TQTL monitoring on two popular CNNs: YOLO and SqueezeDet, and give a comparative study of the results obtained for each architecture.
KW - Autonomous vehicles
KW - Image processing
KW - Monitoring
KW - Perception
KW - Quality Metrics
KW - Temporal Logic
UR - http://www.scopus.com/inward/record.url?scp=85066619950&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85066619950&partnerID=8YFLogxK
U2 - 10.23919/DATE.2019.8715114
DO - 10.23919/DATE.2019.8715114
M3 - Conference contribution
AN - SCOPUS:85066619950
T3 - Proceedings of the 2019 Design, Automation and Test in Europe Conference and Exhibition, DATE 2019
SP - 1433
EP - 1438
BT - Proceedings of the 2019 Design, Automation and Test in Europe Conference and Exhibition, DATE 2019
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 22nd Design, Automation and Test in Europe Conference and Exhibition, DATE 2019
Y2 - 25 March 2019 through 29 March 2019
ER -