Simulation-based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components

Cumhur Erkan Tuncali, Georgios Fainekos, Hisahiro Ito, James Kapinski

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

Many organizations are developing autonomous driving systems, which are expected to be deployed at a large scale in the near future. Despite this, there is a lack of agreement on appropriate methods to test, debug, and certify the performance of these systems. One of the main challenges is that many autonomous driving systems have machine learning (ML) components, such as deep neural networks, for which formal properties are difficult to characterize. We present a testing framework that is compatible with test case generation and automatic falsification methods, which are used to evaluate cyber-physical systems. We demonstrate how the framework can be used to evaluate closed-loop properties of an autonomous driving system model that includes the ML components, all within a virtual environment. We demonstrate how to use test case generation methods, such as covering arrays, as well as requirement falsification methods to automatically identify problematic test scenarios. The resulting framework can be used to increase the reliability of autonomous driving systems.

Original languageEnglish (US)
Title of host publication2018 IEEE Intelligent Vehicles Symposium, IV 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1555-1562
Number of pages8
Volume2018-June
ISBN (Electronic)9781538644522
DOIs
StatePublished - Oct 18 2018
Event2018 IEEE Intelligent Vehicles Symposium, IV 2018 - Changshu, Suzhou, China
Duration: Sep 26 2018Sep 30 2018

Other

Other2018 IEEE Intelligent Vehicles Symposium, IV 2018
CountryChina
CityChangshu, Suzhou
Period9/26/189/30/18

Fingerprint

Test Generation
Autonomous Vehicles
Learning systems
Machine Learning
Virtual reality
Simulation
Testing
Covering Array
Evaluate
Virtual Environments
Closed-loop
Demonstrate
Neural Networks
Scenarios
Requirements
Framework
Cyber Physical System
Deep neural networks

ASJC Scopus subject areas

  • Computer Science Applications
  • Automotive Engineering
  • Modeling and Simulation

Cite this

Tuncali, C. E., Fainekos, G., Ito, H., & Kapinski, J. (2018). Simulation-based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components. In 2018 IEEE Intelligent Vehicles Symposium, IV 2018 (Vol. 2018-June, pp. 1555-1562). [8500421] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IVS.2018.8500421

Simulation-based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components. / Tuncali, Cumhur Erkan; Fainekos, Georgios; Ito, Hisahiro; Kapinski, James.

2018 IEEE Intelligent Vehicles Symposium, IV 2018. Vol. 2018-June Institute of Electrical and Electronics Engineers Inc., 2018. p. 1555-1562 8500421.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Tuncali, CE, Fainekos, G, Ito, H & Kapinski, J 2018, Simulation-based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components. in 2018 IEEE Intelligent Vehicles Symposium, IV 2018. vol. 2018-June, 8500421, Institute of Electrical and Electronics Engineers Inc., pp. 1555-1562, 2018 IEEE Intelligent Vehicles Symposium, IV 2018, Changshu, Suzhou, China, 9/26/18. https://doi.org/10.1109/IVS.2018.8500421
Tuncali CE, Fainekos G, Ito H, Kapinski J. Simulation-based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components. In 2018 IEEE Intelligent Vehicles Symposium, IV 2018. Vol. 2018-June. Institute of Electrical and Electronics Engineers Inc. 2018. p. 1555-1562. 8500421 https://doi.org/10.1109/IVS.2018.8500421
Tuncali, Cumhur Erkan ; Fainekos, Georgios ; Ito, Hisahiro ; Kapinski, James. / Simulation-based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components. 2018 IEEE Intelligent Vehicles Symposium, IV 2018. Vol. 2018-June Institute of Electrical and Electronics Engineers Inc., 2018. pp. 1555-1562
@inproceedings{ccc3b9fbdf5545d78138b2d0a1f6cbc0,
title = "Simulation-based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components",
abstract = "Many organizations are developing autonomous driving systems, which are expected to be deployed at a large scale in the near future. Despite this, there is a lack of agreement on appropriate methods to test, debug, and certify the performance of these systems. One of the main challenges is that many autonomous driving systems have machine learning (ML) components, such as deep neural networks, for which formal properties are difficult to characterize. We present a testing framework that is compatible with test case generation and automatic falsification methods, which are used to evaluate cyber-physical systems. We demonstrate how the framework can be used to evaluate closed-loop properties of an autonomous driving system model that includes the ML components, all within a virtual environment. We demonstrate how to use test case generation methods, such as covering arrays, as well as requirement falsification methods to automatically identify problematic test scenarios. The resulting framework can be used to increase the reliability of autonomous driving systems.",
author = "Tuncali, {Cumhur Erkan} and Georgios Fainekos and Hisahiro Ito and James Kapinski",
year = "2018",
month = "10",
day = "18",
doi = "10.1109/IVS.2018.8500421",
language = "English (US)",
volume = "2018-June",
pages = "1555--1562",
booktitle = "2018 IEEE Intelligent Vehicles Symposium, IV 2018",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - GEN

T1 - Simulation-based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components

AU - Tuncali, Cumhur Erkan

AU - Fainekos, Georgios

AU - Ito, Hisahiro

AU - Kapinski, James

PY - 2018/10/18

Y1 - 2018/10/18

N2 - Many organizations are developing autonomous driving systems, which are expected to be deployed at a large scale in the near future. Despite this, there is a lack of agreement on appropriate methods to test, debug, and certify the performance of these systems. One of the main challenges is that many autonomous driving systems have machine learning (ML) components, such as deep neural networks, for which formal properties are difficult to characterize. We present a testing framework that is compatible with test case generation and automatic falsification methods, which are used to evaluate cyber-physical systems. We demonstrate how the framework can be used to evaluate closed-loop properties of an autonomous driving system model that includes the ML components, all within a virtual environment. We demonstrate how to use test case generation methods, such as covering arrays, as well as requirement falsification methods to automatically identify problematic test scenarios. The resulting framework can be used to increase the reliability of autonomous driving systems.

AB - Many organizations are developing autonomous driving systems, which are expected to be deployed at a large scale in the near future. Despite this, there is a lack of agreement on appropriate methods to test, debug, and certify the performance of these systems. One of the main challenges is that many autonomous driving systems have machine learning (ML) components, such as deep neural networks, for which formal properties are difficult to characterize. We present a testing framework that is compatible with test case generation and automatic falsification methods, which are used to evaluate cyber-physical systems. We demonstrate how the framework can be used to evaluate closed-loop properties of an autonomous driving system model that includes the ML components, all within a virtual environment. We demonstrate how to use test case generation methods, such as covering arrays, as well as requirement falsification methods to automatically identify problematic test scenarios. The resulting framework can be used to increase the reliability of autonomous driving systems.

UR - http://www.scopus.com/inward/record.url?scp=85056759390&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85056759390&partnerID=8YFLogxK

U2 - 10.1109/IVS.2018.8500421

DO - 10.1109/IVS.2018.8500421

M3 - Conference contribution

VL - 2018-June

SP - 1555

EP - 1562

BT - 2018 IEEE Intelligent Vehicles Symposium, IV 2018

PB - Institute of Electrical and Electronics Engineers Inc.

ER -