TY - GEN
T1 - Simulation-based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components
AU - Tuncali, Cumhur Erkan
AU - Fainekos, Georgios
AU - Ito, Hisahiro
AU - Kapinski, James
N1 - Funding Information:
*This work was partially funded by NSF awards CNS 1446730, 1350420 1Toyota Technical Center, Ann Arbor MI, USA cumhur.tuncali, hisahiro.ito, jim.kapinski@toyota.com 2School of Computing, Informatics & Decision Systems Engineering, Arizona State University, USA Georgios.Fainekos@asu.edu
Publisher Copyright:
© 2018 IEEE.
PY - 2018/10/18
Y1 - 2018/10/18
N2 - Many organizations are developing autonomous driving systems, which are expected to be deployed at a large scale in the near future. Despite this, there is a lack of agreement on appropriate methods to test, debug, and certify the performance of these systems. One of the main challenges is that many autonomous driving systems have machine learning (ML) components, such as deep neural networks, for which formal properties are difficult to characterize. We present a testing framework that is compatible with test case generation and automatic falsification methods, which are used to evaluate cyber-physical systems. We demonstrate how the framework can be used to evaluate closed-loop properties of an autonomous driving system model that includes the ML components, all within a virtual environment. We demonstrate how to use test case generation methods, such as covering arrays, as well as requirement falsification methods to automatically identify problematic test scenarios. The resulting framework can be used to increase the reliability of autonomous driving systems.
AB - Many organizations are developing autonomous driving systems, which are expected to be deployed at a large scale in the near future. Despite this, there is a lack of agreement on appropriate methods to test, debug, and certify the performance of these systems. One of the main challenges is that many autonomous driving systems have machine learning (ML) components, such as deep neural networks, for which formal properties are difficult to characterize. We present a testing framework that is compatible with test case generation and automatic falsification methods, which are used to evaluate cyber-physical systems. We demonstrate how the framework can be used to evaluate closed-loop properties of an autonomous driving system model that includes the ML components, all within a virtual environment. We demonstrate how to use test case generation methods, such as covering arrays, as well as requirement falsification methods to automatically identify problematic test scenarios. The resulting framework can be used to increase the reliability of autonomous driving systems.
UR - http://www.scopus.com/inward/record.url?scp=85056759390&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85056759390&partnerID=8YFLogxK
U2 - 10.1109/IVS.2018.8500421
DO - 10.1109/IVS.2018.8500421
M3 - Conference contribution
AN - SCOPUS:85056759390
T3 - IEEE Intelligent Vehicles Symposium, Proceedings
SP - 1555
EP - 1562
BT - 2018 IEEE Intelligent Vehicles Symposium, IV 2018
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 2018 IEEE Intelligent Vehicles Symposium, IV 2018
Y2 - 26 September 2018 through 30 September 2018
ER -