Single-Net continual learning with progressive segmented training

Xiaocong Du, Gouranga Charan, Frank Liu, Yu Cao

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

There is an increasing need of continual learning in dynamic systems, such as the self-driving vehicle, the surveillance drone, and the robotic system. Such a system requires learning from the data stream, training the model to preserve previous information and adapt to a new task, and generating a single-headed vector for future inference. Different from previous approaches with dynamic structures, this work focuses on a single network and model segmentation to prevent catastrophic forgetting. Leveraging the redundant capacity of a single network, model parameters for each task are separated into two groups: one important group which is frozen to preserve current knowledge, and secondary group to be saved (not pruned) for a future learning. A fixed-size memory containing a small amount of previously seen data is further adopted to assist the training. Without additional regularization, the simple yet effective approach of Progressive Segmented Training (PST) successfully incorporates multiple tasks and achieves the state-of-the-art accuracy in the single-head evaluation on CIFAR-10 and CIFAR-100 datasets. Moreover, the segmented training significantly improves computation efficiency in continual learning at the edge.

Original languageEnglish (US)
Title of host publicationProceedings - 18th IEEE International Conference on Machine Learning and Applications, ICMLA 2019
EditorsM. Arif Wani, Taghi M. Khoshgoftaar, Dingding Wang, Huanjing Wang, Naeem Seliya
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1629-1636
Number of pages8
ISBN (Electronic)9781728145495
DOIs
StatePublished - Dec 2019
Event18th IEEE International Conference on Machine Learning and Applications, ICMLA 2019 - Boca Raton, United States
Duration: Dec 16 2019Dec 19 2019

Publication series

NameProceedings - 18th IEEE International Conference on Machine Learning and Applications, ICMLA 2019

Conference

Conference18th IEEE International Conference on Machine Learning and Applications, ICMLA 2019
CountryUnited States
CityBoca Raton
Period12/16/1912/19/19

Keywords

  • Computer Vision
  • Continual Learning
  • Convolutional Neural Network
  • Deep Learning
  • Image Recognition

ASJC Scopus subject areas

  • Strategy and Management
  • Artificial Intelligence
  • Computer Science Applications
  • Decision Sciences (miscellaneous)
  • Signal Processing
  • Media Technology

Fingerprint Dive into the research topics of 'Single-Net continual learning with progressive segmented training'. Together they form a unique fingerprint.

  • Cite this

    Du, X., Charan, G., Liu, F., & Cao, Y. (2019). Single-Net continual learning with progressive segmented training. In M. A. Wani, T. M. Khoshgoftaar, D. Wang, H. Wang, & N. Seliya (Eds.), Proceedings - 18th IEEE International Conference on Machine Learning and Applications, ICMLA 2019 (pp. 1629-1636). [8999103] (Proceedings - 18th IEEE International Conference on Machine Learning and Applications, ICMLA 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICMLA.2019.00267