TY - GEN
T1 - Single-Net continual learning with progressive segmented training
AU - Du, Xiaocong
AU - Charan, Gouranga
AU - Liu, Frank
AU - Cao, Yu
PY - 2019/12
Y1 - 2019/12
N2 - There is an increasing need of continual learning in dynamic systems, such as the self-driving vehicle, the surveillance drone, and the robotic system. Such a system requires learning from the data stream, training the model to preserve previous information and adapt to a new task, and generating a single-headed vector for future inference. Different from previous approaches with dynamic structures, this work focuses on a single network and model segmentation to prevent catastrophic forgetting. Leveraging the redundant capacity of a single network, model parameters for each task are separated into two groups: one important group which is frozen to preserve current knowledge, and secondary group to be saved (not pruned) for a future learning. A fixed-size memory containing a small amount of previously seen data is further adopted to assist the training. Without additional regularization, the simple yet effective approach of Progressive Segmented Training (PST) successfully incorporates multiple tasks and achieves the state-of-the-art accuracy in the single-head evaluation on CIFAR-10 and CIFAR-100 datasets. Moreover, the segmented training significantly improves computation efficiency in continual learning at the edge.
AB - There is an increasing need of continual learning in dynamic systems, such as the self-driving vehicle, the surveillance drone, and the robotic system. Such a system requires learning from the data stream, training the model to preserve previous information and adapt to a new task, and generating a single-headed vector for future inference. Different from previous approaches with dynamic structures, this work focuses on a single network and model segmentation to prevent catastrophic forgetting. Leveraging the redundant capacity of a single network, model parameters for each task are separated into two groups: one important group which is frozen to preserve current knowledge, and secondary group to be saved (not pruned) for a future learning. A fixed-size memory containing a small amount of previously seen data is further adopted to assist the training. Without additional regularization, the simple yet effective approach of Progressive Segmented Training (PST) successfully incorporates multiple tasks and achieves the state-of-the-art accuracy in the single-head evaluation on CIFAR-10 and CIFAR-100 datasets. Moreover, the segmented training significantly improves computation efficiency in continual learning at the edge.
KW - Computer Vision
KW - Continual Learning
KW - Convolutional Neural Network
KW - Deep Learning
KW - Image Recognition
UR - http://www.scopus.com/inward/record.url?scp=85080855021&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85080855021&partnerID=8YFLogxK
U2 - 10.1109/ICMLA.2019.00267
DO - 10.1109/ICMLA.2019.00267
M3 - Conference contribution
T3 - Proceedings - 18th IEEE International Conference on Machine Learning and Applications, ICMLA 2019
SP - 1629
EP - 1636
BT - Proceedings - 18th IEEE International Conference on Machine Learning and Applications, ICMLA 2019
A2 - Wani, M. Arif
A2 - Khoshgoftaar, Taghi M.
A2 - Wang, Dingding
A2 - Wang, Huanjing
A2 - Seliya, Naeem
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 18th IEEE International Conference on Machine Learning and Applications, ICMLA 2019
Y2 - 16 December 2019 through 19 December 2019
ER -