Dropout as an implicit gating mechanism for continual learning

Seyed Iman Mirzadeh, Mehrdad Farajtabar, Hassan Ghasemzadeh

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Scopus citations

Abstract

In recent years, neural networks have demonstrated an outstanding ability to achieve complex learning tasks across various domains. However, they suffer from the "catastrophic forgetting" problem when they face a sequence of learning tasks, where they forget the old ones as they learn new tasks. This problem is also highly related to the "stability-plasticity dilemma". The more plastic the network, the easier it can learn new tasks, but the faster it also forgets previous ones. Conversely, a stable network cannot learn new tasks as fast as a very plastic network. However, it is more reliable to preserve the knowledge it has learned from the previous tasks. Several solutions have been proposed to overcome the forgetting problem by making the neural network parameters more stable, and some of them have mentioned the significance of dropout in continual learning. However, their relationship has not been sufficiently studied yet. In this paper, we investigate this relationship and show that a stable network with dropout learns a gating mechanism such that for different tasks, different paths of the network are active. Our experiments show that the stability achieved by this implicit gating plays a very critical role in leading to performance comparable to or better than other involved continual learning algorithms to overcome catastrophic forgetting.1

Original languageEnglish (US)
Title of host publicationProceedings - 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2020
PublisherIEEE Computer Society
Pages945-951
Number of pages7
ISBN (Electronic)9781728193601
DOIs
StatePublished - Jun 2020
Externally publishedYes
Event2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2020 - Virtual, Online, United States
Duration: Jun 14 2020Jun 19 2020

Publication series

NameIEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops
Volume2020-June
ISSN (Print)2160-7508
ISSN (Electronic)2160-7516

Conference

Conference2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2020
Country/TerritoryUnited States
CityVirtual, Online
Period6/14/206/19/20

ASJC Scopus subject areas

  • Computer Vision and Pattern Recognition
  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'Dropout as an implicit gating mechanism for continual learning'. Together they form a unique fingerprint.

Cite this