ISparse: Output informed sparsification of neural network

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Deep neural networks have demonstrated unprecedented success in various multimedia applications. However, the networks created are often very complex, with large numbers of trainable edges that require extensive computational resources. We note that many successful networks nevertheless often contain large numbers of redundant edges. Moreover, many of these edges may have negligible contributions towards the overall network performance. In this paper, we propose a novel iSparse framework and experimentally show, that we can sparsify the network without impacting the network performance. iSparse leverages a novel edge significance score, E, to determine the importance of an edge with respect to the final network output. Furthermore, iSparse can be applied both while training a model or on top of a pre-trained model, making it a retraining-free approach - leading to a minimal computational overhead. Comparisons of iSparse against Dropout, L1, DropConnect, Retraining-Free, and Lottery-Ticket Hypothesis on benchmark datasets show that iSparse leads to effective network sparsifications.

Original languageEnglish (US)
Title of host publicationICMR 2020 - Proceedings of the 2020 International Conference on Multimedia Retrieval
PublisherAssociation for Computing Machinery, Inc
Pages180-188
Number of pages9
ISBN (Electronic)9781450370875
DOIs
StatePublished - Jun 8 2020
Event10th ACM International Conference on Multimedia Retrieval, ICMR 2020 - Dublin, Ireland
Duration: Jun 8 2020Jun 11 2020

Publication series

NameICMR 2020 - Proceedings of the 2020 International Conference on Multimedia Retrieval

Conference

Conference10th ACM International Conference on Multimedia Retrieval, ICMR 2020
CountryIreland
CityDublin
Period6/8/206/11/20

Keywords

  • DropConnect
  • Dropout
  • Neural network
  • Pruning
  • Sparsification

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications
  • Computer Graphics and Computer-Aided Design

Fingerprint Dive into the research topics of 'ISparse: Output informed sparsification of neural network'. Together they form a unique fingerprint.

  • Cite this

    Garg, Y., & Candan, K. S. (2020). ISparse: Output informed sparsification of neural network. In ICMR 2020 - Proceedings of the 2020 International Conference on Multimedia Retrieval (pp. 180-188). (ICMR 2020 - Proceedings of the 2020 International Conference on Multimedia Retrieval). Association for Computing Machinery, Inc. https://doi.org/10.1145/3372278.3390688