ClosNets: Batchless DNN Training with On-Chip a Priori Sparse Neural Topologies

Mihailo Isakov, Alan Ehret, Michel Kinsy

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations

Abstract

The deployment of deep neural network (DNN) models is generally hindered by their training time. DNN training throughput is commonly limited by the fully-connected layers. This is due to their large size and low data reuse. Large batch sizes are often used to mitigate some of the effects. Increasing batch size can however hurt model accuracy, creating a tradeoff between accuracy and efficiency. We tackle the problem of training DNNs in on-chip memory, allowing us to train models without the use of batching. Pruning and quantizing dense layers can greatly reduce network size, allowing models to fit on the chip, but can only be applied after training. We propose a fully-connected but sparse layer that reduces the memory requirements of DNNs without sacrificing accuracy. We replace a dense matrix with a sparse matrix product with a predetermined topology. This allows us to: (1) train significantly smaller networks without a loss in accuracy, and (2) store weights without having to store connection indices. We therefore achieve significant training speedups due to the fast access to on-chip weights, smaller network size, and a reduced amount of computation per epoch.

Original languageEnglish (US)
Title of host publicationProceedings - 2018 International Conference on Field-Programmable Logic and Applications, FPL 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages55-59
Number of pages5
ISBN (Electronic)9781538685174
DOIs
StatePublished - Nov 9 2018
Externally publishedYes
Event28th International Conference on Field-Programmable Logic and Applications, FPL 2018 - Dublin, Ireland
Duration: Aug 26 2018Aug 30 2018

Publication series

NameProceedings - 2018 International Conference on Field-Programmable Logic and Applications, FPL 2018

Other

Other28th International Conference on Field-Programmable Logic and Applications, FPL 2018
Country/TerritoryIreland
CityDublin
Period8/26/188/30/18

Keywords

  • acceleration
  • hardware
  • neural network
  • sparsity

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Networks and Communications
  • Computer Science Applications
  • Hardware and Architecture
  • Software

Fingerprint

Dive into the research topics of 'ClosNets: Batchless DNN Training with On-Chip a Priori Sparse Neural Topologies'. Together they form a unique fingerprint.

Cite this