Algorithm and hardware design of discrete-time spiking neural networks based on back propagation with binary activations

Shihui Yin, Shreyas K. Venkataramanaiah, Gregory K. Chen, Ram Krishnamurthy, Yu Cao, Chaitali Chakrabarti, Jae-sun Seo

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

We present a new back propagation based training algorithm for discrete-time spiking neural networks (SNN). Inspired by recent deep learning algorithms on binarized neural networks, binary activation with a straight-through gradient estimator is used to model the leaky integrate-fire spiking neuron, overcoming the difficulty in training SNNs using back propagation. Two SNN training algorithms are proposed: (1) SNN with discontinuous integration, which is suitable for rate-coded input spikes, and (2) SNN with continuous integration, which is more general and can handle input spikes with temporal information. Neuromorphic hardware designed in 28nm CMOS exploits the spike sparsity and demonstrates high classification accuracy (>98% on MNIST) and low energy (51.4-773 nJ/image).

Original languageEnglish (US)
Title of host publication2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1-4
Number of pages4
Volume2018-January
ISBN (Electronic)9781509058037
DOIs
StatePublished - Mar 23 2018
Event2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Torino, Italy
Duration: Oct 19 2017Oct 21 2017

Other

Other2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017
CountryItaly
CityTorino
Period10/19/1710/21/17

Fingerprint

spiking
Backpropagation
hardware
Chemical activation
activation
Neural networks
Hardware
spikes
education
neurons
estimators
Learning algorithms
learning
Neurons
CMOS
Fires
gradients

Keywords

  • back propagation
  • neuromorphic hardware
  • Spiking neural networks
  • straight-through estimator

ASJC Scopus subject areas

  • Biomedical Engineering
  • Electrical and Electronic Engineering
  • Instrumentation

Cite this

Yin, S., Venkataramanaiah, S. K., Chen, G. K., Krishnamurthy, R., Cao, Y., Chakrabarti, C., & Seo, J. (2018). Algorithm and hardware design of discrete-time spiking neural networks based on back propagation with binary activations. In 2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Proceedings (Vol. 2018-January, pp. 1-4). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/BIOCAS.2017.8325230

Algorithm and hardware design of discrete-time spiking neural networks based on back propagation with binary activations. / Yin, Shihui; Venkataramanaiah, Shreyas K.; Chen, Gregory K.; Krishnamurthy, Ram; Cao, Yu; Chakrabarti, Chaitali; Seo, Jae-sun.

2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Proceedings. Vol. 2018-January Institute of Electrical and Electronics Engineers Inc., 2018. p. 1-4.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Yin, S, Venkataramanaiah, SK, Chen, GK, Krishnamurthy, R, Cao, Y, Chakrabarti, C & Seo, J 2018, Algorithm and hardware design of discrete-time spiking neural networks based on back propagation with binary activations. in 2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Proceedings. vol. 2018-January, Institute of Electrical and Electronics Engineers Inc., pp. 1-4, 2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017, Torino, Italy, 10/19/17. https://doi.org/10.1109/BIOCAS.2017.8325230
Yin S, Venkataramanaiah SK, Chen GK, Krishnamurthy R, Cao Y, Chakrabarti C et al. Algorithm and hardware design of discrete-time spiking neural networks based on back propagation with binary activations. In 2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Proceedings. Vol. 2018-January. Institute of Electrical and Electronics Engineers Inc. 2018. p. 1-4 https://doi.org/10.1109/BIOCAS.2017.8325230
Yin, Shihui ; Venkataramanaiah, Shreyas K. ; Chen, Gregory K. ; Krishnamurthy, Ram ; Cao, Yu ; Chakrabarti, Chaitali ; Seo, Jae-sun. / Algorithm and hardware design of discrete-time spiking neural networks based on back propagation with binary activations. 2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Proceedings. Vol. 2018-January Institute of Electrical and Electronics Engineers Inc., 2018. pp. 1-4
@inproceedings{9dc8ad9ba35d4b1ebe8ae9d29a100d38,
title = "Algorithm and hardware design of discrete-time spiking neural networks based on back propagation with binary activations",
abstract = "We present a new back propagation based training algorithm for discrete-time spiking neural networks (SNN). Inspired by recent deep learning algorithms on binarized neural networks, binary activation with a straight-through gradient estimator is used to model the leaky integrate-fire spiking neuron, overcoming the difficulty in training SNNs using back propagation. Two SNN training algorithms are proposed: (1) SNN with discontinuous integration, which is suitable for rate-coded input spikes, and (2) SNN with continuous integration, which is more general and can handle input spikes with temporal information. Neuromorphic hardware designed in 28nm CMOS exploits the spike sparsity and demonstrates high classification accuracy (>98{\%} on MNIST) and low energy (51.4-773 nJ/image).",
keywords = "back propagation, neuromorphic hardware, Spiking neural networks, straight-through estimator",
author = "Shihui Yin and Venkataramanaiah, {Shreyas K.} and Chen, {Gregory K.} and Ram Krishnamurthy and Yu Cao and Chaitali Chakrabarti and Jae-sun Seo",
year = "2018",
month = "3",
day = "23",
doi = "10.1109/BIOCAS.2017.8325230",
language = "English (US)",
volume = "2018-January",
pages = "1--4",
booktitle = "2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Proceedings",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - GEN

T1 - Algorithm and hardware design of discrete-time spiking neural networks based on back propagation with binary activations

AU - Yin, Shihui

AU - Venkataramanaiah, Shreyas K.

AU - Chen, Gregory K.

AU - Krishnamurthy, Ram

AU - Cao, Yu

AU - Chakrabarti, Chaitali

AU - Seo, Jae-sun

PY - 2018/3/23

Y1 - 2018/3/23

N2 - We present a new back propagation based training algorithm for discrete-time spiking neural networks (SNN). Inspired by recent deep learning algorithms on binarized neural networks, binary activation with a straight-through gradient estimator is used to model the leaky integrate-fire spiking neuron, overcoming the difficulty in training SNNs using back propagation. Two SNN training algorithms are proposed: (1) SNN with discontinuous integration, which is suitable for rate-coded input spikes, and (2) SNN with continuous integration, which is more general and can handle input spikes with temporal information. Neuromorphic hardware designed in 28nm CMOS exploits the spike sparsity and demonstrates high classification accuracy (>98% on MNIST) and low energy (51.4-773 nJ/image).

AB - We present a new back propagation based training algorithm for discrete-time spiking neural networks (SNN). Inspired by recent deep learning algorithms on binarized neural networks, binary activation with a straight-through gradient estimator is used to model the leaky integrate-fire spiking neuron, overcoming the difficulty in training SNNs using back propagation. Two SNN training algorithms are proposed: (1) SNN with discontinuous integration, which is suitable for rate-coded input spikes, and (2) SNN with continuous integration, which is more general and can handle input spikes with temporal information. Neuromorphic hardware designed in 28nm CMOS exploits the spike sparsity and demonstrates high classification accuracy (>98% on MNIST) and low energy (51.4-773 nJ/image).

KW - back propagation

KW - neuromorphic hardware

KW - Spiking neural networks

KW - straight-through estimator

UR - http://www.scopus.com/inward/record.url?scp=85049926656&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85049926656&partnerID=8YFLogxK

U2 - 10.1109/BIOCAS.2017.8325230

DO - 10.1109/BIOCAS.2017.8325230

M3 - Conference contribution

AN - SCOPUS:85049926656

VL - 2018-January

SP - 1

EP - 4

BT - 2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

ER -