Algorithm and hardware design of discrete-time spiking neural networks based on back propagation with binary activations

Shihui Yin; Shreyas K. Venkataramanaiah; Gregory K. Chen; Ram Krishnamurthy; Yu Cao; Chaitali Chakrabarti; Jae-sun Seo

doi:10.1109/BIOCAS.2017.8325230

Algorithm and hardware design of discrete-time spiking neural networks based on back propagation with binary activations

Shihui Yin, Shreyas K. Venkataramanaiah, Gregory K. Chen, Ram Krishnamurthy, Yu Cao, Chaitali Chakrabarti, Jae-sun Seo

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

20 Scopus citations

Abstract

We present a new back propagation based training algorithm for discrete-time spiking neural networks (SNN). Inspired by recent deep learning algorithms on binarized neural networks, binary activation with a straight-through gradient estimator is used to model the leaky integrate-fire spiking neuron, overcoming the difficulty in training SNNs using back propagation. Two SNN training algorithms are proposed: (1) SNN with discontinuous integration, which is suitable for rate-coded input spikes, and (2) SNN with continuous integration, which is more general and can handle input spikes with temporal information. Neuromorphic hardware designed in 28nm CMOS exploits the spike sparsity and demonstrates high classification accuracy (>98% on MNIST) and low energy (51.4-773 nJ/image).

Original language	English (US)
Title of host publication	2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Proceedings
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1-4
Number of pages	4
ISBN (Electronic)	9781509058037
DOIs	https://doi.org/10.1109/BIOCAS.2017.8325230
State	Published - Mar 23 2018
Event	2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Torino, Italy Duration: Oct 19 2017 → Oct 21 2017

Publication series

Name	2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Proceedings
Volume	2018-January

Other

Other	2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017
Country/Territory	Italy
City	Torino
Period	10/19/17 → 10/21/17

Keywords

Spiking neural networks
back propagation
neuromorphic hardware
straight-through estimator

ASJC Scopus subject areas

Biomedical Engineering
Electrical and Electronic Engineering
Instrumentation

Access to Document

10.1109/BIOCAS.2017.8325230

Cite this

Yin, S., Venkataramanaiah, S. K., Chen, G. K., Krishnamurthy, R., Cao, Y., Chakrabarti, C., & Seo, J. (2018). Algorithm and hardware design of discrete-time spiking neural networks based on back propagation with binary activations. In 2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Proceedings (pp. 1-4). (2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Proceedings; Vol. 2018-January). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/BIOCAS.2017.8325230

Algorithm and hardware design of discrete-time spiking neural networks based on back propagation with binary activations. / Yin, Shihui; Venkataramanaiah, Shreyas K.; Chen, Gregory K.; Krishnamurthy, Ram; Cao, Yu ; Chakrabarti, Chaitali ; Seo, Jae-sun.

2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2018. p. 1-4 (2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Proceedings; Vol. 2018-January).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Yin, S, Venkataramanaiah, SK, Chen, GK, Krishnamurthy, R, Cao, Y , Chakrabarti, C & Seo, J 2018, Algorithm and hardware design of discrete-time spiking neural networks based on back propagation with binary activations. in 2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Proceedings. 2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Proceedings, vol. 2018-January, Institute of Electrical and Electronics Engineers Inc., pp. 1-4, 2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017, Torino, Italy, 10/19/17. https://doi.org/10.1109/BIOCAS.2017.8325230

Yin S, Venkataramanaiah SK, Chen GK, Krishnamurthy R, Cao Y , Chakrabarti C et al. Algorithm and hardware design of discrete-time spiking neural networks based on back propagation with binary activations. In 2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Proceedings. Institute of Electrical and Electronics Engineers Inc. 2018. p. 1-4. (2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Proceedings). https://doi.org/10.1109/BIOCAS.2017.8325230

Yin, Shihui ; Venkataramanaiah, Shreyas K. ; Chen, Gregory K. ; Krishnamurthy, Ram ; Cao, Yu ; Chakrabarti, Chaitali ; Seo, Jae-sun. / Algorithm and hardware design of discrete-time spiking neural networks based on back propagation with binary activations. 2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 1-4 (2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Proceedings).

@inproceedings{9dc8ad9ba35d4b1ebe8ae9d29a100d38,

title = "Algorithm and hardware design of discrete-time spiking neural networks based on back propagation with binary activations",

abstract = "We present a new back propagation based training algorithm for discrete-time spiking neural networks (SNN). Inspired by recent deep learning algorithms on binarized neural networks, binary activation with a straight-through gradient estimator is used to model the leaky integrate-fire spiking neuron, overcoming the difficulty in training SNNs using back propagation. Two SNN training algorithms are proposed: (1) SNN with discontinuous integration, which is suitable for rate-coded input spikes, and (2) SNN with continuous integration, which is more general and can handle input spikes with temporal information. Neuromorphic hardware designed in 28nm CMOS exploits the spike sparsity and demonstrates high classification accuracy (>98% on MNIST) and low energy (51.4-773 nJ/image).",

keywords = "Spiking neural networks, back propagation, neuromorphic hardware, straight-through estimator",

author = "Shihui Yin and Venkataramanaiah, {Shreyas K.} and Chen, {Gregory K.} and Ram Krishnamurthy and Yu Cao and Chaitali Chakrabarti and Jae-sun Seo",

note = "Funding Information: This work was supported in part by the NSF grant 1652866 and Intel Labs. Publisher Copyright: {\textcopyright} 2017 IEEE.; 2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 ; Conference date: 19-10-2017 Through 21-10-2017",

year = "2018",

month = mar,

day = "23",

doi = "10.1109/BIOCAS.2017.8325230",

language = "English (US)",

series = "2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Proceedings",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1--4",

booktitle = "2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Proceedings",

}

TY - GEN

T1 - Algorithm and hardware design of discrete-time spiking neural networks based on back propagation with binary activations

AU - Yin, Shihui

AU - Venkataramanaiah, Shreyas K.

AU - Chen, Gregory K.

AU - Krishnamurthy, Ram

AU - Cao, Yu

AU - Chakrabarti, Chaitali

AU - Seo, Jae-sun

PY - 2018/3/23

Y1 - 2018/3/23

N2 - We present a new back propagation based training algorithm for discrete-time spiking neural networks (SNN). Inspired by recent deep learning algorithms on binarized neural networks, binary activation with a straight-through gradient estimator is used to model the leaky integrate-fire spiking neuron, overcoming the difficulty in training SNNs using back propagation. Two SNN training algorithms are proposed: (1) SNN with discontinuous integration, which is suitable for rate-coded input spikes, and (2) SNN with continuous integration, which is more general and can handle input spikes with temporal information. Neuromorphic hardware designed in 28nm CMOS exploits the spike sparsity and demonstrates high classification accuracy (>98% on MNIST) and low energy (51.4-773 nJ/image).

AB - We present a new back propagation based training algorithm for discrete-time spiking neural networks (SNN). Inspired by recent deep learning algorithms on binarized neural networks, binary activation with a straight-through gradient estimator is used to model the leaky integrate-fire spiking neuron, overcoming the difficulty in training SNNs using back propagation. Two SNN training algorithms are proposed: (1) SNN with discontinuous integration, which is suitable for rate-coded input spikes, and (2) SNN with continuous integration, which is more general and can handle input spikes with temporal information. Neuromorphic hardware designed in 28nm CMOS exploits the spike sparsity and demonstrates high classification accuracy (>98% on MNIST) and low energy (51.4-773 nJ/image).

KW - Spiking neural networks

KW - back propagation

KW - neuromorphic hardware

KW - straight-through estimator

UR - http://www.scopus.com/inward/record.url?scp=85049926656&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85049926656&partnerID=8YFLogxK

U2 - 10.1109/BIOCAS.2017.8325230

DO - 10.1109/BIOCAS.2017.8325230

M3 - Conference contribution

AN - SCOPUS:85049926656

T3 - 2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Proceedings

SP - 1

EP - 4

BT - 2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2017 IEEE Biomedical Circuits and Systems Conference, BioCAS 2017

Y2 - 19 October 2017 through 21 October 2017

ER -

Algorithm and hardware design of discrete-time spiking neural networks based on back propagation with binary activations

Abstract

Publication series

Other

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this