Abstract

Recurrent neural networks (RNNs) provide excellent performance on applications with sequential data such as speech recognition. On-chip implementation of RNNs is difficult due to the significantly large number of parameters and computations. In this work, we first present a training method for LSTM model for language modeling on Penn Treebank dataset with binary weights and multi-bit activations and then map it onto a fully parallel RRAM array architecture ("XNOR-RRAM"). An energy-efficient XNOR-RRAM array based system for LSTM RNN is implemented and benchmarked on Penn Treebank dataset. Our results show that 4-bit activation precision can provide a near-optimal perplexity of 115.3 with an estimated energy-efficiency of 27 TOPS/W.

Original languageEnglish (US)
Title of host publicationProceedings of the IEEE Workshop on Signal Processing Systems, SiPS 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages13-18
Number of pages6
ISBN (Electronic)9781538663189
DOIs
StatePublished - Dec 31 2018
Event2018 IEEE Workshop on Signal Processing Systems, SiPS 2018 - Cape Town, South Africa
Duration: Oct 21 2018Oct 24 2018

Publication series

NameIEEE Workshop on Signal Processing Systems, SiPS: Design and Implementation
Volume2018-October
ISSN (Print)1520-6130

Conference

Conference2018 IEEE Workshop on Signal Processing Systems, SiPS 2018
CountrySouth Africa
CityCape Town
Period10/21/1810/24/18

Fingerprint

Recurrent neural networks
XNOR
Recurrent Neural Networks
Energy Efficient
Activation
Chemical activation
Language Modeling
Speech Recognition
Speech recognition
Energy Efficiency
Energy efficiency
Chip
Binary
Architecture
RRAM
Model

ASJC Scopus subject areas

  • Electrical and Electronic Engineering
  • Signal Processing
  • Applied Mathematics
  • Hardware and Architecture

Cite this

Yin, S., Sun, X., Yu, S., Seo, J., & Chakrabarti, C. (2018). A Parallel RRAM Synaptic Array Architecture for Energy-Efficient Recurrent Neural Networks. In Proceedings of the IEEE Workshop on Signal Processing Systems, SiPS 2018 (pp. 13-18). [8598445] (IEEE Workshop on Signal Processing Systems, SiPS: Design and Implementation; Vol. 2018-October). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/SiPS.2018.8598445

A Parallel RRAM Synaptic Array Architecture for Energy-Efficient Recurrent Neural Networks. / Yin, Shihui; Sun, Xiaoyu; Yu, Shimeng; Seo, Jae-sun; Chakrabarti, Chaitali.

Proceedings of the IEEE Workshop on Signal Processing Systems, SiPS 2018. Institute of Electrical and Electronics Engineers Inc., 2018. p. 13-18 8598445 (IEEE Workshop on Signal Processing Systems, SiPS: Design and Implementation; Vol. 2018-October).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Yin, S, Sun, X, Yu, S, Seo, J & Chakrabarti, C 2018, A Parallel RRAM Synaptic Array Architecture for Energy-Efficient Recurrent Neural Networks. in Proceedings of the IEEE Workshop on Signal Processing Systems, SiPS 2018., 8598445, IEEE Workshop on Signal Processing Systems, SiPS: Design and Implementation, vol. 2018-October, Institute of Electrical and Electronics Engineers Inc., pp. 13-18, 2018 IEEE Workshop on Signal Processing Systems, SiPS 2018, Cape Town, South Africa, 10/21/18. https://doi.org/10.1109/SiPS.2018.8598445
Yin S, Sun X, Yu S, Seo J, Chakrabarti C. A Parallel RRAM Synaptic Array Architecture for Energy-Efficient Recurrent Neural Networks. In Proceedings of the IEEE Workshop on Signal Processing Systems, SiPS 2018. Institute of Electrical and Electronics Engineers Inc. 2018. p. 13-18. 8598445. (IEEE Workshop on Signal Processing Systems, SiPS: Design and Implementation). https://doi.org/10.1109/SiPS.2018.8598445
Yin, Shihui ; Sun, Xiaoyu ; Yu, Shimeng ; Seo, Jae-sun ; Chakrabarti, Chaitali. / A Parallel RRAM Synaptic Array Architecture for Energy-Efficient Recurrent Neural Networks. Proceedings of the IEEE Workshop on Signal Processing Systems, SiPS 2018. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 13-18 (IEEE Workshop on Signal Processing Systems, SiPS: Design and Implementation).
@inproceedings{d407c20e9a3a4996ad9be827d150790c,
title = "A Parallel RRAM Synaptic Array Architecture for Energy-Efficient Recurrent Neural Networks",
abstract = "Recurrent neural networks (RNNs) provide excellent performance on applications with sequential data such as speech recognition. On-chip implementation of RNNs is difficult due to the significantly large number of parameters and computations. In this work, we first present a training method for LSTM model for language modeling on Penn Treebank dataset with binary weights and multi-bit activations and then map it onto a fully parallel RRAM array architecture ({"}XNOR-RRAM{"}). An energy-efficient XNOR-RRAM array based system for LSTM RNN is implemented and benchmarked on Penn Treebank dataset. Our results show that 4-bit activation precision can provide a near-optimal perplexity of 115.3 with an estimated energy-efficiency of 27 TOPS/W.",
author = "Shihui Yin and Xiaoyu Sun and Shimeng Yu and Jae-sun Seo and Chaitali Chakrabarti",
year = "2018",
month = "12",
day = "31",
doi = "10.1109/SiPS.2018.8598445",
language = "English (US)",
series = "IEEE Workshop on Signal Processing Systems, SiPS: Design and Implementation",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "13--18",
booktitle = "Proceedings of the IEEE Workshop on Signal Processing Systems, SiPS 2018",

}

TY - GEN

T1 - A Parallel RRAM Synaptic Array Architecture for Energy-Efficient Recurrent Neural Networks

AU - Yin, Shihui

AU - Sun, Xiaoyu

AU - Yu, Shimeng

AU - Seo, Jae-sun

AU - Chakrabarti, Chaitali

PY - 2018/12/31

Y1 - 2018/12/31

N2 - Recurrent neural networks (RNNs) provide excellent performance on applications with sequential data such as speech recognition. On-chip implementation of RNNs is difficult due to the significantly large number of parameters and computations. In this work, we first present a training method for LSTM model for language modeling on Penn Treebank dataset with binary weights and multi-bit activations and then map it onto a fully parallel RRAM array architecture ("XNOR-RRAM"). An energy-efficient XNOR-RRAM array based system for LSTM RNN is implemented and benchmarked on Penn Treebank dataset. Our results show that 4-bit activation precision can provide a near-optimal perplexity of 115.3 with an estimated energy-efficiency of 27 TOPS/W.

AB - Recurrent neural networks (RNNs) provide excellent performance on applications with sequential data such as speech recognition. On-chip implementation of RNNs is difficult due to the significantly large number of parameters and computations. In this work, we first present a training method for LSTM model for language modeling on Penn Treebank dataset with binary weights and multi-bit activations and then map it onto a fully parallel RRAM array architecture ("XNOR-RRAM"). An energy-efficient XNOR-RRAM array based system for LSTM RNN is implemented and benchmarked on Penn Treebank dataset. Our results show that 4-bit activation precision can provide a near-optimal perplexity of 115.3 with an estimated energy-efficiency of 27 TOPS/W.

UR - http://www.scopus.com/inward/record.url?scp=85061355856&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85061355856&partnerID=8YFLogxK

U2 - 10.1109/SiPS.2018.8598445

DO - 10.1109/SiPS.2018.8598445

M3 - Conference contribution

T3 - IEEE Workshop on Signal Processing Systems, SiPS: Design and Implementation

SP - 13

EP - 18

BT - Proceedings of the IEEE Workshop on Signal Processing Systems, SiPS 2018

PB - Institute of Electrical and Electronics Engineers Inc.

ER -