Joint Optimization of Quantization and Structured Sparsity for Compressed Deep Neural Networks

Gaurav Srivastava, Deepak Kadetotad, Shihui Yin, Visar Berisha, Chaitali Chakrabarti, Jae-sun Seo

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The usage of Deep Neural Networks (DNN) on resource-constrained edge devices has been limited due to their high computation and large memory requirement. In this work, we propose an algorithm to compress DNNs by jointly optimizing structured sparsity and quantization constraints in a single DNN training framework. The proposed algorithm has been extensively validated on high/low capacity DNNs and wide/deep sparse DNNs. Further, we perform Pareto-optimal analysis to extract optimal DNN models from a large set of trained DNN models. The optimal structurally-compressed DNN model achieves ~50X weight memory reduction without test accuracy degradation, compared to floating-point uncompressed DNN.

Original languageEnglish (US)
Title of host publication2019 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2019 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1393-1397
Number of pages5
ISBN (Electronic)9781479981311
DOIs
StatePublished - May 1 2019
Event44th IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2019 - Brighton, United Kingdom
Duration: May 12 2019May 17 2019

Publication series

NameICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
Volume2019-May
ISSN (Print)1520-6149

Conference

Conference44th IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2019
CountryUnited Kingdom
CityBrighton
Period5/12/195/17/19

Fingerprint

Data storage equipment
Deep neural networks
Degradation

ASJC Scopus subject areas

  • Software
  • Signal Processing
  • Electrical and Electronic Engineering

Cite this

Srivastava, G., Kadetotad, D., Yin, S., Berisha, V., Chakrabarti, C., & Seo, J. (2019). Joint Optimization of Quantization and Structured Sparsity for Compressed Deep Neural Networks. In 2019 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2019 - Proceedings (pp. 1393-1397). [8682791] (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings; Vol. 2019-May). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICASSP.2019.8682791

Joint Optimization of Quantization and Structured Sparsity for Compressed Deep Neural Networks. / Srivastava, Gaurav; Kadetotad, Deepak; Yin, Shihui; Berisha, Visar; Chakrabarti, Chaitali; Seo, Jae-sun.

2019 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2019 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2019. p. 1393-1397 8682791 (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings; Vol. 2019-May).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Srivastava, G, Kadetotad, D, Yin, S, Berisha, V, Chakrabarti, C & Seo, J 2019, Joint Optimization of Quantization and Structured Sparsity for Compressed Deep Neural Networks. in 2019 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2019 - Proceedings., 8682791, ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, vol. 2019-May, Institute of Electrical and Electronics Engineers Inc., pp. 1393-1397, 44th IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2019, Brighton, United Kingdom, 5/12/19. https://doi.org/10.1109/ICASSP.2019.8682791
Srivastava G, Kadetotad D, Yin S, Berisha V, Chakrabarti C, Seo J. Joint Optimization of Quantization and Structured Sparsity for Compressed Deep Neural Networks. In 2019 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2019 - Proceedings. Institute of Electrical and Electronics Engineers Inc. 2019. p. 1393-1397. 8682791. (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings). https://doi.org/10.1109/ICASSP.2019.8682791
Srivastava, Gaurav ; Kadetotad, Deepak ; Yin, Shihui ; Berisha, Visar ; Chakrabarti, Chaitali ; Seo, Jae-sun. / Joint Optimization of Quantization and Structured Sparsity for Compressed Deep Neural Networks. 2019 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2019 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 1393-1397 (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings).
@inproceedings{1bbab8ad01b344a8af91489ec8c68027,
title = "Joint Optimization of Quantization and Structured Sparsity for Compressed Deep Neural Networks",
abstract = "The usage of Deep Neural Networks (DNN) on resource-constrained edge devices has been limited due to their high computation and large memory requirement. In this work, we propose an algorithm to compress DNNs by jointly optimizing structured sparsity and quantization constraints in a single DNN training framework. The proposed algorithm has been extensively validated on high/low capacity DNNs and wide/deep sparse DNNs. Further, we perform Pareto-optimal analysis to extract optimal DNN models from a large set of trained DNN models. The optimal structurally-compressed DNN model achieves ~50X weight memory reduction without test accuracy degradation, compared to floating-point uncompressed DNN.",
author = "Gaurav Srivastava and Deepak Kadetotad and Shihui Yin and Visar Berisha and Chaitali Chakrabarti and Jae-sun Seo",
year = "2019",
month = "5",
day = "1",
doi = "10.1109/ICASSP.2019.8682791",
language = "English (US)",
series = "ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "1393--1397",
booktitle = "2019 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2019 - Proceedings",

}

TY - GEN

T1 - Joint Optimization of Quantization and Structured Sparsity for Compressed Deep Neural Networks

AU - Srivastava, Gaurav

AU - Kadetotad, Deepak

AU - Yin, Shihui

AU - Berisha, Visar

AU - Chakrabarti, Chaitali

AU - Seo, Jae-sun

PY - 2019/5/1

Y1 - 2019/5/1

N2 - The usage of Deep Neural Networks (DNN) on resource-constrained edge devices has been limited due to their high computation and large memory requirement. In this work, we propose an algorithm to compress DNNs by jointly optimizing structured sparsity and quantization constraints in a single DNN training framework. The proposed algorithm has been extensively validated on high/low capacity DNNs and wide/deep sparse DNNs. Further, we perform Pareto-optimal analysis to extract optimal DNN models from a large set of trained DNN models. The optimal structurally-compressed DNN model achieves ~50X weight memory reduction without test accuracy degradation, compared to floating-point uncompressed DNN.

AB - The usage of Deep Neural Networks (DNN) on resource-constrained edge devices has been limited due to their high computation and large memory requirement. In this work, we propose an algorithm to compress DNNs by jointly optimizing structured sparsity and quantization constraints in a single DNN training framework. The proposed algorithm has been extensively validated on high/low capacity DNNs and wide/deep sparse DNNs. Further, we perform Pareto-optimal analysis to extract optimal DNN models from a large set of trained DNN models. The optimal structurally-compressed DNN model achieves ~50X weight memory reduction without test accuracy degradation, compared to floating-point uncompressed DNN.

UR - http://www.scopus.com/inward/record.url?scp=85068959788&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85068959788&partnerID=8YFLogxK

U2 - 10.1109/ICASSP.2019.8682791

DO - 10.1109/ICASSP.2019.8682791

M3 - Conference contribution

T3 - ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

SP - 1393

EP - 1397

BT - 2019 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2019 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

ER -