Are existing knowledge transfer techniques effective for deep learning on edge devices?

Ragini Sharma, Saman Biookaghazadeh, Ming Zhao

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

With the emergence of edge computing paradigm, many applications such as image recognition and augmented reality require to perform machine learning (ML) and artificial intelligence (AI) tasks on edge devices. Most AI and ML models are large and computational-heavy, whereas edge devices are usually equipped with limited computational and storage resources. Such models can be compressed and reduced for deployment on edge devices, but they may loose their capability and not perform well. Recent works used knowledge transfer techniques to transfer information from a large network (termed teacher) to a small one (termed student) in order to improve the performance of the latter. This approach seems to be promising for learning on edge devices, but a thorough investigation on its effectiveness is lacking. This paper provides an extensive study on the performance (in both accuracy and convergence speed) of knowledge transfer, considering different student architectures and different techniques for transferring knowledge from teacher to student. The results show that the performance of KT does vary by architectures and transfer techniques. A good performance improvement is obtained by transferring knowledge from both the intermediate layers and last layer of the teacher to a shallower student. But other architectures and transfer techniques do not fare so well and some of them even lead to negative performance impact.

Original languageEnglish (US)
Title of host publicationHPDC 2018 - Proceedings of The 27th International Symposium on High-Performance Parallel and Distributed Computing Posters/Doctoral Consortium
PublisherAssociation for Computing Machinery, Inc
Pages15-16
Number of pages2
ISBN (Electronic)9781450358996
DOIs
StatePublished - Jun 11 2018
Event27th ACM International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2018 - Tempe, United States
Duration: Jun 11 2018 → …

Other

Other27th ACM International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2018
CountryUnited States
CityTempe
Period6/11/18 → …

Fingerprint

Students
Artificial intelligence
Learning systems
Image recognition
Augmented reality
Deep learning

Keywords

  • Deep learning
  • Edge computing
  • Knowledge transfer
  • Neural networks

ASJC Scopus subject areas

  • Computer Science Applications
  • Software
  • Computational Theory and Mathematics

Cite this

Sharma, R., Biookaghazadeh, S., & Zhao, M. (2018). Are existing knowledge transfer techniques effective for deep learning on edge devices? In HPDC 2018 - Proceedings of The 27th International Symposium on High-Performance Parallel and Distributed Computing Posters/Doctoral Consortium (pp. 15-16). Association for Computing Machinery, Inc. https://doi.org/10.1145/3220192.3220459

Are existing knowledge transfer techniques effective for deep learning on edge devices? / Sharma, Ragini; Biookaghazadeh, Saman; Zhao, Ming.

HPDC 2018 - Proceedings of The 27th International Symposium on High-Performance Parallel and Distributed Computing Posters/Doctoral Consortium. Association for Computing Machinery, Inc, 2018. p. 15-16.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Sharma, R, Biookaghazadeh, S & Zhao, M 2018, Are existing knowledge transfer techniques effective for deep learning on edge devices? in HPDC 2018 - Proceedings of The 27th International Symposium on High-Performance Parallel and Distributed Computing Posters/Doctoral Consortium. Association for Computing Machinery, Inc, pp. 15-16, 27th ACM International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2018, Tempe, United States, 6/11/18. https://doi.org/10.1145/3220192.3220459
Sharma R, Biookaghazadeh S, Zhao M. Are existing knowledge transfer techniques effective for deep learning on edge devices? In HPDC 2018 - Proceedings of The 27th International Symposium on High-Performance Parallel and Distributed Computing Posters/Doctoral Consortium. Association for Computing Machinery, Inc. 2018. p. 15-16 https://doi.org/10.1145/3220192.3220459
Sharma, Ragini ; Biookaghazadeh, Saman ; Zhao, Ming. / Are existing knowledge transfer techniques effective for deep learning on edge devices?. HPDC 2018 - Proceedings of The 27th International Symposium on High-Performance Parallel and Distributed Computing Posters/Doctoral Consortium. Association for Computing Machinery, Inc, 2018. pp. 15-16
@inproceedings{758d2d2478694799be0bb2ad8ac8f14e,
title = "Are existing knowledge transfer techniques effective for deep learning on edge devices?",
abstract = "With the emergence of edge computing paradigm, many applications such as image recognition and augmented reality require to perform machine learning (ML) and artificial intelligence (AI) tasks on edge devices. Most AI and ML models are large and computational-heavy, whereas edge devices are usually equipped with limited computational and storage resources. Such models can be compressed and reduced for deployment on edge devices, but they may loose their capability and not perform well. Recent works used knowledge transfer techniques to transfer information from a large network (termed teacher) to a small one (termed student) in order to improve the performance of the latter. This approach seems to be promising for learning on edge devices, but a thorough investigation on its effectiveness is lacking. This paper provides an extensive study on the performance (in both accuracy and convergence speed) of knowledge transfer, considering different student architectures and different techniques for transferring knowledge from teacher to student. The results show that the performance of KT does vary by architectures and transfer techniques. A good performance improvement is obtained by transferring knowledge from both the intermediate layers and last layer of the teacher to a shallower student. But other architectures and transfer techniques do not fare so well and some of them even lead to negative performance impact.",
keywords = "Deep learning, Edge computing, Knowledge transfer, Neural networks",
author = "Ragini Sharma and Saman Biookaghazadeh and Ming Zhao",
year = "2018",
month = "6",
day = "11",
doi = "10.1145/3220192.3220459",
language = "English (US)",
pages = "15--16",
booktitle = "HPDC 2018 - Proceedings of The 27th International Symposium on High-Performance Parallel and Distributed Computing Posters/Doctoral Consortium",
publisher = "Association for Computing Machinery, Inc",

}

TY - GEN

T1 - Are existing knowledge transfer techniques effective for deep learning on edge devices?

AU - Sharma, Ragini

AU - Biookaghazadeh, Saman

AU - Zhao, Ming

PY - 2018/6/11

Y1 - 2018/6/11

N2 - With the emergence of edge computing paradigm, many applications such as image recognition and augmented reality require to perform machine learning (ML) and artificial intelligence (AI) tasks on edge devices. Most AI and ML models are large and computational-heavy, whereas edge devices are usually equipped with limited computational and storage resources. Such models can be compressed and reduced for deployment on edge devices, but they may loose their capability and not perform well. Recent works used knowledge transfer techniques to transfer information from a large network (termed teacher) to a small one (termed student) in order to improve the performance of the latter. This approach seems to be promising for learning on edge devices, but a thorough investigation on its effectiveness is lacking. This paper provides an extensive study on the performance (in both accuracy and convergence speed) of knowledge transfer, considering different student architectures and different techniques for transferring knowledge from teacher to student. The results show that the performance of KT does vary by architectures and transfer techniques. A good performance improvement is obtained by transferring knowledge from both the intermediate layers and last layer of the teacher to a shallower student. But other architectures and transfer techniques do not fare so well and some of them even lead to negative performance impact.

AB - With the emergence of edge computing paradigm, many applications such as image recognition and augmented reality require to perform machine learning (ML) and artificial intelligence (AI) tasks on edge devices. Most AI and ML models are large and computational-heavy, whereas edge devices are usually equipped with limited computational and storage resources. Such models can be compressed and reduced for deployment on edge devices, but they may loose their capability and not perform well. Recent works used knowledge transfer techniques to transfer information from a large network (termed teacher) to a small one (termed student) in order to improve the performance of the latter. This approach seems to be promising for learning on edge devices, but a thorough investigation on its effectiveness is lacking. This paper provides an extensive study on the performance (in both accuracy and convergence speed) of knowledge transfer, considering different student architectures and different techniques for transferring knowledge from teacher to student. The results show that the performance of KT does vary by architectures and transfer techniques. A good performance improvement is obtained by transferring knowledge from both the intermediate layers and last layer of the teacher to a shallower student. But other architectures and transfer techniques do not fare so well and some of them even lead to negative performance impact.

KW - Deep learning

KW - Edge computing

KW - Knowledge transfer

KW - Neural networks

UR - http://www.scopus.com/inward/record.url?scp=85050115844&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85050115844&partnerID=8YFLogxK

U2 - 10.1145/3220192.3220459

DO - 10.1145/3220192.3220459

M3 - Conference contribution

SP - 15

EP - 16

BT - HPDC 2018 - Proceedings of The 27th International Symposium on High-Performance Parallel and Distributed Computing Posters/Doctoral Consortium

PB - Association for Computing Machinery, Inc

ER -