Abstract

With the emergence of edge computing paradigm, many applications such as image recognition and augmented reality require to perform machine learning (ML) and artificial intelligence (AI) tasks on edge devices. Most AI and ML models are large and computational-heavy, whereas edge devices are usually equipped with limited computational and storage resources. Such models can be compressed and reduced for deployment on edge devices, but they may lose their capability and not perform well. Recent works used knowledge transfer techniques to transfer information from a large network (termed teacher) to a small one (termed student) in order to improve the performance of the latter. This approach seems to be promising for learning on edge devices, but a thorough investigation on its effectiveness is lacking. This paper provides an extensive study on the performance (in both accuracy and convergence speed) of knowledge transfer, considering different student architectures and different techniques for transferring knowledge from teacher to student. The results show that the performance of KT does vary by architectures and transfer techniques. A good performance improvement is obtained by transferring knowledge from both the intermediate layers and last layer of the teacher to a shallower student. But other architectures and transfer techniques do not fare so well and some of them even lead to negative performance impact.

Original languageEnglish (US)
Title of host publicationProceedings - 2018 IEEE International Conference on Edge Computing, EDGE 2018 - Part of the 2018 IEEE World Congress on Services
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages42-49
Number of pages8
ISBN (Electronic)9781538672389
DOIs
StatePublished - Sep 26 2018
Event2018 IEEE International Conference on Edge Computing, EDGE 2018 - San Francisco, United States
Duration: Jul 2 2018Jul 7 2018

Other

Other2018 IEEE International Conference on Edge Computing, EDGE 2018
CountryUnited States
CitySan Francisco
Period7/2/187/7/18

Fingerprint

Knowledge Transfer
Students
Artificial intelligence
Learning systems
Artificial Intelligence
Machine Learning
Image recognition
Augmented reality
Image Recognition
Information Transfer
Augmented Reality
Speed of Convergence
Paradigm
Learning
Deep learning
Vary
Resources
Computing
Model
Architecture

Keywords

  • Cloud computing
  • Deep neural networks
  • Edge computing
  • Knowledge transfer

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications
  • Hardware and Architecture
  • Control and Optimization

Cite this

Sharma, R., Biookaghazadeh, S., Li, B., & Zhao, M. (2018). Are existing knowledge transfer techniques effective for deep learning with edge devices? In Proceedings - 2018 IEEE International Conference on Edge Computing, EDGE 2018 - Part of the 2018 IEEE World Congress on Services (pp. 42-49). [8473375] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/EDGE.2018.00013

Are existing knowledge transfer techniques effective for deep learning with edge devices? / Sharma, Ragini; Biookaghazadeh, Saman; Li, Baoxin; Zhao, Ming.

Proceedings - 2018 IEEE International Conference on Edge Computing, EDGE 2018 - Part of the 2018 IEEE World Congress on Services. Institute of Electrical and Electronics Engineers Inc., 2018. p. 42-49 8473375.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Sharma, R, Biookaghazadeh, S, Li, B & Zhao, M 2018, Are existing knowledge transfer techniques effective for deep learning with edge devices? in Proceedings - 2018 IEEE International Conference on Edge Computing, EDGE 2018 - Part of the 2018 IEEE World Congress on Services., 8473375, Institute of Electrical and Electronics Engineers Inc., pp. 42-49, 2018 IEEE International Conference on Edge Computing, EDGE 2018, San Francisco, United States, 7/2/18. https://doi.org/10.1109/EDGE.2018.00013
Sharma R, Biookaghazadeh S, Li B, Zhao M. Are existing knowledge transfer techniques effective for deep learning with edge devices? In Proceedings - 2018 IEEE International Conference on Edge Computing, EDGE 2018 - Part of the 2018 IEEE World Congress on Services. Institute of Electrical and Electronics Engineers Inc. 2018. p. 42-49. 8473375 https://doi.org/10.1109/EDGE.2018.00013
Sharma, Ragini ; Biookaghazadeh, Saman ; Li, Baoxin ; Zhao, Ming. / Are existing knowledge transfer techniques effective for deep learning with edge devices?. Proceedings - 2018 IEEE International Conference on Edge Computing, EDGE 2018 - Part of the 2018 IEEE World Congress on Services. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 42-49
@inproceedings{a658085d32184764b0ff46f130262d69,
title = "Are existing knowledge transfer techniques effective for deep learning with edge devices?",
abstract = "With the emergence of edge computing paradigm, many applications such as image recognition and augmented reality require to perform machine learning (ML) and artificial intelligence (AI) tasks on edge devices. Most AI and ML models are large and computational-heavy, whereas edge devices are usually equipped with limited computational and storage resources. Such models can be compressed and reduced for deployment on edge devices, but they may lose their capability and not perform well. Recent works used knowledge transfer techniques to transfer information from a large network (termed teacher) to a small one (termed student) in order to improve the performance of the latter. This approach seems to be promising for learning on edge devices, but a thorough investigation on its effectiveness is lacking. This paper provides an extensive study on the performance (in both accuracy and convergence speed) of knowledge transfer, considering different student architectures and different techniques for transferring knowledge from teacher to student. The results show that the performance of KT does vary by architectures and transfer techniques. A good performance improvement is obtained by transferring knowledge from both the intermediate layers and last layer of the teacher to a shallower student. But other architectures and transfer techniques do not fare so well and some of them even lead to negative performance impact.",
keywords = "Cloud computing, Deep neural networks, Edge computing, Knowledge transfer",
author = "Ragini Sharma and Saman Biookaghazadeh and Baoxin Li and Ming Zhao",
year = "2018",
month = "9",
day = "26",
doi = "10.1109/EDGE.2018.00013",
language = "English (US)",
pages = "42--49",
booktitle = "Proceedings - 2018 IEEE International Conference on Edge Computing, EDGE 2018 - Part of the 2018 IEEE World Congress on Services",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - GEN

T1 - Are existing knowledge transfer techniques effective for deep learning with edge devices?

AU - Sharma, Ragini

AU - Biookaghazadeh, Saman

AU - Li, Baoxin

AU - Zhao, Ming

PY - 2018/9/26

Y1 - 2018/9/26

N2 - With the emergence of edge computing paradigm, many applications such as image recognition and augmented reality require to perform machine learning (ML) and artificial intelligence (AI) tasks on edge devices. Most AI and ML models are large and computational-heavy, whereas edge devices are usually equipped with limited computational and storage resources. Such models can be compressed and reduced for deployment on edge devices, but they may lose their capability and not perform well. Recent works used knowledge transfer techniques to transfer information from a large network (termed teacher) to a small one (termed student) in order to improve the performance of the latter. This approach seems to be promising for learning on edge devices, but a thorough investigation on its effectiveness is lacking. This paper provides an extensive study on the performance (in both accuracy and convergence speed) of knowledge transfer, considering different student architectures and different techniques for transferring knowledge from teacher to student. The results show that the performance of KT does vary by architectures and transfer techniques. A good performance improvement is obtained by transferring knowledge from both the intermediate layers and last layer of the teacher to a shallower student. But other architectures and transfer techniques do not fare so well and some of them even lead to negative performance impact.

AB - With the emergence of edge computing paradigm, many applications such as image recognition and augmented reality require to perform machine learning (ML) and artificial intelligence (AI) tasks on edge devices. Most AI and ML models are large and computational-heavy, whereas edge devices are usually equipped with limited computational and storage resources. Such models can be compressed and reduced for deployment on edge devices, but they may lose their capability and not perform well. Recent works used knowledge transfer techniques to transfer information from a large network (termed teacher) to a small one (termed student) in order to improve the performance of the latter. This approach seems to be promising for learning on edge devices, but a thorough investigation on its effectiveness is lacking. This paper provides an extensive study on the performance (in both accuracy and convergence speed) of knowledge transfer, considering different student architectures and different techniques for transferring knowledge from teacher to student. The results show that the performance of KT does vary by architectures and transfer techniques. A good performance improvement is obtained by transferring knowledge from both the intermediate layers and last layer of the teacher to a shallower student. But other architectures and transfer techniques do not fare so well and some of them even lead to negative performance impact.

KW - Cloud computing

KW - Deep neural networks

KW - Edge computing

KW - Knowledge transfer

UR - http://www.scopus.com/inward/record.url?scp=85055667496&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85055667496&partnerID=8YFLogxK

U2 - 10.1109/EDGE.2018.00013

DO - 10.1109/EDGE.2018.00013

M3 - Conference contribution

SP - 42

EP - 49

BT - Proceedings - 2018 IEEE International Conference on Edge Computing, EDGE 2018 - Part of the 2018 IEEE World Congress on Services

PB - Institute of Electrical and Electronics Engineers Inc.

ER -