Abstract

With the emergence of edge computing paradigm, many applications such as image recognition and augmented reality require to perform machine learning (ML) and artificial intelligence (AI) tasks on edge devices. Most AI and ML models are large and computational-heavy, whereas edge devices are usually equipped with limited computational and storage resources. Such models can be compressed and reduced for deployment on edge devices, but they may lose their capability and not perform well. Recent works used knowledge transfer techniques to transfer information from a large network (termed teacher) to a small one (termed student) in order to improve the performance of the latter. This approach seems to be promising for learning on edge devices, but a thorough investigation on its effectiveness is lacking. This paper provides an extensive study on the performance (in both accuracy and convergence speed) of knowledge transfer, considering different student architectures and different techniques for transferring knowledge from teacher to student. The results show that the performance of KT does vary by architectures and transfer techniques. A good performance improvement is obtained by transferring knowledge from both the intermediate layers and last layer of the teacher to a shallower student. But other architectures and transfer techniques do not fare so well and some of them even lead to negative performance impact.

Original languageEnglish (US)
Title of host publicationProceedings - 2018 IEEE International Conference on Edge Computing, EDGE 2018 - Part of the 2018 IEEE World Congress on Services
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages42-49
Number of pages8
ISBN (Electronic)9781538672389
DOIs
StatePublished - Sep 26 2018
Event2018 IEEE International Conference on Edge Computing, EDGE 2018 - San Francisco, United States
Duration: Jul 2 2018Jul 7 2018

Publication series

NameProceedings - 2018 IEEE International Conference on Edge Computing, EDGE 2018 - Part of the 2018 IEEE World Congress on Services

Other

Other2018 IEEE International Conference on Edge Computing, EDGE 2018
CountryUnited States
CitySan Francisco
Period7/2/187/7/18

Keywords

  • Cloud computing
  • Deep neural networks
  • Edge computing
  • Knowledge transfer

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications
  • Hardware and Architecture
  • Control and Optimization

Fingerprint Dive into the research topics of 'Are existing knowledge transfer techniques effective for deep learning with edge devices?'. Together they form a unique fingerprint.

  • Cite this

    Sharma, R., Biookaghazadeh, S., Li, B., & Zhao, M. (2018). Are existing knowledge transfer techniques effective for deep learning with edge devices? In Proceedings - 2018 IEEE International Conference on Edge Computing, EDGE 2018 - Part of the 2018 IEEE World Congress on Services (pp. 42-49). [8473375] (Proceedings - 2018 IEEE International Conference on Edge Computing, EDGE 2018 - Part of the 2018 IEEE World Congress on Services). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/EDGE.2018.00013