TY - GEN
T1 - Are existing knowledge transfer techniques effective for deep learning with edge devices?
AU - Sharma, Ragini
AU - Biookaghazadeh, Saman
AU - Li, Baoxin
AU - Zhao, Ming
N1 - Publisher Copyright:
© 2018 IEEE.
PY - 2018/9/26
Y1 - 2018/9/26
N2 - With the emergence of edge computing paradigm, many applications such as image recognition and augmented reality require to perform machine learning (ML) and artificial intelligence (AI) tasks on edge devices. Most AI and ML models are large and computational-heavy, whereas edge devices are usually equipped with limited computational and storage resources. Such models can be compressed and reduced for deployment on edge devices, but they may lose their capability and not perform well. Recent works used knowledge transfer techniques to transfer information from a large network (termed teacher) to a small one (termed student) in order to improve the performance of the latter. This approach seems to be promising for learning on edge devices, but a thorough investigation on its effectiveness is lacking. This paper provides an extensive study on the performance (in both accuracy and convergence speed) of knowledge transfer, considering different student architectures and different techniques for transferring knowledge from teacher to student. The results show that the performance of KT does vary by architectures and transfer techniques. A good performance improvement is obtained by transferring knowledge from both the intermediate layers and last layer of the teacher to a shallower student. But other architectures and transfer techniques do not fare so well and some of them even lead to negative performance impact.
AB - With the emergence of edge computing paradigm, many applications such as image recognition and augmented reality require to perform machine learning (ML) and artificial intelligence (AI) tasks on edge devices. Most AI and ML models are large and computational-heavy, whereas edge devices are usually equipped with limited computational and storage resources. Such models can be compressed and reduced for deployment on edge devices, but they may lose their capability and not perform well. Recent works used knowledge transfer techniques to transfer information from a large network (termed teacher) to a small one (termed student) in order to improve the performance of the latter. This approach seems to be promising for learning on edge devices, but a thorough investigation on its effectiveness is lacking. This paper provides an extensive study on the performance (in both accuracy and convergence speed) of knowledge transfer, considering different student architectures and different techniques for transferring knowledge from teacher to student. The results show that the performance of KT does vary by architectures and transfer techniques. A good performance improvement is obtained by transferring knowledge from both the intermediate layers and last layer of the teacher to a shallower student. But other architectures and transfer techniques do not fare so well and some of them even lead to negative performance impact.
KW - Cloud computing
KW - Deep neural networks
KW - Edge computing
KW - Knowledge transfer
UR - http://www.scopus.com/inward/record.url?scp=85055667496&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85055667496&partnerID=8YFLogxK
U2 - 10.1109/EDGE.2018.00013
DO - 10.1109/EDGE.2018.00013
M3 - Conference contribution
AN - SCOPUS:85055667496
T3 - Proceedings - 2018 IEEE International Conference on Edge Computing, EDGE 2018 - Part of the 2018 IEEE World Congress on Services
SP - 42
EP - 49
BT - Proceedings - 2018 IEEE International Conference on Edge Computing, EDGE 2018 - Part of the 2018 IEEE World Congress on Services
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 2018 IEEE International Conference on Edge Computing, EDGE 2018
Y2 - 2 July 2018 through 7 July 2018
ER -