Exploring the use of synthetic gradients for distributed deep learning across cloud and edge resources

Yitao Chen, Kaiqi Zhao, Baoxin Li, Ming Zhao

Research output: Contribution to conferencePaperpeer-review

1 Scopus citations

Abstract

With the explosive growth of data, largely contributed by the rapidly and widely deployed smart devices on the edge, we need to rethink the training paradigm for learning on such real-world data. The conventional cloud-only approach can hardly keep up with the computational demand from these deep learning tasks; and the traditional back propagation based training method also makes it difficult to scale out the training. Fortunately, the continuous advancement in System on Chip (SoC) hardware is transforming edge devices into capable computing platforms, and can potentially be exploited to address these challenges. These observations have motivated this paper’s study on the use of synthetic gradients for distributed training cross cloud and edge devices. We employ synthetic gradients into various neural network models to comprehensively evaluate its feasibility in terms of accuracy and convergence speed. We distribute the training of the various layers of a model using synthetic gradients, and evaluate its effectiveness on the edge by using resource-limited containers to emulate edge devices. The evaluation result shows that the synthetic gradient approach can achieve comparable accuracy compared to the conventional back propagation, for an eight-layer model with both fully-connected and convolutional layers. For a more complex model (VGG16), the training suffers from some accuracy degradation (up to 15%). But it achieves 11% improvement in training speed when the layers of a model are decoupled and trained on separate resource-limited containers, compared to the training of the whole model using the conventional method on the physical machine.

Original languageEnglish (US)
StatePublished - 2019
Event2nd USENIX Workshop on Hot Topics in Edge Computing, HotEdge 2019, co-located with USENIX ATC 2019 - Renton, United States
Duration: Jul 9 2019 → …

Conference

Conference2nd USENIX Workshop on Hot Topics in Edge Computing, HotEdge 2019, co-located with USENIX ATC 2019
CountryUnited States
CityRenton
Period7/9/19 → …

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Hardware and Architecture
  • Information Systems and Management

Fingerprint Dive into the research topics of 'Exploring the use of synthetic gradients for distributed deep learning across cloud and edge resources'. Together they form a unique fingerprint.

Cite this