Communication and Computation Reduction for Split Learning using Asynchronous Training

Xing Chen, Jingtao Li, Chaitali Chakrabarti

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Split learning is a promising privacy-preserving distributed learning scheme that has low computation requirement at the edge device but has the disadvantage of high communication overhead between edge device and server. To reduce the communication overhead, this paper proposes a loss-based asynchronous training scheme that updates the client-side model less frequently and only sends/receives activations/gradients in selected epochs. To further reduce the communication over-head, the activations/gradients are quantized using 8-bit floating point prior to transmission. An added benefit of the proposed communication reduction method is that the computations at the client side are reduced due to reduction in the number of client model updates. Furthermore, the privacy of the proposed communication reduction based split learning method is almost the same as traditional split learning. Simulation results on VGG11, VGG13 and ResNet18 models on CIFAR-10 show that the communication cost is reduced by 1.64x-106.7x and the computations in the client are reduced by 2.86x-32.1x when the accuracy degradation is less than 0.5% for the single-client case. For 5 and 10-client cases, the communication cost reduction is 11.9x and 11.3x on VGG11 for 0.5% loss in accuracy.

Original languageEnglish (US)
Title of host publicationProceedings - 2021 IEEE Workshop on Signal Processing Systems, SiPS 2021
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages76-81
Number of pages6
ISBN (Electronic)9781665401449
DOIs
StatePublished - 2021
Event2021 IEEE Workshop on Signal Processing Systems, SiPS 2021 - Coimbra, Portugal
Duration: Oct 19 2021Oct 21 2021

Publication series

NameIEEE Workshop on Signal Processing Systems, SiPS: Design and Implementation
Volume2021-October
ISSN (Print)1520-6130

Conference

Conference2021 IEEE Workshop on Signal Processing Systems, SiPS 2021
Country/TerritoryPortugal
CityCoimbra
Period10/19/2110/21/21

Keywords

  • Asynchronous training
  • Communication reduction
  • Quantization
  • Split learning

ASJC Scopus subject areas

  • Electrical and Electronic Engineering
  • Signal Processing
  • Applied Mathematics
  • Hardware and Architecture

Fingerprint

Dive into the research topics of 'Communication and Computation Reduction for Split Learning using Asynchronous Training'. Together they form a unique fingerprint.

Cite this