Decentralized scheduling with data locality for data-parallel computation on peer-to-peer networks

Weina Wang, Matthew Barnard, Lei Ying

Research output: Chapter in Book/Report/Conference proceedingConference contribution

6 Scopus citations

Abstract

Despite distributed in computation and data storage, current data-parallel computing systems are centralized in task scheduling, which results in hierarchies that create single point of failure, limit scalability, and increase administration costs. In this paper, we propose a fully decentralized scheduling algorithm for data-parallel computing systems on peer-to-peer (P2P) networks. Our scheduling algorithm eliminates the centralized scheduler by letting each node in the network make scheduling decisions. To achieve good performance, data locality, which stresses the efficiency of colocating tasks with their input data, and load-balancing, should be considered jointly, and in a decentralized fashion. By exploring a backpressure-based approach, the proposed task scheduling algorithm strikes the right balance between data locality and load-balancing with each node only knowing the status information of part of the nodes in the network, and proves to maximize the throughput.

Original languageEnglish (US)
Title of host publication2015 53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages337-344
Number of pages8
ISBN (Print)9781509018239
DOIs
StatePublished - Apr 4 2016
Event53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015 - Monticello, United States
Duration: Sep 29 2015Oct 2 2015

Other

Other53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015
CountryUnited States
CityMonticello
Period9/29/1510/2/15

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications
  • Control and Systems Engineering

Fingerprint Dive into the research topics of 'Decentralized scheduling with data locality for data-parallel computation on peer-to-peer networks'. Together they form a unique fingerprint.

  • Cite this

    Wang, W., Barnard, M., & Ying, L. (2016). Decentralized scheduling with data locality for data-parallel computation on peer-to-peer networks. In 2015 53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015 (pp. 337-344). [7447024] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ALLERTON.2015.7447024