Decentralized scheduling with data locality for data-parallel computation on peer-to-peer networks

Weina Wang, Matthew Barnard, Lei Ying

Research output: Chapter in Book/Report/Conference proceedingConference contribution

5 Citations (Scopus)

Abstract

Despite distributed in computation and data storage, current data-parallel computing systems are centralized in task scheduling, which results in hierarchies that create single point of failure, limit scalability, and increase administration costs. In this paper, we propose a fully decentralized scheduling algorithm for data-parallel computing systems on peer-to-peer (P2P) networks. Our scheduling algorithm eliminates the centralized scheduler by letting each node in the network make scheduling decisions. To achieve good performance, data locality, which stresses the efficiency of colocating tasks with their input data, and load-balancing, should be considered jointly, and in a decentralized fashion. By exploring a backpressure-based approach, the proposed task scheduling algorithm strikes the right balance between data locality and load-balancing with each node only knowing the status information of part of the nodes in the network, and proves to maximize the throughput.

Original languageEnglish (US)
Title of host publication2015 53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages337-344
Number of pages8
ISBN (Print)9781509018239
DOIs
StatePublished - Apr 4 2016
Event53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015 - Monticello, United States
Duration: Sep 29 2015Oct 2 2015

Other

Other53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015
CountryUnited States
CityMonticello
Period9/29/1510/2/15

Fingerprint

Peer to peer networks
Scheduling algorithms
Scheduling
Parallel processing systems
Resource allocation
Scalability
Throughput
Data storage equipment
Costs

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications
  • Control and Systems Engineering

Cite this

Wang, W., Barnard, M., & Ying, L. (2016). Decentralized scheduling with data locality for data-parallel computation on peer-to-peer networks. In 2015 53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015 (pp. 337-344). [7447024] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ALLERTON.2015.7447024

Decentralized scheduling with data locality for data-parallel computation on peer-to-peer networks. / Wang, Weina; Barnard, Matthew; Ying, Lei.

2015 53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015. Institute of Electrical and Electronics Engineers Inc., 2016. p. 337-344 7447024.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Wang, W, Barnard, M & Ying, L 2016, Decentralized scheduling with data locality for data-parallel computation on peer-to-peer networks. in 2015 53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015., 7447024, Institute of Electrical and Electronics Engineers Inc., pp. 337-344, 53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015, Monticello, United States, 9/29/15. https://doi.org/10.1109/ALLERTON.2015.7447024
Wang W, Barnard M, Ying L. Decentralized scheduling with data locality for data-parallel computation on peer-to-peer networks. In 2015 53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015. Institute of Electrical and Electronics Engineers Inc. 2016. p. 337-344. 7447024 https://doi.org/10.1109/ALLERTON.2015.7447024
Wang, Weina ; Barnard, Matthew ; Ying, Lei. / Decentralized scheduling with data locality for data-parallel computation on peer-to-peer networks. 2015 53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015. Institute of Electrical and Electronics Engineers Inc., 2016. pp. 337-344
@inproceedings{5630f594b5ca4ffebc0e211ec62047e8,
title = "Decentralized scheduling with data locality for data-parallel computation on peer-to-peer networks",
abstract = "Despite distributed in computation and data storage, current data-parallel computing systems are centralized in task scheduling, which results in hierarchies that create single point of failure, limit scalability, and increase administration costs. In this paper, we propose a fully decentralized scheduling algorithm for data-parallel computing systems on peer-to-peer (P2P) networks. Our scheduling algorithm eliminates the centralized scheduler by letting each node in the network make scheduling decisions. To achieve good performance, data locality, which stresses the efficiency of colocating tasks with their input data, and load-balancing, should be considered jointly, and in a decentralized fashion. By exploring a backpressure-based approach, the proposed task scheduling algorithm strikes the right balance between data locality and load-balancing with each node only knowing the status information of part of the nodes in the network, and proves to maximize the throughput.",
author = "Weina Wang and Matthew Barnard and Lei Ying",
year = "2016",
month = "4",
day = "4",
doi = "10.1109/ALLERTON.2015.7447024",
language = "English (US)",
isbn = "9781509018239",
pages = "337--344",
booktitle = "2015 53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - GEN

T1 - Decentralized scheduling with data locality for data-parallel computation on peer-to-peer networks

AU - Wang, Weina

AU - Barnard, Matthew

AU - Ying, Lei

PY - 2016/4/4

Y1 - 2016/4/4

N2 - Despite distributed in computation and data storage, current data-parallel computing systems are centralized in task scheduling, which results in hierarchies that create single point of failure, limit scalability, and increase administration costs. In this paper, we propose a fully decentralized scheduling algorithm for data-parallel computing systems on peer-to-peer (P2P) networks. Our scheduling algorithm eliminates the centralized scheduler by letting each node in the network make scheduling decisions. To achieve good performance, data locality, which stresses the efficiency of colocating tasks with their input data, and load-balancing, should be considered jointly, and in a decentralized fashion. By exploring a backpressure-based approach, the proposed task scheduling algorithm strikes the right balance between data locality and load-balancing with each node only knowing the status information of part of the nodes in the network, and proves to maximize the throughput.

AB - Despite distributed in computation and data storage, current data-parallel computing systems are centralized in task scheduling, which results in hierarchies that create single point of failure, limit scalability, and increase administration costs. In this paper, we propose a fully decentralized scheduling algorithm for data-parallel computing systems on peer-to-peer (P2P) networks. Our scheduling algorithm eliminates the centralized scheduler by letting each node in the network make scheduling decisions. To achieve good performance, data locality, which stresses the efficiency of colocating tasks with their input data, and load-balancing, should be considered jointly, and in a decentralized fashion. By exploring a backpressure-based approach, the proposed task scheduling algorithm strikes the right balance between data locality and load-balancing with each node only knowing the status information of part of the nodes in the network, and proves to maximize the throughput.

UR - http://www.scopus.com/inward/record.url?scp=84969894797&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84969894797&partnerID=8YFLogxK

U2 - 10.1109/ALLERTON.2015.7447024

DO - 10.1109/ALLERTON.2015.7447024

M3 - Conference contribution

AN - SCOPUS:84969894797

SN - 9781509018239

SP - 337

EP - 344

BT - 2015 53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015

PB - Institute of Electrical and Electronics Engineers Inc.

ER -