Data locality in MapReduce: A network perspective

Weina Wang, Lei Ying

Research output: Chapter in Book/Report/Conference proceedingConference contribution

5 Citations (Scopus)

Abstract

In MapReduce, placing computation near its input data is considered to be desirable since otherwise the data transmission introduces an additional delay to the task execution. This data locality problem has been studied in the literature. Most existing scheduling algorithms in MapReduce focus on improving performance through increasing locality. In this paper, we view the data locality problem from a network perspective. The key observation is that if we make appropriate use of the network to route the data chunk to the machine where it will be processed in advance, then processing a remote task is the same as processing a local task. In other words, instead of bringing computation close to data, we can also bring data close to computation to improve the system performance. However, to benefit from such a strategy, we must (i) balance the tasks assigned to local machines and those assigned to remote machines, and (ii) design the routing algorithm to avoid network congestion. Taking these challenges into consideration, we propose a scheduling/routing algorithm, named the Joint Scheduler, which utilizes both the computing resources and the communication network efficiently. To show that the Joint Scheduler has superior performance, we prove that the Join Scheduler can support any load that can be supported by some other algorithm, i.e., achieve the maximum capacity region. Simulation results demonstrate that with popularity skew, the Joint Scheduler improves the throughput significantly (more than 30% in our simulations) compared to the Hadoop Fair Scheduler with delay scheduling, which is the de facto industry standard. The delay performance is also evaluated through simulations, where we can see a significant delay reduce under the Joint Scheduler with moderate to heavy load.

Original languageEnglish (US)
Title of host publication2014 52nd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2014
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1110-1117
Number of pages8
ISBN (Print)9781479980093
DOIs
StatePublished - Jan 30 2014
Event2014 52nd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2014 - Monticello, United States
Duration: Sep 30 2014Oct 3 2014

Other

Other2014 52nd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2014
CountryUnited States
CityMonticello
Period9/30/1410/3/14

Fingerprint

Routing algorithms
Scheduling algorithms
Processing
Data communication systems
Telecommunication networks
Scheduling
Throughput
Industry

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications

Cite this

Wang, W., & Ying, L. (2014). Data locality in MapReduce: A network perspective. In 2014 52nd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2014 (pp. 1110-1117). [7028579] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ALLERTON.2014.7028579

Data locality in MapReduce : A network perspective. / Wang, Weina; Ying, Lei.

2014 52nd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2014. Institute of Electrical and Electronics Engineers Inc., 2014. p. 1110-1117 7028579.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Wang, W & Ying, L 2014, Data locality in MapReduce: A network perspective. in 2014 52nd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2014., 7028579, Institute of Electrical and Electronics Engineers Inc., pp. 1110-1117, 2014 52nd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2014, Monticello, United States, 9/30/14. https://doi.org/10.1109/ALLERTON.2014.7028579
Wang W, Ying L. Data locality in MapReduce: A network perspective. In 2014 52nd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2014. Institute of Electrical and Electronics Engineers Inc. 2014. p. 1110-1117. 7028579 https://doi.org/10.1109/ALLERTON.2014.7028579
Wang, Weina ; Ying, Lei. / Data locality in MapReduce : A network perspective. 2014 52nd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2014. Institute of Electrical and Electronics Engineers Inc., 2014. pp. 1110-1117
@inproceedings{5628c2560dfb476e9acb4fdbe5e52dd5,
title = "Data locality in MapReduce: A network perspective",
abstract = "In MapReduce, placing computation near its input data is considered to be desirable since otherwise the data transmission introduces an additional delay to the task execution. This data locality problem has been studied in the literature. Most existing scheduling algorithms in MapReduce focus on improving performance through increasing locality. In this paper, we view the data locality problem from a network perspective. The key observation is that if we make appropriate use of the network to route the data chunk to the machine where it will be processed in advance, then processing a remote task is the same as processing a local task. In other words, instead of bringing computation close to data, we can also bring data close to computation to improve the system performance. However, to benefit from such a strategy, we must (i) balance the tasks assigned to local machines and those assigned to remote machines, and (ii) design the routing algorithm to avoid network congestion. Taking these challenges into consideration, we propose a scheduling/routing algorithm, named the Joint Scheduler, which utilizes both the computing resources and the communication network efficiently. To show that the Joint Scheduler has superior performance, we prove that the Join Scheduler can support any load that can be supported by some other algorithm, i.e., achieve the maximum capacity region. Simulation results demonstrate that with popularity skew, the Joint Scheduler improves the throughput significantly (more than 30{\%} in our simulations) compared to the Hadoop Fair Scheduler with delay scheduling, which is the de facto industry standard. The delay performance is also evaluated through simulations, where we can see a significant delay reduce under the Joint Scheduler with moderate to heavy load.",
author = "Weina Wang and Lei Ying",
year = "2014",
month = "1",
day = "30",
doi = "10.1109/ALLERTON.2014.7028579",
language = "English (US)",
isbn = "9781479980093",
pages = "1110--1117",
booktitle = "2014 52nd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2014",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - GEN

T1 - Data locality in MapReduce

T2 - A network perspective

AU - Wang, Weina

AU - Ying, Lei

PY - 2014/1/30

Y1 - 2014/1/30

N2 - In MapReduce, placing computation near its input data is considered to be desirable since otherwise the data transmission introduces an additional delay to the task execution. This data locality problem has been studied in the literature. Most existing scheduling algorithms in MapReduce focus on improving performance through increasing locality. In this paper, we view the data locality problem from a network perspective. The key observation is that if we make appropriate use of the network to route the data chunk to the machine where it will be processed in advance, then processing a remote task is the same as processing a local task. In other words, instead of bringing computation close to data, we can also bring data close to computation to improve the system performance. However, to benefit from such a strategy, we must (i) balance the tasks assigned to local machines and those assigned to remote machines, and (ii) design the routing algorithm to avoid network congestion. Taking these challenges into consideration, we propose a scheduling/routing algorithm, named the Joint Scheduler, which utilizes both the computing resources and the communication network efficiently. To show that the Joint Scheduler has superior performance, we prove that the Join Scheduler can support any load that can be supported by some other algorithm, i.e., achieve the maximum capacity region. Simulation results demonstrate that with popularity skew, the Joint Scheduler improves the throughput significantly (more than 30% in our simulations) compared to the Hadoop Fair Scheduler with delay scheduling, which is the de facto industry standard. The delay performance is also evaluated through simulations, where we can see a significant delay reduce under the Joint Scheduler with moderate to heavy load.

AB - In MapReduce, placing computation near its input data is considered to be desirable since otherwise the data transmission introduces an additional delay to the task execution. This data locality problem has been studied in the literature. Most existing scheduling algorithms in MapReduce focus on improving performance through increasing locality. In this paper, we view the data locality problem from a network perspective. The key observation is that if we make appropriate use of the network to route the data chunk to the machine where it will be processed in advance, then processing a remote task is the same as processing a local task. In other words, instead of bringing computation close to data, we can also bring data close to computation to improve the system performance. However, to benefit from such a strategy, we must (i) balance the tasks assigned to local machines and those assigned to remote machines, and (ii) design the routing algorithm to avoid network congestion. Taking these challenges into consideration, we propose a scheduling/routing algorithm, named the Joint Scheduler, which utilizes both the computing resources and the communication network efficiently. To show that the Joint Scheduler has superior performance, we prove that the Join Scheduler can support any load that can be supported by some other algorithm, i.e., achieve the maximum capacity region. Simulation results demonstrate that with popularity skew, the Joint Scheduler improves the throughput significantly (more than 30% in our simulations) compared to the Hadoop Fair Scheduler with delay scheduling, which is the de facto industry standard. The delay performance is also evaluated through simulations, where we can see a significant delay reduce under the Joint Scheduler with moderate to heavy load.

UR - http://www.scopus.com/inward/record.url?scp=84946692901&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84946692901&partnerID=8YFLogxK

U2 - 10.1109/ALLERTON.2014.7028579

DO - 10.1109/ALLERTON.2014.7028579

M3 - Conference contribution

AN - SCOPUS:84946692901

SN - 9781479980093

SP - 1110

EP - 1117

BT - 2014 52nd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2014

PB - Institute of Electrical and Electronics Engineers Inc.

ER -