Data locality in MapReduce: A network perspective

Weina Wang, Lei Ying

Research output: Contribution to journalArticle

9 Scopus citations

Abstract

Data locality, a critical consideration for the performance of task scheduling in MapReduce, has been addressed in the literature by increasing the number of locally processed tasks. In this paper, we view the data locality problem from a network perspective. The key observation is that if we make appropriate use of the network to route the data chunk to the machine where it will be processed in advance, then processing a remote task is the same as processing a local task. However, to benefit from such a strategy, we must (i) balance the tasks assigned to local machines and those assigned to remote machines, and (ii) design the routing algorithm to avoid network congestion. Taking these challenges into consideration, we propose a scheduling/routing algorithm, named the Joint Scheduler, which utilizes both the computing resources and the communication network efficiently. We prove that the Joint Scheduler is throughput optimal; i.e., it supports any load that is supportable by any other algorithm. Simulation results demonstrate that with popularity skew, the Joint Scheduler improves the throughput and delay performance significantly compared to the Hadoop Fair Scheduler with delay scheduling, which is the de facto industry standard.

Original languageEnglish (US)
Pages (from-to)1-11
Number of pages11
JournalPerformance Evaluation
Volume96
DOIs
StatePublished - Feb 2016

Keywords

  • Data locality
  • MapReduce
  • Routing
  • Scheduling
  • Throughput

ASJC Scopus subject areas

  • Software
  • Modeling and Simulation
  • Hardware and Architecture
  • Computer Networks and Communications

Fingerprint Dive into the research topics of 'Data locality in MapReduce: A network perspective'. Together they form a unique fingerprint.

  • Cite this