Map task scheduling in MapReduce with data locality: Throughput and heavy-traffic optimality

Weina Wang, Kai Zhu, Lei Ying, Jian Tan, Li Zhang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

57 Scopus citations

Abstract

Scheduling map tasks to improve data locality is crucial to the performance of MapReduce. Many works have been devoted to increasing data locality for better efficiency. However, to the best of our knowledge, fundamental limits of MapReduce computing clusters with data locality, including the capacity region and theoretical bounds on the delay performance, have not been studied. In this paper, we address these problems from a stochastic network perspective. Our focus is to strike the right balance between data-locality and load-balancing to simultaneously maximize throughput and minimize delay. We present a new queueing architecture and propose a map task scheduling algorithm constituted by the Join the Shortest Queue policy together with the MaxWeight policy. We identify an outer bound on the capacity region, and then prove that the proposed algorithm stabilizes any arrival rate vector strictly within this outer bound. It shows that the algorithm is throughput optimal and the outer bound coincides with the actual capacity region. Further, we study the number of backlogged tasks under the proposed algorithm, which is directly related to the delay performance based on Little's law. We prove that the proposed algorithm is heavy-traffic optimal, i.e., it asymptotically minimizes the number of backlogged tasks as the arrival rate vector approaches the boundary of the capacity region. Therefore, the proposed algorithm is also delay optimal in the heavy-traffic regime.

Original languageEnglish (US)
Title of host publication2013 Proceedings IEEE INFOCOM 2013
Pages1609-1617
Number of pages9
DOIs
StatePublished - Sep 2 2013
Event32nd IEEE Conference on Computer Communications, IEEE INFOCOM 2013 - Turin, Italy
Duration: Apr 14 2013Apr 19 2013

Publication series

NameProceedings - IEEE INFOCOM
ISSN (Print)0743-166X

Other

Other32nd IEEE Conference on Computer Communications, IEEE INFOCOM 2013
CountryItaly
CityTurin
Period4/14/134/19/13

    Fingerprint

ASJC Scopus subject areas

  • Computer Science(all)
  • Electrical and Electronic Engineering

Cite this

Wang, W., Zhu, K., Ying, L., Tan, J., & Zhang, L. (2013). Map task scheduling in MapReduce with data locality: Throughput and heavy-traffic optimality. In 2013 Proceedings IEEE INFOCOM 2013 (pp. 1609-1617). [6566957] (Proceedings - IEEE INFOCOM). https://doi.org/10.1109/INFCOM.2013.6566957