IBIS: Interposed big-data I/O scheduler

Yiqi Xu, Adrian Suarez, Ming Zhao

Research output: Chapter in Book/Report/Conference proceedingConference contribution

8 Scopus citations

Abstract

Existing big-data systems (e.g., Hadoop/MapReduce) do not expose management of shared storage I/O resources. As such, application's performance may degrade in unpredictable ways under I/O contention, even with fair sharing of computing resources. This paper proposes IBIS, a new Interposed Big-data I/O Scheduler, to provide performance differentiation for competing applications' I/Os in a shared MapReduce-type big-data system. IBIS is implemented in Hadoop by interposing HDFS I/Os and use an SFQ-based proportional-sharing algorithm. Experiments show that the IBIS provides strong performance isolation for one application against another highly I/O-intensive application. IBIS also enforces good proportional sharing of the global bandwidth among competing parallel applications, by coordinating distributed IBIS schedulers to deal with the uneven distribution of local services in big-data systems.

Original languageEnglish (US)
Title of host publicationHPDC 2013 - Proceedings of the 22nd ACM International Symposium on High-Performance Parallel and Distributed Computing
Pages109-110
Number of pages2
DOIs
StatePublished - 2013
Externally publishedYes
Event22nd ACM International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2013 - New York, NY, United States
Duration: Jun 17 2013Jun 21 2013

Other

Other22nd ACM International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2013
CountryUnited States
CityNew York, NY
Period6/17/136/21/13

Keywords

  • distributed storage
  • proportional sharing

ASJC Scopus subject areas

  • Software

Fingerprint Dive into the research topics of 'IBIS: Interposed big-data I/O scheduler'. Together they form a unique fingerprint.

Cite this