TY - GEN
T1 - IBIS
T2 - 22nd ACM International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2013
AU - Xu, Yiqi
AU - Suarez, Adrian
AU - Zhao, Ming
PY - 2013
Y1 - 2013
N2 - Existing big-data systems (e.g., Hadoop/MapReduce) do not expose management of shared storage I/O resources. As such, application's performance may degrade in unpredictable ways under I/O contention, even with fair sharing of computing resources. This paper proposes IBIS, a new Interposed Big-data I/O Scheduler, to provide performance differentiation for competing applications' I/Os in a shared MapReduce-type big-data system. IBIS is implemented in Hadoop by interposing HDFS I/Os and use an SFQ-based proportional-sharing algorithm. Experiments show that the IBIS provides strong performance isolation for one application against another highly I/O-intensive application. IBIS also enforces good proportional sharing of the global bandwidth among competing parallel applications, by coordinating distributed IBIS schedulers to deal with the uneven distribution of local services in big-data systems.
AB - Existing big-data systems (e.g., Hadoop/MapReduce) do not expose management of shared storage I/O resources. As such, application's performance may degrade in unpredictable ways under I/O contention, even with fair sharing of computing resources. This paper proposes IBIS, a new Interposed Big-data I/O Scheduler, to provide performance differentiation for competing applications' I/Os in a shared MapReduce-type big-data system. IBIS is implemented in Hadoop by interposing HDFS I/Os and use an SFQ-based proportional-sharing algorithm. Experiments show that the IBIS provides strong performance isolation for one application against another highly I/O-intensive application. IBIS also enforces good proportional sharing of the global bandwidth among competing parallel applications, by coordinating distributed IBIS schedulers to deal with the uneven distribution of local services in big-data systems.
KW - distributed storage
KW - proportional sharing
UR - http://www.scopus.com/inward/record.url?scp=84880079234&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84880079234&partnerID=8YFLogxK
U2 - 10.1145/2462902.2462922
DO - 10.1145/2462902.2462922
M3 - Conference contribution
AN - SCOPUS:84880079234
SN - 9781450319102
T3 - HPDC 2013 - Proceedings of the 22nd ACM International Symposium on High-Performance Parallel and Distributed Computing
SP - 109
EP - 110
BT - HPDC 2013 - Proceedings of the 22nd ACM International Symposium on High-Performance Parallel and Distributed Computing
PB - Association for Computing Machinery
Y2 - 17 June 2013 through 21 June 2013
ER -