Virtualization-based bandwidth management for parallel storage systems

Yiqi Xu, Lixi Wang, Dulcardo Arteaga, Ming Zhao, Yonggang Liu, Renato Figueiredo

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Citations (Scopus)

Abstract

This paper presents a new parallel storage management approach which supports the allocation of shared storage bandwidth on a per-application basis. Existing parallel storage systems are unable to differentiate I/Os from different applications and meet per-application bandwidth requirement. This limitation presents a hurdle for applications to achieve their desired performance, which will become even more challenging as high-performance computing (HPC) systems continue to scale up with respect to both the amount of available resources and the number of concurrent applications. This paper proposes a novel solution to address this challenge through the virtualization of parallel file systems (PFSes). Such PFS virtualization is achieved with user-level PFS proxies, which interpose between native PFS clients and servers and schedule the I/Os from different applications according to the resource sharing algorithm (e.g., SFQ(D)). In this way, virtual PFSes can be created on a perapplication basis, each with a specific bandwidth share allocated according to its I/O requirement. This approach is applicable to different PFS-based parallel storage systems and can be transparently integrated with existing as well as future HPC systems. A prototype of this approach is implemented upon PVFS2, a widely used PFS, and evaluated with experiments using a typical parallel I/O benchmark (IOR). Results show that this approach's overhead is very small and it achieves effective proportional sharing under different usage scenarios.

Original languageEnglish (US)
Title of host publicationProceedings of the 2010 5th Petascale Data Storage Workshop, PDSW'10, Held in Conjunction with SC'10 - The International Conference for High Performance Computing, Networking, Storage and Analysis
DOIs
StatePublished - 2010
Externally publishedYes
Event2010 5th Petascale Data Storage Workshop, PDSW'10, Held in Conjunction with SC'10 - The International Conference for High Performance Computing, Networking, Storage and Analysis - New Orleans, LA, United States
Duration: Nov 15 2010Nov 15 2010

Other

Other2010 5th Petascale Data Storage Workshop, PDSW'10, Held in Conjunction with SC'10 - The International Conference for High Performance Computing, Networking, Storage and Analysis
CountryUnited States
CityNew Orleans, LA
Period11/15/1011/15/10

Fingerprint

Bandwidth
Storage management
Virtualization
Computer systems
Servers
Experiments

ASJC Scopus subject areas

  • Computer Science Applications
  • Software

Cite this

Xu, Y., Wang, L., Arteaga, D., Zhao, M., Liu, Y., & Figueiredo, R. (2010). Virtualization-based bandwidth management for parallel storage systems. In Proceedings of the 2010 5th Petascale Data Storage Workshop, PDSW'10, Held in Conjunction with SC'10 - The International Conference for High Performance Computing, Networking, Storage and Analysis [5668094] https://doi.org/10.1109/PDSW.2010.5668094

Virtualization-based bandwidth management for parallel storage systems. / Xu, Yiqi; Wang, Lixi; Arteaga, Dulcardo; Zhao, Ming; Liu, Yonggang; Figueiredo, Renato.

Proceedings of the 2010 5th Petascale Data Storage Workshop, PDSW'10, Held in Conjunction with SC'10 - The International Conference for High Performance Computing, Networking, Storage and Analysis. 2010. 5668094.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Xu, Y, Wang, L, Arteaga, D, Zhao, M, Liu, Y & Figueiredo, R 2010, Virtualization-based bandwidth management for parallel storage systems. in Proceedings of the 2010 5th Petascale Data Storage Workshop, PDSW'10, Held in Conjunction with SC'10 - The International Conference for High Performance Computing, Networking, Storage and Analysis., 5668094, 2010 5th Petascale Data Storage Workshop, PDSW'10, Held in Conjunction with SC'10 - The International Conference for High Performance Computing, Networking, Storage and Analysis, New Orleans, LA, United States, 11/15/10. https://doi.org/10.1109/PDSW.2010.5668094
Xu Y, Wang L, Arteaga D, Zhao M, Liu Y, Figueiredo R. Virtualization-based bandwidth management for parallel storage systems. In Proceedings of the 2010 5th Petascale Data Storage Workshop, PDSW'10, Held in Conjunction with SC'10 - The International Conference for High Performance Computing, Networking, Storage and Analysis. 2010. 5668094 https://doi.org/10.1109/PDSW.2010.5668094
Xu, Yiqi ; Wang, Lixi ; Arteaga, Dulcardo ; Zhao, Ming ; Liu, Yonggang ; Figueiredo, Renato. / Virtualization-based bandwidth management for parallel storage systems. Proceedings of the 2010 5th Petascale Data Storage Workshop, PDSW'10, Held in Conjunction with SC'10 - The International Conference for High Performance Computing, Networking, Storage and Analysis. 2010.
@inproceedings{1a4baeb461b34db6b1067136005386d1,
title = "Virtualization-based bandwidth management for parallel storage systems",
abstract = "This paper presents a new parallel storage management approach which supports the allocation of shared storage bandwidth on a per-application basis. Existing parallel storage systems are unable to differentiate I/Os from different applications and meet per-application bandwidth requirement. This limitation presents a hurdle for applications to achieve their desired performance, which will become even more challenging as high-performance computing (HPC) systems continue to scale up with respect to both the amount of available resources and the number of concurrent applications. This paper proposes a novel solution to address this challenge through the virtualization of parallel file systems (PFSes). Such PFS virtualization is achieved with user-level PFS proxies, which interpose between native PFS clients and servers and schedule the I/Os from different applications according to the resource sharing algorithm (e.g., SFQ(D)). In this way, virtual PFSes can be created on a perapplication basis, each with a specific bandwidth share allocated according to its I/O requirement. This approach is applicable to different PFS-based parallel storage systems and can be transparently integrated with existing as well as future HPC systems. A prototype of this approach is implemented upon PVFS2, a widely used PFS, and evaluated with experiments using a typical parallel I/O benchmark (IOR). Results show that this approach's overhead is very small and it achieves effective proportional sharing under different usage scenarios.",
author = "Yiqi Xu and Lixi Wang and Dulcardo Arteaga and Ming Zhao and Yonggang Liu and Renato Figueiredo",
year = "2010",
doi = "10.1109/PDSW.2010.5668094",
language = "English (US)",
isbn = "9781424489121",
booktitle = "Proceedings of the 2010 5th Petascale Data Storage Workshop, PDSW'10, Held in Conjunction with SC'10 - The International Conference for High Performance Computing, Networking, Storage and Analysis",

}

TY - GEN

T1 - Virtualization-based bandwidth management for parallel storage systems

AU - Xu, Yiqi

AU - Wang, Lixi

AU - Arteaga, Dulcardo

AU - Zhao, Ming

AU - Liu, Yonggang

AU - Figueiredo, Renato

PY - 2010

Y1 - 2010

N2 - This paper presents a new parallel storage management approach which supports the allocation of shared storage bandwidth on a per-application basis. Existing parallel storage systems are unable to differentiate I/Os from different applications and meet per-application bandwidth requirement. This limitation presents a hurdle for applications to achieve their desired performance, which will become even more challenging as high-performance computing (HPC) systems continue to scale up with respect to both the amount of available resources and the number of concurrent applications. This paper proposes a novel solution to address this challenge through the virtualization of parallel file systems (PFSes). Such PFS virtualization is achieved with user-level PFS proxies, which interpose between native PFS clients and servers and schedule the I/Os from different applications according to the resource sharing algorithm (e.g., SFQ(D)). In this way, virtual PFSes can be created on a perapplication basis, each with a specific bandwidth share allocated according to its I/O requirement. This approach is applicable to different PFS-based parallel storage systems and can be transparently integrated with existing as well as future HPC systems. A prototype of this approach is implemented upon PVFS2, a widely used PFS, and evaluated with experiments using a typical parallel I/O benchmark (IOR). Results show that this approach's overhead is very small and it achieves effective proportional sharing under different usage scenarios.

AB - This paper presents a new parallel storage management approach which supports the allocation of shared storage bandwidth on a per-application basis. Existing parallel storage systems are unable to differentiate I/Os from different applications and meet per-application bandwidth requirement. This limitation presents a hurdle for applications to achieve their desired performance, which will become even more challenging as high-performance computing (HPC) systems continue to scale up with respect to both the amount of available resources and the number of concurrent applications. This paper proposes a novel solution to address this challenge through the virtualization of parallel file systems (PFSes). Such PFS virtualization is achieved with user-level PFS proxies, which interpose between native PFS clients and servers and schedule the I/Os from different applications according to the resource sharing algorithm (e.g., SFQ(D)). In this way, virtual PFSes can be created on a perapplication basis, each with a specific bandwidth share allocated according to its I/O requirement. This approach is applicable to different PFS-based parallel storage systems and can be transparently integrated with existing as well as future HPC systems. A prototype of this approach is implemented upon PVFS2, a widely used PFS, and evaluated with experiments using a typical parallel I/O benchmark (IOR). Results show that this approach's overhead is very small and it achieves effective proportional sharing under different usage scenarios.

UR - http://www.scopus.com/inward/record.url?scp=78651381106&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=78651381106&partnerID=8YFLogxK

U2 - 10.1109/PDSW.2010.5668094

DO - 10.1109/PDSW.2010.5668094

M3 - Conference contribution

AN - SCOPUS:78651381106

SN - 9781424489121

BT - Proceedings of the 2010 5th Petascale Data Storage Workshop, PDSW'10, Held in Conjunction with SC'10 - The International Conference for High Performance Computing, Networking, Storage and Analysis

ER -