Support for data-intensive, variable-granularity Grid applications via distributed file system virtualization - A case study of Light Scattering Spectroscopy

Jithendar Paladugula, Ming Zhao, Renato J. Figueiredo

Research output: Chapter in Book/Report/Conference proceedingConference contribution

6 Citations (Scopus)

Abstract

A key challenge faced by large-scale, distributed applications in Grid environments is efficient, seamless data management. In particular, for applications that can benefit from access to data at variable granularities, data management can pose additional programming burdens to an application developer. This paper presents a case for the use of virtualized distributed file systems as a basis for data management for data-intensive, variable-granularity applications. The approach leverages on-demand transfer mechanisms of existing, de-facto network file system clients and servers that support transfers of partial data sets in an application-transparent fashion, and complement them with user-level performance and functionality enhancements such as caching and encrypted communication channels. The paper uses a nascent application from the medical imaging field (Light Scattering Spectroscopy - LSS) as a motivation for the approach, and as a basis for evaluating its performance. Results from performance experiments that consider the 16-processor parallel execution of LSS analysis and database generation programs show that, in the presence of data locality, a virtualized wide-area distributed file system setup and configured by Grid middleware can achieve performance levels close (13% overhead or less) to that of a local disk, and superior (up to 680% speedup) to non-virtualized distributed file systems.

Original languageEnglish (US)
Title of host publicationProceedings of the Second International Workshop on Challenges of Large Applications in Distributed Environments
Pages12-21
Number of pages10
DOIs
StatePublished - 2004
Externally publishedYes
Eventof Large Applications in Distributed Environments - Honolulu, HI, United States
Duration: Jun 7 2004Jun 7 2004

Other

Otherof Large Applications in Distributed Environments
CountryUnited States
CityHonolulu, HI
Period6/7/046/7/04

Fingerprint

Light scattering
Spectroscopy
Information management
Medical imaging
Middleware
Virtualization
Servers
Experiments

ASJC Scopus subject areas

  • Engineering(all)

Cite this

Paladugula, J., Zhao, M., & Figueiredo, R. J. (2004). Support for data-intensive, variable-granularity Grid applications via distributed file system virtualization - A case study of Light Scattering Spectroscopy. In Proceedings of the Second International Workshop on Challenges of Large Applications in Distributed Environments (pp. 12-21) https://doi.org/10.1109/CLADE.2004.1309088

Support for data-intensive, variable-granularity Grid applications via distributed file system virtualization - A case study of Light Scattering Spectroscopy. / Paladugula, Jithendar; Zhao, Ming; Figueiredo, Renato J.

Proceedings of the Second International Workshop on Challenges of Large Applications in Distributed Environments. 2004. p. 12-21.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Paladugula, J, Zhao, M & Figueiredo, RJ 2004, Support for data-intensive, variable-granularity Grid applications via distributed file system virtualization - A case study of Light Scattering Spectroscopy. in Proceedings of the Second International Workshop on Challenges of Large Applications in Distributed Environments. pp. 12-21, of Large Applications in Distributed Environments, Honolulu, HI, United States, 6/7/04. https://doi.org/10.1109/CLADE.2004.1309088
Paladugula J, Zhao M, Figueiredo RJ. Support for data-intensive, variable-granularity Grid applications via distributed file system virtualization - A case study of Light Scattering Spectroscopy. In Proceedings of the Second International Workshop on Challenges of Large Applications in Distributed Environments. 2004. p. 12-21 https://doi.org/10.1109/CLADE.2004.1309088
Paladugula, Jithendar ; Zhao, Ming ; Figueiredo, Renato J. / Support for data-intensive, variable-granularity Grid applications via distributed file system virtualization - A case study of Light Scattering Spectroscopy. Proceedings of the Second International Workshop on Challenges of Large Applications in Distributed Environments. 2004. pp. 12-21
@inproceedings{1fc634c796004dc6a98fea43f4ac8eea,
title = "Support for data-intensive, variable-granularity Grid applications via distributed file system virtualization - A case study of Light Scattering Spectroscopy",
abstract = "A key challenge faced by large-scale, distributed applications in Grid environments is efficient, seamless data management. In particular, for applications that can benefit from access to data at variable granularities, data management can pose additional programming burdens to an application developer. This paper presents a case for the use of virtualized distributed file systems as a basis for data management for data-intensive, variable-granularity applications. The approach leverages on-demand transfer mechanisms of existing, de-facto network file system clients and servers that support transfers of partial data sets in an application-transparent fashion, and complement them with user-level performance and functionality enhancements such as caching and encrypted communication channels. The paper uses a nascent application from the medical imaging field (Light Scattering Spectroscopy - LSS) as a motivation for the approach, and as a basis for evaluating its performance. Results from performance experiments that consider the 16-processor parallel execution of LSS analysis and database generation programs show that, in the presence of data locality, a virtualized wide-area distributed file system setup and configured by Grid middleware can achieve performance levels close (13{\%} overhead or less) to that of a local disk, and superior (up to 680{\%} speedup) to non-virtualized distributed file systems.",
author = "Jithendar Paladugula and Ming Zhao and Figueiredo, {Renato J.}",
year = "2004",
doi = "10.1109/CLADE.2004.1309088",
language = "English (US)",
isbn = "0769521150",
pages = "12--21",
booktitle = "Proceedings of the Second International Workshop on Challenges of Large Applications in Distributed Environments",

}

TY - GEN

T1 - Support for data-intensive, variable-granularity Grid applications via distributed file system virtualization - A case study of Light Scattering Spectroscopy

AU - Paladugula, Jithendar

AU - Zhao, Ming

AU - Figueiredo, Renato J.

PY - 2004

Y1 - 2004

N2 - A key challenge faced by large-scale, distributed applications in Grid environments is efficient, seamless data management. In particular, for applications that can benefit from access to data at variable granularities, data management can pose additional programming burdens to an application developer. This paper presents a case for the use of virtualized distributed file systems as a basis for data management for data-intensive, variable-granularity applications. The approach leverages on-demand transfer mechanisms of existing, de-facto network file system clients and servers that support transfers of partial data sets in an application-transparent fashion, and complement them with user-level performance and functionality enhancements such as caching and encrypted communication channels. The paper uses a nascent application from the medical imaging field (Light Scattering Spectroscopy - LSS) as a motivation for the approach, and as a basis for evaluating its performance. Results from performance experiments that consider the 16-processor parallel execution of LSS analysis and database generation programs show that, in the presence of data locality, a virtualized wide-area distributed file system setup and configured by Grid middleware can achieve performance levels close (13% overhead or less) to that of a local disk, and superior (up to 680% speedup) to non-virtualized distributed file systems.

AB - A key challenge faced by large-scale, distributed applications in Grid environments is efficient, seamless data management. In particular, for applications that can benefit from access to data at variable granularities, data management can pose additional programming burdens to an application developer. This paper presents a case for the use of virtualized distributed file systems as a basis for data management for data-intensive, variable-granularity applications. The approach leverages on-demand transfer mechanisms of existing, de-facto network file system clients and servers that support transfers of partial data sets in an application-transparent fashion, and complement them with user-level performance and functionality enhancements such as caching and encrypted communication channels. The paper uses a nascent application from the medical imaging field (Light Scattering Spectroscopy - LSS) as a motivation for the approach, and as a basis for evaluating its performance. Results from performance experiments that consider the 16-processor parallel execution of LSS analysis and database generation programs show that, in the presence of data locality, a virtualized wide-area distributed file system setup and configured by Grid middleware can achieve performance levels close (13% overhead or less) to that of a local disk, and superior (up to 680% speedup) to non-virtualized distributed file systems.

UR - http://www.scopus.com/inward/record.url?scp=10444282125&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=10444282125&partnerID=8YFLogxK

U2 - 10.1109/CLADE.2004.1309088

DO - 10.1109/CLADE.2004.1309088

M3 - Conference contribution

SN - 0769521150

SN - 9780769521152

SP - 12

EP - 21

BT - Proceedings of the Second International Workshop on Challenges of Large Applications in Distributed Environments

ER -