Kaleido: Enabling Efficient Scientific Data Processing on Big-Data Systems

Saman Biookaghazadeh, Shujia Zhou, Ming Zhao

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Citations (Scopus)

Abstract

Big-Data systems are increasingly important for solving the data-driven problems in many science domains including geosciences. However, existing big-data systems cannot support the efficient processing of self-describing data formats such as NetCDF which are commonly used by scientific communities for data distribution and sharing. This limitation presents a serious hurdle to the further adoption of big-data systems by science domains. This paper presents Kaleido, a solution to this problem by enabling big-data systems to efficiently store and process scientific data. Specifically, it enables Hadoop to directly store NetCDF data on HDFS, and process them in MapReduce using convenient APIs. It also enables Hive to support queries on NetCDF data, transparent to the users. Moreover, it employs optimizations tailored to scientific data, particularly dimension-aware layout which allows efficient execution of subset queries targeting any dimension of the multi-dimensional data. The paper presents a comprehensive evaluation of Kaleido using representative queries on a typical geoscientific dataset. The results show that Kaleido achieves substantial speedup and space saving compared to existing solutions for storing and processing NetCDF data on Hadoop, and it also substantially outperforms the state-of-the-art solutions for supporting subset queries on scientific data.

Original languageEnglish (US)
Title of host publication2017 IEEE International Conference on Networking, Architecture, and Storage, NAS 2017 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781538634868
DOIs
StatePublished - Sep 6 2017
Event2017 IEEE International Conference on Networking, Architecture, and Storage, NAS 2017 - Shenzhen, China
Duration: Aug 7 2017Aug 9 2017

Other

Other2017 IEEE International Conference on Networking, Architecture, and Storage, NAS 2017
CountryChina
CityShenzhen
Period8/7/178/9/17

Fingerprint

Application programming interfaces (API)
Big data
Processing

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Hardware and Architecture

Cite this

Biookaghazadeh, S., Zhou, S., & Zhao, M. (2017). Kaleido: Enabling Efficient Scientific Data Processing on Big-Data Systems. In 2017 IEEE International Conference on Networking, Architecture, and Storage, NAS 2017 - Proceedings [8026864] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/NAS.2017.8026864

Kaleido : Enabling Efficient Scientific Data Processing on Big-Data Systems. / Biookaghazadeh, Saman; Zhou, Shujia; Zhao, Ming.

2017 IEEE International Conference on Networking, Architecture, and Storage, NAS 2017 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2017. 8026864.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Biookaghazadeh, S, Zhou, S & Zhao, M 2017, Kaleido: Enabling Efficient Scientific Data Processing on Big-Data Systems. in 2017 IEEE International Conference on Networking, Architecture, and Storage, NAS 2017 - Proceedings., 8026864, Institute of Electrical and Electronics Engineers Inc., 2017 IEEE International Conference on Networking, Architecture, and Storage, NAS 2017, Shenzhen, China, 8/7/17. https://doi.org/10.1109/NAS.2017.8026864
Biookaghazadeh S, Zhou S, Zhao M. Kaleido: Enabling Efficient Scientific Data Processing on Big-Data Systems. In 2017 IEEE International Conference on Networking, Architecture, and Storage, NAS 2017 - Proceedings. Institute of Electrical and Electronics Engineers Inc. 2017. 8026864 https://doi.org/10.1109/NAS.2017.8026864
Biookaghazadeh, Saman ; Zhou, Shujia ; Zhao, Ming. / Kaleido : Enabling Efficient Scientific Data Processing on Big-Data Systems. 2017 IEEE International Conference on Networking, Architecture, and Storage, NAS 2017 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2017.
@inproceedings{ef1262d5310d4e0da5c6f22064f85a8d,
title = "Kaleido: Enabling Efficient Scientific Data Processing on Big-Data Systems",
abstract = "Big-Data systems are increasingly important for solving the data-driven problems in many science domains including geosciences. However, existing big-data systems cannot support the efficient processing of self-describing data formats such as NetCDF which are commonly used by scientific communities for data distribution and sharing. This limitation presents a serious hurdle to the further adoption of big-data systems by science domains. This paper presents Kaleido, a solution to this problem by enabling big-data systems to efficiently store and process scientific data. Specifically, it enables Hadoop to directly store NetCDF data on HDFS, and process them in MapReduce using convenient APIs. It also enables Hive to support queries on NetCDF data, transparent to the users. Moreover, it employs optimizations tailored to scientific data, particularly dimension-aware layout which allows efficient execution of subset queries targeting any dimension of the multi-dimensional data. The paper presents a comprehensive evaluation of Kaleido using representative queries on a typical geoscientific dataset. The results show that Kaleido achieves substantial speedup and space saving compared to existing solutions for storing and processing NetCDF data on Hadoop, and it also substantially outperforms the state-of-the-art solutions for supporting subset queries on scientific data.",
author = "Saman Biookaghazadeh and Shujia Zhou and Ming Zhao",
year = "2017",
month = "9",
day = "6",
doi = "10.1109/NAS.2017.8026864",
language = "English (US)",
booktitle = "2017 IEEE International Conference on Networking, Architecture, and Storage, NAS 2017 - Proceedings",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
address = "United States",

}

TY - GEN

T1 - Kaleido

T2 - Enabling Efficient Scientific Data Processing on Big-Data Systems

AU - Biookaghazadeh, Saman

AU - Zhou, Shujia

AU - Zhao, Ming

PY - 2017/9/6

Y1 - 2017/9/6

N2 - Big-Data systems are increasingly important for solving the data-driven problems in many science domains including geosciences. However, existing big-data systems cannot support the efficient processing of self-describing data formats such as NetCDF which are commonly used by scientific communities for data distribution and sharing. This limitation presents a serious hurdle to the further adoption of big-data systems by science domains. This paper presents Kaleido, a solution to this problem by enabling big-data systems to efficiently store and process scientific data. Specifically, it enables Hadoop to directly store NetCDF data on HDFS, and process them in MapReduce using convenient APIs. It also enables Hive to support queries on NetCDF data, transparent to the users. Moreover, it employs optimizations tailored to scientific data, particularly dimension-aware layout which allows efficient execution of subset queries targeting any dimension of the multi-dimensional data. The paper presents a comprehensive evaluation of Kaleido using representative queries on a typical geoscientific dataset. The results show that Kaleido achieves substantial speedup and space saving compared to existing solutions for storing and processing NetCDF data on Hadoop, and it also substantially outperforms the state-of-the-art solutions for supporting subset queries on scientific data.

AB - Big-Data systems are increasingly important for solving the data-driven problems in many science domains including geosciences. However, existing big-data systems cannot support the efficient processing of self-describing data formats such as NetCDF which are commonly used by scientific communities for data distribution and sharing. This limitation presents a serious hurdle to the further adoption of big-data systems by science domains. This paper presents Kaleido, a solution to this problem by enabling big-data systems to efficiently store and process scientific data. Specifically, it enables Hadoop to directly store NetCDF data on HDFS, and process them in MapReduce using convenient APIs. It also enables Hive to support queries on NetCDF data, transparent to the users. Moreover, it employs optimizations tailored to scientific data, particularly dimension-aware layout which allows efficient execution of subset queries targeting any dimension of the multi-dimensional data. The paper presents a comprehensive evaluation of Kaleido using representative queries on a typical geoscientific dataset. The results show that Kaleido achieves substantial speedup and space saving compared to existing solutions for storing and processing NetCDF data on Hadoop, and it also substantially outperforms the state-of-the-art solutions for supporting subset queries on scientific data.

UR - http://www.scopus.com/inward/record.url?scp=85015184100&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85015184100&partnerID=8YFLogxK

U2 - 10.1109/NAS.2017.8026864

DO - 10.1109/NAS.2017.8026864

M3 - Conference contribution

AN - SCOPUS:85015184100

BT - 2017 IEEE International Conference on Networking, Architecture, and Storage, NAS 2017 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

ER -