A hybrid parallel computing model to support scalable processing of big oceanographic spatial data

Miaomiao Song, WenWen Li, Wenqing Li, Enxiao Liu, Dingfeng Yu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Oceanographic sciences are facing big challenges due to the deluge of big data. As ofa 2010, the amount of new data stored in the world main countries, led by the US, has grown over 7 exabytes. Although the computer hardware is quickly evolving, with faster processor frequency, multi-core technology, and larger memory, traditional reprocessing paradigm on a single-desktop basis still suffers from significant limitations in its low computational efficiency and scalability. In this paper, we report our effort in developing a hybrid parallel computing model which utilizes Graphic Processing Unit (GPU) to accelerate Hadoop Map Reduce system. In each computing node, the actual reprocessing is offloaded from a CPU to a GPU to further boost up the system performance. We describe the architecture design of the proposed model and the automated task/data assignment on each GPU-enabled compute node. Electronic Navigational Charts in ocean fields involves a huge amount of spatio-temporal data. Reprojection of these data between different coordinate reference systems, which is a computation-intensive task, is selected as the use case. Systematic experiments were conducted to demonstrate the good performance of the proposed model.

Original languageEnglish (US)
Title of host publicationGeo-Spatial Knowledge and Intelligence - 4th International Conference on Geo-Informatics in Resource Management and Sustainable Ecosystem, GRMSE 2016, Revised Selected Papers
PublisherSpringer Verlag
Pages276-285
Number of pages10
Volume699
ISBN (Print)9789811039683
DOIs
StatePublished - 2017
Event4th International Conference on Geo-Informatics in Resource Management and Sustainable Ecosystem, GRMSE 2016 - Kowloon, Hong Kong
Duration: Nov 18 2016Nov 20 2016

Publication series

NameCommunications in Computer and Information Science
Volume699
ISSN (Print)18650929

Other

Other4th International Conference on Geo-Informatics in Resource Management and Sustainable Ecosystem, GRMSE 2016
CountryHong Kong
City Kowloon
Period11/18/1611/20/16

Fingerprint

Parallel processing systems
Processing
Computational efficiency
Computer hardware
Program processors
Scalability
Data storage equipment
Graphics processing unit
Experiments

Keywords

  • Coordinate projection
  • GPU general computing
  • Hadoop MapReduce
  • Oceanographic spatial data
  • Parallel computing

ASJC Scopus subject areas

  • Computer Science(all)

Cite this

Song, M., Li, W., Li, W., Liu, E., & Yu, D. (2017). A hybrid parallel computing model to support scalable processing of big oceanographic spatial data. In Geo-Spatial Knowledge and Intelligence - 4th International Conference on Geo-Informatics in Resource Management and Sustainable Ecosystem, GRMSE 2016, Revised Selected Papers (Vol. 699, pp. 276-285). (Communications in Computer and Information Science; Vol. 699). Springer Verlag. https://doi.org/10.1007/978-981-10-3969-0_32

A hybrid parallel computing model to support scalable processing of big oceanographic spatial data. / Song, Miaomiao; Li, WenWen; Li, Wenqing; Liu, Enxiao; Yu, Dingfeng.

Geo-Spatial Knowledge and Intelligence - 4th International Conference on Geo-Informatics in Resource Management and Sustainable Ecosystem, GRMSE 2016, Revised Selected Papers. Vol. 699 Springer Verlag, 2017. p. 276-285 (Communications in Computer and Information Science; Vol. 699).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Song, M, Li, W, Li, W, Liu, E & Yu, D 2017, A hybrid parallel computing model to support scalable processing of big oceanographic spatial data. in Geo-Spatial Knowledge and Intelligence - 4th International Conference on Geo-Informatics in Resource Management and Sustainable Ecosystem, GRMSE 2016, Revised Selected Papers. vol. 699, Communications in Computer and Information Science, vol. 699, Springer Verlag, pp. 276-285, 4th International Conference on Geo-Informatics in Resource Management and Sustainable Ecosystem, GRMSE 2016, Kowloon, Hong Kong, 11/18/16. https://doi.org/10.1007/978-981-10-3969-0_32
Song M, Li W, Li W, Liu E, Yu D. A hybrid parallel computing model to support scalable processing of big oceanographic spatial data. In Geo-Spatial Knowledge and Intelligence - 4th International Conference on Geo-Informatics in Resource Management and Sustainable Ecosystem, GRMSE 2016, Revised Selected Papers. Vol. 699. Springer Verlag. 2017. p. 276-285. (Communications in Computer and Information Science). https://doi.org/10.1007/978-981-10-3969-0_32
Song, Miaomiao ; Li, WenWen ; Li, Wenqing ; Liu, Enxiao ; Yu, Dingfeng. / A hybrid parallel computing model to support scalable processing of big oceanographic spatial data. Geo-Spatial Knowledge and Intelligence - 4th International Conference on Geo-Informatics in Resource Management and Sustainable Ecosystem, GRMSE 2016, Revised Selected Papers. Vol. 699 Springer Verlag, 2017. pp. 276-285 (Communications in Computer and Information Science).
@inproceedings{90bfaca13a754f43beeb5b1cca15899e,
title = "A hybrid parallel computing model to support scalable processing of big oceanographic spatial data",
abstract = "Oceanographic sciences are facing big challenges due to the deluge of big data. As ofa 2010, the amount of new data stored in the world main countries, led by the US, has grown over 7 exabytes. Although the computer hardware is quickly evolving, with faster processor frequency, multi-core technology, and larger memory, traditional reprocessing paradigm on a single-desktop basis still suffers from significant limitations in its low computational efficiency and scalability. In this paper, we report our effort in developing a hybrid parallel computing model which utilizes Graphic Processing Unit (GPU) to accelerate Hadoop Map Reduce system. In each computing node, the actual reprocessing is offloaded from a CPU to a GPU to further boost up the system performance. We describe the architecture design of the proposed model and the automated task/data assignment on each GPU-enabled compute node. Electronic Navigational Charts in ocean fields involves a huge amount of spatio-temporal data. Reprojection of these data between different coordinate reference systems, which is a computation-intensive task, is selected as the use case. Systematic experiments were conducted to demonstrate the good performance of the proposed model.",
keywords = "Coordinate projection, GPU general computing, Hadoop MapReduce, Oceanographic spatial data, Parallel computing",
author = "Miaomiao Song and WenWen Li and Wenqing Li and Enxiao Liu and Dingfeng Yu",
year = "2017",
doi = "10.1007/978-981-10-3969-0_32",
language = "English (US)",
isbn = "9789811039683",
volume = "699",
series = "Communications in Computer and Information Science",
publisher = "Springer Verlag",
pages = "276--285",
booktitle = "Geo-Spatial Knowledge and Intelligence - 4th International Conference on Geo-Informatics in Resource Management and Sustainable Ecosystem, GRMSE 2016, Revised Selected Papers",
address = "Germany",

}

TY - GEN

T1 - A hybrid parallel computing model to support scalable processing of big oceanographic spatial data

AU - Song, Miaomiao

AU - Li, WenWen

AU - Li, Wenqing

AU - Liu, Enxiao

AU - Yu, Dingfeng

PY - 2017

Y1 - 2017

N2 - Oceanographic sciences are facing big challenges due to the deluge of big data. As ofa 2010, the amount of new data stored in the world main countries, led by the US, has grown over 7 exabytes. Although the computer hardware is quickly evolving, with faster processor frequency, multi-core technology, and larger memory, traditional reprocessing paradigm on a single-desktop basis still suffers from significant limitations in its low computational efficiency and scalability. In this paper, we report our effort in developing a hybrid parallel computing model which utilizes Graphic Processing Unit (GPU) to accelerate Hadoop Map Reduce system. In each computing node, the actual reprocessing is offloaded from a CPU to a GPU to further boost up the system performance. We describe the architecture design of the proposed model and the automated task/data assignment on each GPU-enabled compute node. Electronic Navigational Charts in ocean fields involves a huge amount of spatio-temporal data. Reprojection of these data between different coordinate reference systems, which is a computation-intensive task, is selected as the use case. Systematic experiments were conducted to demonstrate the good performance of the proposed model.

AB - Oceanographic sciences are facing big challenges due to the deluge of big data. As ofa 2010, the amount of new data stored in the world main countries, led by the US, has grown over 7 exabytes. Although the computer hardware is quickly evolving, with faster processor frequency, multi-core technology, and larger memory, traditional reprocessing paradigm on a single-desktop basis still suffers from significant limitations in its low computational efficiency and scalability. In this paper, we report our effort in developing a hybrid parallel computing model which utilizes Graphic Processing Unit (GPU) to accelerate Hadoop Map Reduce system. In each computing node, the actual reprocessing is offloaded from a CPU to a GPU to further boost up the system performance. We describe the architecture design of the proposed model and the automated task/data assignment on each GPU-enabled compute node. Electronic Navigational Charts in ocean fields involves a huge amount of spatio-temporal data. Reprojection of these data between different coordinate reference systems, which is a computation-intensive task, is selected as the use case. Systematic experiments were conducted to demonstrate the good performance of the proposed model.

KW - Coordinate projection

KW - GPU general computing

KW - Hadoop MapReduce

KW - Oceanographic spatial data

KW - Parallel computing

UR - http://www.scopus.com/inward/record.url?scp=85014879160&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85014879160&partnerID=8YFLogxK

U2 - 10.1007/978-981-10-3969-0_32

DO - 10.1007/978-981-10-3969-0_32

M3 - Conference contribution

SN - 9789811039683

VL - 699

T3 - Communications in Computer and Information Science

SP - 276

EP - 285

BT - Geo-Spatial Knowledge and Intelligence - 4th International Conference on Geo-Informatics in Resource Management and Sustainable Ecosystem, GRMSE 2016, Revised Selected Papers

PB - Springer Verlag

ER -