Efficient execution of the WRF model and other HPC applications in the cloud

Hector A. Duran-Limon, Jesus Flores-Contreras, Nikos Parlavantzas, Ming Zhao, Angel Meulenert-Peña

Research output: Contribution to journalArticle

6 Citations (Scopus)

Abstract

There are many scientific applications that have high performance computing (HPC) demands. Such demands are traditionally supported by cluster- or Grid-based systems. Cloud computing, which has experienced a tremendous growth, emerged as an approach to provide on-demand access to computing resources. The cloud computing paradigm offers a number of advantages over other distributed platforms. For example, the access to resources is flexible and cost-effective since it is not necessary to invest a large amount of money on a computing infrastructure nor pay salaries for maintenance functions. Therefore, the possibility of using cloud computing for running high performance computing applications is attractive. However, it has been shown elsewhere that current cloud computing platforms are not suitable for running some of these kinds of applications since the performance offered is very poor. The reason is mainly the overhead from virtualisation which is extensively used by most cloud computing platforms as a means to optimise resource usage. Furthermore, running HPC applications in current cloud platforms is a complex task that in many cases requires configuring a cluster of virtual machines (VMs). In this paper, we present a lightweight virtualisation approach for efficiently running the Weather Research and Forecasting (WRF) model (a computing- and communication-intensive application) in a cloud computing environment. Our approach also provides a higher-level programming model that automates the process of configuring a cluster of VMs. We assume such a cloud environment can be shared with other types of HPC applications such as mpiBLAST (an embarrassingly parallel application), and MiniFE (a memory-intensive application). Our experimental results show that lightweight virtualisation imposes about 5 % overhead and it substantially outperforms traditional heavyweight virtualisation such as KVM.

Original languageEnglish (US)
Pages (from-to)1-18
Number of pages18
JournalEarth Science Informatics
DOIs
StateAccepted/In press - Mar 14 2016
Externally publishedYes

Fingerprint

weather
resource
infrastructure
communication
cost
demand

Keywords

  • Cloud computing
  • High performance computing
  • Lightweight virtual machines
  • Virtualisation
  • WRF model

ASJC Scopus subject areas

  • Earth and Planetary Sciences(all)

Cite this

Duran-Limon, H. A., Flores-Contreras, J., Parlavantzas, N., Zhao, M., & Meulenert-Peña, A. (Accepted/In press). Efficient execution of the WRF model and other HPC applications in the cloud. Earth Science Informatics, 1-18. https://doi.org/10.1007/s12145-016-0253-7

Efficient execution of the WRF model and other HPC applications in the cloud. / Duran-Limon, Hector A.; Flores-Contreras, Jesus; Parlavantzas, Nikos; Zhao, Ming; Meulenert-Peña, Angel.

In: Earth Science Informatics, 14.03.2016, p. 1-18.

Research output: Contribution to journalArticle

Duran-Limon, Hector A. ; Flores-Contreras, Jesus ; Parlavantzas, Nikos ; Zhao, Ming ; Meulenert-Peña, Angel. / Efficient execution of the WRF model and other HPC applications in the cloud. In: Earth Science Informatics. 2016 ; pp. 1-18.
@article{fc6c318a29af4360b7cb87424be093b4,
title = "Efficient execution of the WRF model and other HPC applications in the cloud",
abstract = "There are many scientific applications that have high performance computing (HPC) demands. Such demands are traditionally supported by cluster- or Grid-based systems. Cloud computing, which has experienced a tremendous growth, emerged as an approach to provide on-demand access to computing resources. The cloud computing paradigm offers a number of advantages over other distributed platforms. For example, the access to resources is flexible and cost-effective since it is not necessary to invest a large amount of money on a computing infrastructure nor pay salaries for maintenance functions. Therefore, the possibility of using cloud computing for running high performance computing applications is attractive. However, it has been shown elsewhere that current cloud computing platforms are not suitable for running some of these kinds of applications since the performance offered is very poor. The reason is mainly the overhead from virtualisation which is extensively used by most cloud computing platforms as a means to optimise resource usage. Furthermore, running HPC applications in current cloud platforms is a complex task that in many cases requires configuring a cluster of virtual machines (VMs). In this paper, we present a lightweight virtualisation approach for efficiently running the Weather Research and Forecasting (WRF) model (a computing- and communication-intensive application) in a cloud computing environment. Our approach also provides a higher-level programming model that automates the process of configuring a cluster of VMs. We assume such a cloud environment can be shared with other types of HPC applications such as mpiBLAST (an embarrassingly parallel application), and MiniFE (a memory-intensive application). Our experimental results show that lightweight virtualisation imposes about 5 {\%} overhead and it substantially outperforms traditional heavyweight virtualisation such as KVM.",
keywords = "Cloud computing, High performance computing, Lightweight virtual machines, Virtualisation, WRF model",
author = "Duran-Limon, {Hector A.} and Jesus Flores-Contreras and Nikos Parlavantzas and Ming Zhao and Angel Meulenert-Pe{\~n}a",
year = "2016",
month = "3",
day = "14",
doi = "10.1007/s12145-016-0253-7",
language = "English (US)",
pages = "1--18",
journal = "Earth Science Informatics",
issn = "1865-0473",
publisher = "Springer Verlag",

}

TY - JOUR

T1 - Efficient execution of the WRF model and other HPC applications in the cloud

AU - Duran-Limon, Hector A.

AU - Flores-Contreras, Jesus

AU - Parlavantzas, Nikos

AU - Zhao, Ming

AU - Meulenert-Peña, Angel

PY - 2016/3/14

Y1 - 2016/3/14

N2 - There are many scientific applications that have high performance computing (HPC) demands. Such demands are traditionally supported by cluster- or Grid-based systems. Cloud computing, which has experienced a tremendous growth, emerged as an approach to provide on-demand access to computing resources. The cloud computing paradigm offers a number of advantages over other distributed platforms. For example, the access to resources is flexible and cost-effective since it is not necessary to invest a large amount of money on a computing infrastructure nor pay salaries for maintenance functions. Therefore, the possibility of using cloud computing for running high performance computing applications is attractive. However, it has been shown elsewhere that current cloud computing platforms are not suitable for running some of these kinds of applications since the performance offered is very poor. The reason is mainly the overhead from virtualisation which is extensively used by most cloud computing platforms as a means to optimise resource usage. Furthermore, running HPC applications in current cloud platforms is a complex task that in many cases requires configuring a cluster of virtual machines (VMs). In this paper, we present a lightweight virtualisation approach for efficiently running the Weather Research and Forecasting (WRF) model (a computing- and communication-intensive application) in a cloud computing environment. Our approach also provides a higher-level programming model that automates the process of configuring a cluster of VMs. We assume such a cloud environment can be shared with other types of HPC applications such as mpiBLAST (an embarrassingly parallel application), and MiniFE (a memory-intensive application). Our experimental results show that lightweight virtualisation imposes about 5 % overhead and it substantially outperforms traditional heavyweight virtualisation such as KVM.

AB - There are many scientific applications that have high performance computing (HPC) demands. Such demands are traditionally supported by cluster- or Grid-based systems. Cloud computing, which has experienced a tremendous growth, emerged as an approach to provide on-demand access to computing resources. The cloud computing paradigm offers a number of advantages over other distributed platforms. For example, the access to resources is flexible and cost-effective since it is not necessary to invest a large amount of money on a computing infrastructure nor pay salaries for maintenance functions. Therefore, the possibility of using cloud computing for running high performance computing applications is attractive. However, it has been shown elsewhere that current cloud computing platforms are not suitable for running some of these kinds of applications since the performance offered is very poor. The reason is mainly the overhead from virtualisation which is extensively used by most cloud computing platforms as a means to optimise resource usage. Furthermore, running HPC applications in current cloud platforms is a complex task that in many cases requires configuring a cluster of virtual machines (VMs). In this paper, we present a lightweight virtualisation approach for efficiently running the Weather Research and Forecasting (WRF) model (a computing- and communication-intensive application) in a cloud computing environment. Our approach also provides a higher-level programming model that automates the process of configuring a cluster of VMs. We assume such a cloud environment can be shared with other types of HPC applications such as mpiBLAST (an embarrassingly parallel application), and MiniFE (a memory-intensive application). Our experimental results show that lightweight virtualisation imposes about 5 % overhead and it substantially outperforms traditional heavyweight virtualisation such as KVM.

KW - Cloud computing

KW - High performance computing

KW - Lightweight virtual machines

KW - Virtualisation

KW - WRF model

UR - http://www.scopus.com/inward/record.url?scp=84961124571&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84961124571&partnerID=8YFLogxK

U2 - 10.1007/s12145-016-0253-7

DO - 10.1007/s12145-016-0253-7

M3 - Article

AN - SCOPUS:84961124571

SP - 1

EP - 18

JO - Earth Science Informatics

JF - Earth Science Informatics

SN - 1865-0473

ER -