15 Citations (Scopus)

Abstract

Light detection and ranging (LiDAR) data are essential for scientific discoveries such as Earth and ecological sciences, environmental applications, and responding to natural disasters. While collecting LiDAR data over large areas is quite possible the subsequent processing steps typically involve large computational demands. Efficiently storing, managing, and processing LiDAR data are the prerequisite steps for enabling these LiDAR-based applications. However, handling LiDAR data poses grand geoprocessing challenges due to data and computational intensity. To tackle such challenges, we developed a general-purpose scalable framework coupled with a sophisticated data decomposition and parallelization strategy to efficiently handle ‘big’ LiDAR data collections. The contributions of this research were (1) a tile-based spatial index to manage big LiDAR data in the scalable and fault-tolerable Hadoop distributed file system, (2) two spatial decomposition techniques to enable efficient parallelization of different types of LiDAR processing tasks, and (3) by coupling existing LiDAR processing tools with Hadoop, a variety of LiDAR data processing tasks can be conducted in parallel in a highly scalable distributed computing environment using an online geoprocessing application. A proof-of-concept prototype is presented here to demonstrate the feasibility, performance, and scalability of the proposed framework.

Original languageEnglish (US)
Pages (from-to)1-22
Number of pages22
JournalInternational Journal of Digital Earth
DOIs
StateAccepted/In press - Dec 23 2016

Fingerprint

Processing
detection
decomposition
Decomposition
Distributed computer systems
natural disaster
Tile
Disasters
Scalability
Earth (planet)

Keywords

  • Big data
  • Hadoop MapReduce
  • LAStools
  • online geoprocessing
  • parallel
  • spatial decomposition

ASJC Scopus subject areas

  • Software
  • Computer Science Applications
  • Earth and Planetary Sciences(all)

Cite this

A general-purpose framework for parallel processing of large-scale LiDAR data. / Li, Zhenlong; Hodgson, Michael E.; Li, WenWen.

In: International Journal of Digital Earth, 23.12.2016, p. 1-22.

Research output: Contribution to journalArticle

@article{119feffc6e3945bf9f117285c1c81ae5,
title = "A general-purpose framework for parallel processing of large-scale LiDAR data",
abstract = "Light detection and ranging (LiDAR) data are essential for scientific discoveries such as Earth and ecological sciences, environmental applications, and responding to natural disasters. While collecting LiDAR data over large areas is quite possible the subsequent processing steps typically involve large computational demands. Efficiently storing, managing, and processing LiDAR data are the prerequisite steps for enabling these LiDAR-based applications. However, handling LiDAR data poses grand geoprocessing challenges due to data and computational intensity. To tackle such challenges, we developed a general-purpose scalable framework coupled with a sophisticated data decomposition and parallelization strategy to efficiently handle ‘big’ LiDAR data collections. The contributions of this research were (1) a tile-based spatial index to manage big LiDAR data in the scalable and fault-tolerable Hadoop distributed file system, (2) two spatial decomposition techniques to enable efficient parallelization of different types of LiDAR processing tasks, and (3) by coupling existing LiDAR processing tools with Hadoop, a variety of LiDAR data processing tasks can be conducted in parallel in a highly scalable distributed computing environment using an online geoprocessing application. A proof-of-concept prototype is presented here to demonstrate the feasibility, performance, and scalability of the proposed framework.",
keywords = "Big data, Hadoop MapReduce, LAStools, online geoprocessing, parallel, spatial decomposition",
author = "Zhenlong Li and Hodgson, {Michael E.} and WenWen Li",
year = "2016",
month = "12",
day = "23",
doi = "10.1080/17538947.2016.1269842",
language = "English (US)",
pages = "1--22",
journal = "International Journal of Digital Earth",
issn = "1753-8947",
publisher = "Taylor and Francis Ltd.",

}

TY - JOUR

T1 - A general-purpose framework for parallel processing of large-scale LiDAR data

AU - Li, Zhenlong

AU - Hodgson, Michael E.

AU - Li, WenWen

PY - 2016/12/23

Y1 - 2016/12/23

N2 - Light detection and ranging (LiDAR) data are essential for scientific discoveries such as Earth and ecological sciences, environmental applications, and responding to natural disasters. While collecting LiDAR data over large areas is quite possible the subsequent processing steps typically involve large computational demands. Efficiently storing, managing, and processing LiDAR data are the prerequisite steps for enabling these LiDAR-based applications. However, handling LiDAR data poses grand geoprocessing challenges due to data and computational intensity. To tackle such challenges, we developed a general-purpose scalable framework coupled with a sophisticated data decomposition and parallelization strategy to efficiently handle ‘big’ LiDAR data collections. The contributions of this research were (1) a tile-based spatial index to manage big LiDAR data in the scalable and fault-tolerable Hadoop distributed file system, (2) two spatial decomposition techniques to enable efficient parallelization of different types of LiDAR processing tasks, and (3) by coupling existing LiDAR processing tools with Hadoop, a variety of LiDAR data processing tasks can be conducted in parallel in a highly scalable distributed computing environment using an online geoprocessing application. A proof-of-concept prototype is presented here to demonstrate the feasibility, performance, and scalability of the proposed framework.

AB - Light detection and ranging (LiDAR) data are essential for scientific discoveries such as Earth and ecological sciences, environmental applications, and responding to natural disasters. While collecting LiDAR data over large areas is quite possible the subsequent processing steps typically involve large computational demands. Efficiently storing, managing, and processing LiDAR data are the prerequisite steps for enabling these LiDAR-based applications. However, handling LiDAR data poses grand geoprocessing challenges due to data and computational intensity. To tackle such challenges, we developed a general-purpose scalable framework coupled with a sophisticated data decomposition and parallelization strategy to efficiently handle ‘big’ LiDAR data collections. The contributions of this research were (1) a tile-based spatial index to manage big LiDAR data in the scalable and fault-tolerable Hadoop distributed file system, (2) two spatial decomposition techniques to enable efficient parallelization of different types of LiDAR processing tasks, and (3) by coupling existing LiDAR processing tools with Hadoop, a variety of LiDAR data processing tasks can be conducted in parallel in a highly scalable distributed computing environment using an online geoprocessing application. A proof-of-concept prototype is presented here to demonstrate the feasibility, performance, and scalability of the proposed framework.

KW - Big data

KW - Hadoop MapReduce

KW - LAStools

KW - online geoprocessing

KW - parallel

KW - spatial decomposition

UR - http://www.scopus.com/inward/record.url?scp=85008349838&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85008349838&partnerID=8YFLogxK

U2 - 10.1080/17538947.2016.1269842

DO - 10.1080/17538947.2016.1269842

M3 - Article

AN - SCOPUS:85008349838

SP - 1

EP - 22

JO - International Journal of Digital Earth

JF - International Journal of Digital Earth

SN - 1753-8947

ER -