Spatial data management in apache spark: the GeoSpark perspective and beyond

Jia Yu, Zongsi Zhang, Mohamed Elsayed

Research output: Contribution to journalArticle

19 Scopus citations

Abstract

The paper presents the details of designing and developing GeoSpark, which extends the core engine of Apache Spark and SparkSQL to support spatial data types, indexes, and geometrical operations at scale. The paper also gives a detailed analysis of the technical challenges and opportunities of extending Apache Spark to support state-of-the-art spatial data partitioning techniques: uniform grid, R-tree, Quad-Tree, and KDB-Tree. The paper also shows how building local spatial indexes, e.g., R-Tree or Quad-Tree, on each Spark data partition can speed up the local computation and hence decrease the overall runtime of the spatial analytics program. Furthermore, the paper introduces a comprehensive experiment analysis that surveys and experimentally evaluates the performance of running de-facto spatial operations like spatial range, spatial K-Nearest Neighbors (KNN), and spatial join queries in the Apache Spark ecosystem. Extensive experiments on real spatial datasets show that GeoSpark achieves up to two orders of magnitude faster run time performance than existing Hadoop-based systems and up to an order of magnitude faster performance than Spark-based systems.

Original languageEnglish (US)
JournalGeoInformatica
DOIs
StateAccepted/In press - Jan 1 2018

Keywords

  • Big geospatial data
  • Distributed computing
  • Spatial databases

ASJC Scopus subject areas

  • Information Systems
  • Geography, Planning and Development

Fingerprint Dive into the research topics of 'Spatial data management in apache spark: the GeoSpark perspective and beyond'. Together they form a unique fingerprint.

  • Cite this