Leveraging STARE for Co-aligned data locality with netCDF and python MPI

Kwo Sen Kuo; Hongfeng Yu; Yu Pan; Michael L. Rilee

doi:10.1109/IGARSS.2019.8900423

Leveraging STARE for Co-aligned data locality with netCDF and python MPI

Kwo Sen Kuo, Hongfeng Yu, Yu Pan, Michael L. Rilee

Computer Science and Engineering

Research output: Contribution to conference › Paper › peer-review

4 Scopus citations

Abstract

We have leveraged STARE indexing to package partitioned data chunks from diverse datasets into netCDF files, distributed them on a cluster of 16 lightweight nodes with their placements spatiotemporally co-aligned, and demonstrated a few integrative analyses using netCDF parallel I/O and Python MPI, with single-user performance and scalability comparable to, or even better than, that of a parallel array database management system (ADBMS) such as SciDB. However, records of the node location and STARE index ranges for each data chunk, similar to the chunk maps of SciDB, must be maintained and consulted by the I/O and analysis code for coordinating the analytic operations in parallel, in order to achieve the good performance and scalability.

Original language	English (US)
Pages	10063-10066
Number of pages	4
DOIs	https://doi.org/10.1109/IGARSS.2019.8900423
State	Published - 2019
Event	39th IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2019 - Yokohama, Japan Duration: Jul 28 2019 → Aug 2 2019

Conference

Conference	39th IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2019
Country/Territory	Japan
City	Yokohama
Period	7/28/19 → 8/2/19

Keywords

Big Data
Data-intensive analysis
Interoperability
Parallel processing
Scalability

ASJC Scopus subject areas

Computer Science Applications
General Earth and Planetary Sciences

Access to Document

10.1109/IGARSS.2019.8900423

Cite this

@conference{0b069d7d549c49a38c2d9b1fc09f4b07,

title = "Leveraging STARE for Co-aligned data locality with netCDF and python MPI",

abstract = "We have leveraged STARE indexing to package partitioned data chunks from diverse datasets into netCDF files, distributed them on a cluster of 16 lightweight nodes with their placements spatiotemporally co-aligned, and demonstrated a few integrative analyses using netCDF parallel I/O and Python MPI, with single-user performance and scalability comparable to, or even better than, that of a parallel array database management system (ADBMS) such as SciDB. However, records of the node location and STARE index ranges for each data chunk, similar to the chunk maps of SciDB, must be maintained and consulted by the I/O and analysis code for coordinating the analytic operations in parallel, in order to achieve the good performance and scalability.",

keywords = "Big Data, Data-intensive analysis, Interoperability, Parallel processing, Scalability",

author = "Kuo, {Kwo Sen} and Hongfeng Yu and Yu Pan and Rilee, {Michael L.}",

note = "Funding Information: Acknowledgement – We are grateful for the support provided by NASA Advancing Collaborative Connections for Earth System Science (ACCESS) program, NSF EarthCube program, and NASA Advanced Information Systems Technology (AIST) program that made this research and development possible. Publisher Copyright: {\textcopyright}2019 IEEE; 39th IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2019 ; Conference date: 28-07-2019 Through 02-08-2019",

year = "2019",

doi = "10.1109/IGARSS.2019.8900423",

language = "English (US)",

pages = "10063--10066",

}

TY - CONF

T1 - Leveraging STARE for Co-aligned data locality with netCDF and python MPI

AU - Kuo, Kwo Sen

AU - Yu, Hongfeng

AU - Pan, Yu

AU - Rilee, Michael L.

N1 - Funding Information: Acknowledgement – We are grateful for the support provided by NASA Advancing Collaborative Connections for Earth System Science (ACCESS) program, NSF EarthCube program, and NASA Advanced Information Systems Technology (AIST) program that made this research and development possible. Publisher Copyright: ©2019 IEEE

PY - 2019

Y1 - 2019

N2 - We have leveraged STARE indexing to package partitioned data chunks from diverse datasets into netCDF files, distributed them on a cluster of 16 lightweight nodes with their placements spatiotemporally co-aligned, and demonstrated a few integrative analyses using netCDF parallel I/O and Python MPI, with single-user performance and scalability comparable to, or even better than, that of a parallel array database management system (ADBMS) such as SciDB. However, records of the node location and STARE index ranges for each data chunk, similar to the chunk maps of SciDB, must be maintained and consulted by the I/O and analysis code for coordinating the analytic operations in parallel, in order to achieve the good performance and scalability.

AB - We have leveraged STARE indexing to package partitioned data chunks from diverse datasets into netCDF files, distributed them on a cluster of 16 lightweight nodes with their placements spatiotemporally co-aligned, and demonstrated a few integrative analyses using netCDF parallel I/O and Python MPI, with single-user performance and scalability comparable to, or even better than, that of a parallel array database management system (ADBMS) such as SciDB. However, records of the node location and STARE index ranges for each data chunk, similar to the chunk maps of SciDB, must be maintained and consulted by the I/O and analysis code for coordinating the analytic operations in parallel, in order to achieve the good performance and scalability.

KW - Big Data

KW - Data-intensive analysis

KW - Interoperability

KW - Parallel processing

KW - Scalability

UR - http://www.scopus.com/inward/record.url?scp=85099975885&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85099975885&partnerID=8YFLogxK

U2 - 10.1109/IGARSS.2019.8900423

DO - 10.1109/IGARSS.2019.8900423

M3 - Paper

AN - SCOPUS:85099975885

SP - 10063

EP - 10066

T2 - 39th IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2019

Y2 - 28 July 2019 through 2 August 2019

ER -

Leveraging STARE for Co-aligned data locality with netCDF and python MPI

Abstract

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this