FlexIO: I/O middleware for location-flexible scientific data analytics

Fang Zheng, Hongbo Zou, Greg Eisenhauer, Karsten Schwan, Matthew Wolf, Jai Dayal, Tuan Anh Nguyen, Jianting Cao, Hasan Abbasi, Scott Klasky, Norbert Podhorszki, Hongfeng Yu

Research output: Contribution to conferencePaper

45 Scopus citations

Abstract

Increasingly severe I/O bottlenecks on High-End Computing machines are prompting scientists to process simulation output data online while simulations are running and before storing data on disk. There are several options to place data analytics along the I/O path: on compute nodes, on separate nodes dedicated to analytics, or after data is stored on persistent storage. Since different placements have different impact on performance and cost, there is a consequent need for flexibility in the location of data analytics. The FlexIO middleware described in this paper makes it easy for scientists to obtain such flexibility, by offering simple abstractions and diverse data movement methods to couple simulation with analytics. Various placement policies can be built on top of FlexIO to exploit the trade-offs in performing analytics at different levels of the I/O hierarchy. Experimental results demonstrate that FlexIO can support a variety of simulation and analytics workloads at large scale through flexible placement options, efficient data movement, and dynamic deployment of data manipulation functionalities.

Original languageEnglish (US)
Pages320-331
Number of pages12
DOIs
StatePublished - Oct 7 2013
Event27th IEEE International Parallel and Distributed Processing Symposium, IPDPS 2013 - Boston, MA, United States
Duration: May 20 2013May 24 2013

Other

Other27th IEEE International Parallel and Distributed Processing Symposium, IPDPS 2013
CountryUnited States
CityBoston, MA
Period5/20/135/24/13

    Fingerprint

Keywords

  • Flexibility
  • I/O
  • In Situ Data Analytics
  • Placemen

ASJC Scopus subject areas

  • Software

Cite this

Zheng, F., Zou, H., Eisenhauer, G., Schwan, K., Wolf, M., Dayal, J., Nguyen, T. A., Cao, J., Abbasi, H., Klasky, S., Podhorszki, N., & Yu, H. (2013). FlexIO: I/O middleware for location-flexible scientific data analytics. 320-331. Paper presented at 27th IEEE International Parallel and Distributed Processing Symposium, IPDPS 2013, Boston, MA, United States. https://doi.org/10.1109/IPDPS.2013.46