Asynchronous Coupling Framework: DataSpaces

DataSpaces is a data sharing framework that enables dynamic and asynchronous applications interactions. It provides the abstraction of a virtual semantically specialized shared space that can be associatively and asynchronously accessed using simple yet powerful and flexible operators (e.g., put() and get()) with appropriate data selectors or filters. These operators are agnostic of the location, e.g., source/destination, as well as the data distribution and decomposition of the interacting application components. It also provides a runtime system for "in-the-space" data manipulation and/or reduction, using predefined or customized and user-defined functions, which can be dynamically downloaded and executed at runtime while the data is in-transit through the space. DataSpaces has an extensible architecture and can provide new data services, e.g., data subscription and notification.

The DataSpaces framework provides flexible, decoupled and asynchronous data sharing semantics that enables interactions between multiple distributed application services. It easily integrates into the data pipeline of data workflow engines to complement or replace the more traditional file-based approaches. It alleviates the performance penalties associated with these approaches, e.g., latency, variability, by providing transparent and memory-to-memory data sharing.

The key idea underlying the design of DataSpaces is an efficient addressing/lookup mechanism that uses an indexing and addressing scheme that is semantically meaningful to the applications. This addressing scheme is used to index into a dynamic distributed hash table, which is implemented across the nodes of the space, i.e., a relatively small and dynamic partition called the staging area. It automatically load balances data inserted into the space, as well as the data look-up operations. In the current year, we extended DataSpaces to support homogeneous as well as heterogeneous distributed data types and analyzed its behavior while coupling heterogeneous applications. We also explored multiple application scenarios in both high-performance parallel environments as well as using distributed resources. The results obtained demonstrated overall framework scalability, as well as its ability to sustain high data volume traffic for large-scale applications, for example, the data re-distribution in a scenario where one application running on 1024 processors is coupled with a different application running on 8192 processors and the applications exchanged 256GB of data per application iteration for 100 iterations.

References:

1. C. Docan, M. Parashar and S. Klasky, “DataSpaces: An Interaction and Coordination Framework for Coupled Simulation Workflows”, 19th ACM International Symposium on High Performance and Distributed Computing (HPDC'10), Chicago, Illinois, June 2010.
2. C. Docan, F. Zhang, M. Parashar, J. Cummings, N. Podhorszki and S. Klasky, “Experiments with Memory-to-Memory Coupling for End-to-End Fusion Simulation Workflows”, 10th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid'10), Melbourne, Australia, May 2010.
3. F. Zheng, H. Abbasi, C. Docan, J. Lofstead, S. Klasky, Q. Liu, M. Parashar, N. Podhorszki, K. Schwan and M. Wolf, “PreDatA - Preparatory Data Analytics on Peta-Scale Machines”, 24th IEEE International Parallel and Distributed Processing Symposium (IPDPS'10), Atlanta, Georgia, April 2010.
4. J. Cummings, A. Sim, A. Shoshani, J. Lofstead, K. Schwan, C. Docan, M. Parashar, S. Klasky, N. Podhorszki and R. Barreto, “EFFIS: an End-to-end Framework for Fusion Integrated Simulation.”, 18th Euromicro International Conference on Parallel, Distributed and Network-Based Computing (PDP'10), Pisa, Italy, February 2010.
5. N. Podhorszki, S. Klasky, Q. Liu, C. Docan, M. Parashar, H. Abbasi, J. F. Lofstead, K. Schwan, M. Wolf, F. Zheng and J. C. Cummings “Plasma Fusion Code Coupling using Scalable I/O Services and Scientific Workflows”, 4th Workshop on Workflows in Support of Large-Scale Science (WORKS'09), Portland, Oregon, November 2009.