Modern science requires studying datasets so large that they can overwhelm most work flows due to the excessive costs of data movements. This problem is particularly prominent at largescale high…

This is a $100,000 cooperative agreement awarded by the National Aeronautics and Space Administration (NASA) Shared Services Center to the University of Utah's Office of Sponsored Projects Division. The contract is focused on developing a framework to enable efficient scientific computations by progressively transferring only the most relevant data to meet desired error bounds and constraints, while considering the underlying heterogeneous system architecture. The framework will include a novel data model for storing and working with scalar and vector fields, as well as a runtime system that automates data reduction and resource allocation decisions for in-situ application workflows. This contract is not associated with a set-aside designation and has a performance period ending on November 30, 2025. The University of Utah is a major recipient of federal research funding, conducting cutting-edge work across scientific and medical disciplines.

Modern science requires studying datasets so large that they can overwhelm most work flows due to the excessive costs of data movements. This problem is particularly prominent at largescale high performance computing centers tackling hard scientific and engineering challenges such as NASA advanced supercomputing (NAS), where scientific simulations frequently generate terabytes to petabytes or more of data per run. Moreover, such simulations are often just components of a broader ecosystem involving the acquisition, processing, and transmission of large amounts of data acquired from sensing devices deployed in the field or carefully controlled experimental facilities. Although lossy data compression can be a key aspect in addressing this problem, most existing approaches still incur superfluous data movements not tailored to the requirements of the scientific task at hand in terms of error tolerance, hardware, space, or time constraints, leading to inefficient computations that hamper scientific progress. In addition, the complexity of making science codes and compression codes to execute insitu and interoperate is exacerbated by diverse machines employing heterogeneous hardware, such as the pleiades supercomputer at NAS. To empower efficient scientific computations, we propose a data reduction framework that progressively transfers only the most relevant data to meet the desired error bounds and other constraints for the task at hand while being aware of the underlying heterogeneous system architecture of the computing facility. In particular, the framework will advance the field of scientific data reduction in two main research areas: 1. Development of a novel data model for (ondisk) storing and (inmemory) working with scalar and vector fields that simultaneously achieves high compression ratios and efficient progressive streaming in both precision and resolution. 2. Development of a novel runtime system, based on dataspaces framework, that automates data reduction and resource allocation decisions for insitu application workflows, adaptively satisfying a range of user, application, and system constraints. Although low-resolution or low precision data approximations are often enough to answer a specific scientific question with satisfactory accuracy, different questions may require different types and levels of approximations. However, most existing data compression approaches either require recompression for each different approximation or employ a fixed decompression order regardless of the task at hand, leading to excessive data movements. Thus, we propose a flexible data model that allows progressive decoding of data chunks to form approximations at any desired combination of resolution and precision, all from a single compressed data layout. In our ongoing research, we have developed a prototypical data model which not only enables tradeoffs in the precision resolution space but is also highly competitive in compression ratio against state-of-the-art compressors while using orders of magnitude less memory and time. To fully access the effectiveness of our compressed data structures, we will complement them with algorithms that include both fundamental data processing routines and real-world scientific analysis pipelines, which will also be used to demonstrate the efficacy of our data model in practice. Leveraging the flexibility of our progressive data reduction involves making online decisions about the choice of reduction method and associated parameters based on application and system constraints. These constraints include accuracy, resolution, errors introduced, impact on application performance, or required systems resources such as memory, i/o, and CPU (or accelerator) cores. Therefore, IT is key for US to explore and develop an autonomic runtime system for the dynamic scheduling of data reduction tasks and effective data management on diverse computational hardware.