Objectives: with "stare: spatiotemporal adaptive-resolution encoding" we address focus 2.1.3 "cloud optimized preprocessing and data transformation." current earth science data processing features…

The National Aeronautics and Space Administration (NASA) awarded a $1,267,396.00 cooperative agreement to Rilee Systems Technologies LLC, a for-profit limited liability company, to address Focus Area 2.1.3 "Cloud Optimized Preprocessing and Data Transformation" under the "STARE: Spatiotemporal Adaptive-Resolution Encoding" project. The objective is to simplify the current Earth science data processing flow by moving preprocessing activities to the data archives, eliminating the costs of transferring, creating, and maintaining redundant, idiosyncratic local data archives. The critical new technology being applied is the STARE platform, which provides a unifying approach for diverse data models and allows efficient integration and processing of Earth science data in the cloud. This contract does not have a set-aside designation.

Objectives: with "stare: spatiotemporal adaptive-resolution encoding" we address focus 2.1.3 "cloud optimized preprocessing and data transformation." current earth science data processing features large, centralized archives providing exceptional browse and search capabilities used by researchers who identify then download data in file form to local compute/storage resources for preprocessing and integration prior to analysis. This data flow forces end-users to devote scarce resources to support the transfer, storage, and management of archived data, as well as specialist expertise in the various different kinds of data sets of their research domain. We propose to simplify this flow by moving preprocessing activities to the archived data, eliminating the costs of transferring, creating, and maintaining redundant, idiosyncratic local archives, developed by researchers who are generally not archivists, nor the expert producers of the original data. With stare providing a unifying platform for diverse data models (swath, point, grid), eosdis data archives will be able to produce higher-level products made to order for end-user researchers. Methods: the critical, new technology we apply is stare. Stare's spatial component (SC) has descended from the hierarchical triangular mesh (HTM) spherical indexing originally developed for the sloan digital sky survey, in which storage and computational efficiency was key. The stare/SC recursively divides the earth's surface into a set of quad-trees allowing any point on earth to be identified with a single number. The stare temporal component (TC) has similar properties. For observations, these stare indices contain both location and resolution information, promoting efficient data placement on distributed, cloud resources minimizing costly data transport between nodes for operations such as joining, intersecting, (conditional) subsetting, and RE-gridding diverse datasets. Stare automatically CO-aligns diverse data in the cloud, placing spatiotemporally close data on the same compute/storage node, for a relatively small cost in metadata. Stare thus allows diverse data to be efficiently integrated for analysis in the cloud, providing a foundation on which existing tools and processing methods can be placed. Much capability, e.g. preprocessing, searching, visualization, etc., has been developed to support researchers' use of earth science data. In the course of the proposed work, we will show how existing tools and methods benefit from the stare-enabled platform. This can be via a tight integration as has been done, for example, incorporating stare with the distributed array database scidb, along with RE-gridding functions, and fast parallel, geographic intersections. Or current tools may simply be applied to the results of stare-enabled distributed processing, e.g. fast granule intersection, in a more conventional, but cloud-based, data processing flow. Significance: as a unifying platform, stare supports conventional processing, analysis, and visualization tools, bringing the opportunity for massively increasing the amount of data researchers can use. In the longer term, as tools evolve to take greater advantage of stare's integrative capabilities, we can move from the current focus on the expensive low-level manipulation of data files to an ability to interact with earth science data at a higher level, with query-based declarative tools and user interfaces that favor scientific inquiry rather than data management. At the very least, stare helps automate critical spatiotemporal functions while making efficient use of cloud computing, promising to eliminate the need for researchers to devote time, money, and expertise to the redundant transfer of archived data to their own, local systems. The time and effort saved improves scientific quality and the productivity of the current researchers and reduces the cost-of-entry for others who might seek value from eosdis data resources.