Data intensive scientific workflows are at a pivotal time in which traditional local computing resources are no longer capable of meeting the storage or computing demands of scientists. In the earth…

This is a cooperative agreement awarded by NASA to the University Corporation for Atmospheric Research (UCAR), operating as the National Center for Atmospheric Research (NCAR), with a potential value of $465,378.16. The objective is to facilitate the Earth System Sciences (ESS) community's transition to cloud computing by developing technologies that build on existing open-source tools and integrate with the Pangeo ecosystem. Key tasks include deploying a scalable cloud-based JupyterHub, integrating NASA data discovery tools with cloud-based data access protocols, and providing an advanced, cloud-optimized framework for remote data retrieval and processing. The work aims to revolutionize how scientists access and leverage large Earth observation datasets co-located with a range of computing resources in the cloud. There is no set-aside designation for this award.

Data intensive scientific workflows are at a pivotal time in which traditional local computing resources are no longer capable of meeting the storage or computing demands of scientists. In the earth system sciences (ESS) community, we are facing an explosion of data volumes where new datasets, sourced from models, in-situ observations, and remote sensing platforms, are being made available at prohibitively large volumes to store at even medium to large high performance computing (HPC) centers. NASA has estimated that by 2025, IT will be storing upwards of 250 petabytes (PB) of its data using commercial cloud services (e.g. amazon web services [AWS]). Availability of these data in cloud environments, CO-located with a wide range of computing resources, will revolutionize how scientists use these datasets and provide opportunities for important scientific advancements. Fully leveraging these opportunities will require new approaches in the way the ESS community handles data access, processing and analysis. These technologies will be deployable on commercial cloud infrastructure where earth observing system data and information system (eosdis) is anticipated to be stored. At present, tools for working with these datasets consist of convenient interfaces for discovering and downloading data (e.g. NASA's earthdata search) from individual distributed active archive centers (daacs). We anticipate that the transition to cloud storage for many of these daacs will bring immense opportunities and specific challenges to researchers. Our proposal will facilitate the ESS community's transition into cloud computing by developing technologies that build on existing open-source tools (e.g. python, jupyter) by integrating building on top of the growing pangeo ecosystem. Our first task will be to deploy a scalable cloud-based jupyterhub on AWS for community use. Jupyterhub is a multi-user, multilanguage interactive computing environment that facilitates open-ended, exploratory analysis and data visualization. Content ('notebooks') developed on jupyterhub are both functional and fluid; in the manner of an 'executable paper' combining data, processing and interpretation, a necessary departure from traditional publication as a sequence of static artifacts. Our second task will be to integrate existing NASA data discovery tools with cloud based data access protocols. While existing data discovery tools, such as CMR/gibs, provide convenient access to dataset metadata but navigating the access, retrieval, and processing steps for these datasets is left to individual users. We will develop an advanced python API that leverages high-level tools like xarray and dask allowing scientists to accelerate their analysis. Integration of this API with the pangeo ecosystem will provide our API with cutting edge scientific tools for pre-processing, regridding, machine learning, and visualization. Our third task will leverage our advanced API for data discovery and processing to provide an advanced, cloud-optimized framework for remote data retrieval. Our approach to a data retrieval system goes beyond simple slice and download operations (e.g. opendap) and leverages our advanced API for data discovery, access, and processing to also provide server-side perfunctory processing. We will demonstrate the use of these tools with several datasets including north american land data assimilation system (nldas), gravity recovery and climate experiment (grace), and sentinel-1 synthetic aperture radar. The example applications will serve as templates for the broader community and real-world applications for evaluation of the cloud services and applications we develop. We also propose to help accelerate a shift in the ESS culture toward cloud computing by providing short but intensive training opportunities. Our work will provide new ways for scientists to collaborate and make full use of NASA satellite datasets.