Cooperative Agreement 80NSSC18M0159

DATA INTENSIVE SCIENTIFIC WORKFLOWS ARE AT A PIVOTAL TIME IN WHICH TRADITIONAL LOCAL COMPUTING RESOURCES ARE NO LONGER CAPABLE OF MEETING THE STORAGE OR COMPUTING DEMANDS OF SCIENTISTS. IN THE EARTH SYSTEM SCIENCES (ESS) COMMUNITY WE ARE FACING AN EXPLOSION OF DATA VOLUMES WHERE NEW DATASETS SOURCED FROM MODELS IN-SITU OBSERVATIONS AND REMOTE SENSING PLATFORMS ARE BEING MADE AVAILABLE AT PROHIBITIVELY LARGE VOLUMES TO STORE AT EVEN MEDIUM TO LARGE HIGH PERFORMANCE COMPUTING (HPC) CENTERS. NASA HAS ESTIMATED THAT BY 2025 IT WILL BE STORING UPWARDS OF 250 PETABYTES (PB) OF ITS DATA USING COMMERCIAL CLOUD SERVICES (E.G. AMAZON WEB SERVICES [AWS]). AVAILABILITY OF THESE DATA IN CLOUD ENVIRONMENTS CO-LOCATED WITH A WIDE RANGE OF COMPUTING RESOURCES WILL REVOLUTIONIZE HOW SCIENTISTS USE THESE DATASETS AND PROVIDE OPPORTUNITIES FOR IMPORTANT SCIENTIFIC ADVANCEMENTS. FULLY LEVERAGING THESE OPPORTUNITIES WILL REQUIRE NEW APPROACHES IN THE WAY THE ESS COMMUNITY HANDLES DATA ACCESS PROCESSING AND ANALYSIS. THESE TECHNOLOGIES WILL BE DEPLOYABLE ON COMMERCIAL CLOUD INFRASTRUCTURE WHERE EARTH OBSERVING SYSTEM DATA AND INFORMATION SYSTEM (EOSDIS) IS ANTICIPATED TO BE STORED. AT PRESENT TOOLS FOR WORKING WITH THESE DATASETS CONSIST OF CONVENIENT INTERFACES FOR DISCOVERING AND DOWNLOADING DATA (E.G. NASA'S EARTHDATA SEARCH) FROM INDIVIDUAL DISTRIBUTED ACTIVE ARCHIVE CENTERS (DAACS). WE ANTICIPATE THAT THE TRANSITION TO CLOUD STORAGE FOR MANY OF THESE DAACS WILL BRING IMMENSE OPPORTUNITIES AND SPECIFIC CHALLENGES TO RESEARCHERS. OUR PROPOSAL WILL FACILITATE THE ESS COMMUNITY'S TRANSITION INTO CLOUD COMPUTING BY DEVELOPING TECHNOLOGIES THAT BUILD ON EXISTING OPEN-SOURCE TOOLS (E.G. PYTHON JUPYTER) BY INTEGRATING BUILDING ON TOP OF THE GROWING PANGEO ECOSYSTEM. OUR FIRST TASK WILL BE TO DEPLOY A SCALABLE CLOUD-BASED JUPYTERHUB ON AWS FOR COMMUNITY USE. JUPYTERHUB IS A MULTI-USER MULTILANGUAGE INTERACTIVE COMPUTING ENVIRONMENT THAT FACILITATES OPEN-ENDED EXPLORATORY ANALYSIS AND DATA VISUALIZATION. CONTENT ('NOTEBOOKS') DEVELOPED ON JUPYTERHUB ARE BOTH FUNCTIONAL AND FLUID; IN THE MANNER OF AN 'EXECUTABLE PAPER' COMBINING DATA PROCESSING AND INTERPRETATION A NECESSARY DEPARTURE FROM TRADITIONAL PUBLICATION AS A SEQUENCE OF STATIC ARTIFACTS. OUR SECOND TASK WILL BE TO INTEGRATE EXISTING NASA DATA DISCOVERY TOOLS WITH CLOUD BASED DATA ACCESS PROTOCOLS. WHILE EXISTING DATA DISCOVERY TOOLS SUCH AS CMR/GIBS PROVIDE CONVENIENT ACCESS TO DATASET METADATA BUT NAVIGATING THE ACCESS RETRIEVAL AND PROCESSING STEPS FOR THESE DATASETS IS LEFT TO INDIVIDUAL USERS. WE WILL DEVELOP AN ADVANCED PYTHON API THAT LEVERAGES HIGH-LEVEL TOOLS LIKE XARRAY AND DASK ALLOWING SCIENTISTS TO ACCELERATE THEIR ANALYSIS. INTEGRATION OF THIS API WITH THE PANGEO ECOSYSTEM WILL PROVIDE OUR API WITH CUTTING EDGE SCIENTIFIC TOOLS FOR PRE-PROCESSING REGRIDDING MACHINE LEARNING AND VISUALIZATION. OUR THIRD TASK WILL LEVERAGE OUR ADVANCED API FOR DATA DISCOVERY AND PROCESSING TO PROVIDE AN ADVANCED CLOUD-OPTIMIZED FRAMEWORK FOR REMOTE DATA RETRIEVAL. OUR APPROACH TO A DATA RETRIEVAL SYSTEM GOES BEYOND SIMPLE SLICE AND DOWNLOAD OPERATIONS (E.G. OPENDAP) AND LEVERAGES OUR ADVANCED API FOR DATA DISCOVERY ACCESS AND PROCESSING TO ALSO PROVIDE SERVER-SIDE PERFUNCTORY PROCESSING. WE WILL DEMONSTRATE THE USE OF THESE TOOLS WITH SEVERAL DATASETS INCLUDING NORTH AMERICAN LAND DATA ASSIMILATION SYSTEM (NLDAS) GRAVITY RECOVERY AND CLIMATE EXPERIMENT (GRACE) AND SENTINEL-1 SYNTHETIC APERTURE RADAR. THE EXAMPLE APPLICATIONS WILL SERVE AS TEMPLATES FOR THE BROADER COMMUNITY AND REAL-WORLD APPLICATIONS FOR EVALUATION OF THE CLOUD SERVICES AND APPLICATIONS WE DEVELOP. WE ALSO PROPOSE TO HELP ACCELERATE A SHIFT IN THE ESS CULTURE TOWARD CLOUD COMPUTING BY PROVIDING SHORT BUT INTENSIVE TRAINING OPPORTUNITIES. OUR WORK WILL PROVIDE NEW WAYS FOR SCIENTISTS TO COLLABORATE AND MAKE FULL USE OF NASA SATELLITE DATASETS.