SOW_Next Generation Open and Scalable Data Analysis Processing and Visualization Solution_SOW.pdf

Work Statement (23APR23) I. Generalized work statement.

This section delineates the generalized work to be performed. As this work pertains especially to NASA’s Earth Data Science, new strategies are required in order to expand collaboration and develop advanced concepts to meet the evolving and futuristic data needs of the research community. These tasks focus on Earth data science needs including improving science data acquisition, improving the management and stewardship of science data, novel approaches to analyze and visualize science data, and improving the use of NASA science data by other government agencies, and national and international research organizations. Highly specialized large data science techniques are required to accomplish the development and research needed for this task.

A. The contractor shall develop solutions for:

1. Integration of NASA’s specialized Earth science data, services and tools into custom research application and applications workflows;

2. Infusing trends across the informatics, data science and information technology fields;

3. Developing new, effective solutions for science data management and dissemination;

4. Rapid prototyping, development, and testing of advanced concepts in scientific data and information systems tackling search, access and use challenges;

5. Developing systematic approaches to data management and data stewardship supporting distributed archive centers;

6. Providing system architecture analysis for the Inter Agency Implementation and Advanced Concepts Team (IMPACT); and

7. Providing data curation for collaborations within NASA, with other national and international government organizations, industry, and academic partners.

B. To develop these solutions, the contractor shall provide resources to ensure IMPACT maintains cutting-edge leadership in:

1. Science informatics;

2. Artificial intelligence, including machine learning expertise with specialization in spatio-temporal algorithms;

3. Science data systems and domain science; and

4. Scientific data storage, archiving, and management including evolving cloud applications.

II. IMPACT projects associated with this contract.

IMPACT is a collaborative, interdisciplinary team of civil service and contractors that works to further NASA’s Earth Science Data System (ESDS) Program goal of overseeing the lifecycle of Earth science data to maximize the scientific return of NASA's missions and experiments for research and applied scientists, decision makers, and the society at large. IMPACT’s three focus areas are interagency collaboration, assessment and evaluation, and research and development of advanced concepts.

The IMPACT portfolio is comprised of approximately 22 projects. These projects have diverse funding sources; therefore, monthly reporting and invoicing shall provide delineation of the respective projects. Specific project tasks are assigned on an annual basis. All developed software solutions first produced in the performance of this contract shall be delivered to the Government and shall not contain any unauthorized restrictive markings in accordance with the contract’s data rights clause and in furtherance of NASA’s interest in including such software in the NASA Open Source Catalog.

IMPACT works in a seamless environment generally characterized by each project being comprised of members from the civil service, the University of Alabama in Huntsville from a collaborative agreement, the Universities Space Research Association from a collaborative agreement, other contractors, and research and development team members from this contract.

Each project's collaborative, agile statement of work is updated on an annual basis identifying the total effort and milestones planned for the fiscal year. IMPACT utilizes the Scaled Agile Framework (SAFe®) which is synchronized with our Headquarters customers.

Current IMPACT projects requiring research and development from this contract include:

A. System Architecture and Earth Observation (EO) Data on the Cloud (EODC) B. NASA's Visualization, Exploration, and Data Analysis (VEDA) Project C. Scalable Science Data Processing D. System Operations, Maintenance, and Scalability E. Green House Gas F. NASA’s Commercial Smallsat Data Acquisition Program (CSDAP) G. Algorithm Publication Tool (APT) H. Multi-Mission Algorithm and Analysis Platform (MAAP) I. Airborne Data Management Group (ADMG) J. Harmonized Landsat Sentinel (HLS) Global Processing K. Science Discovery Engine L. Satellite Needs Working Group (SNWG) M. NASA Interagency Technical Report Observation (NITRO) N. Machine Learning

III. Base contract period (12 months) Each IMPACT project develops an integrated statement of work on an annual basis which denotes responsibilities for each team member. Below is a summary of the annual tasks to be performed within each respective collaborative, agile project.

Task 1 – System Architecture and Earth Observation (EO) Data on the Cloud (EODC)

System architecture activities will focus on architectural analysis and identification of software solutions for the IMPACT portfolio of projects and promote interoperability among projects which facilitate Earth science data analysis in the cloud.

The Contractor shall provide subject matter expertise as a system architect and for EO data on the cloud with key focus to:

o Evaluate, assess and identify software solutions;

o Determine functional and nonfunctional requirements for new software;

o Develop a strategy for cross project collaborations and reuse existing open source solutions, as needed;

o Define guidelines and standards for software development workflows;

o Deliver a framework for publishing datasets to the cloud in cloud-optimized formats; and o Improve accessibility of EO data in the cloud through metadata, tools and capacity building.

Task 2 – NASA's Visualization, Exploration, and Data Analysis (VEDA) Project

The VEDA project provides sustained support to NASA HQ’s Earth Science Data Systems (ESDS) Program to advance interactive visualization, exploratory visual data analysis, geographic information system (GIS), and within the cloud computing and high-performance computing environments. This will support the ESDS programs with current needs and position to proactively address future needs and initiatives.

The Contractor shall supply VEDA development support including:

I. Data System in NASA cloud o Incorporate multiple types of data, including raster, vector, and tabular data, into the data system.

o Integrate heterogeneous datasets from various sources, including satellites, ground measurements, and other sensors, ensuring compatibility with different sensors, spatial and temporal resolutions.

o Integrate various data sources and accommodate different metadata profiles.

o Develop and implement data governance framework

II. Platform and System in NASA cloud o Develop a uniform data ingestion pipeline to facilitate data transformation and integration.

o Implement a light-weight metadata catalog to organize and manage metadata effectively.

o Provide Application Programming Interfaces (APIs) for data discovery, access, and visualization.

o Develop a mapping interface for geospatial data visualization and interaction.

o Establish a pipeline to ingest, publish, and display derived insights in a user-friendly manner.

o Develop GIS compatible data services and plugins for data in the system

III. Usability o Design the system to support different user personas in both exploratory and explanatory modes.

o Offer appropriate data visualizations based on data type and narratives, including customizable widgets for user personalization.

o Provide overview information for quick data understanding and offer detailed information on demand.

IV. General o Adopt a modular design approach to allow for the replacement or upgrading of system components as technologies evolve.

o Ensure that the platform and system are performant and capable of supporting interactivity with large-scale data.

o Support existing projects: Earth Information System and NASA-ESA-JAXA trilateral dashboard o Develop tools to collect and report metrics.

V. Open Science o Develop the system and its ecosystem as open and reusable software, adhering to open science principles.

o Engage the community in the development process to encourage buy-in from a diverse pool of developers and users.

o Align design choices, including APIs and data formats, with existing and evolving community standards and best practices.

Task 3 - Scalable Science Data Processing

The Contractor shall supply VEDA research support including:

• Develop and implement a re-useable science data processing with cloud-native components

• Deploy and maintain data processing solutions in NASA cloud

Task 4 – System Operations, Maintenance, and Scalability

The Contractor shall supply VEDA development support including:

Task 5 – Green House Gas

I. Implementation of the VEDA System for the GHG Center Monitoring and Information System

1.1. Implement the VEDA system as a redeployable technology stack to support the GHG Center Monitoring and Information System within NASA cloud.

1.2. Collaborate actively with various agencies, including science teams, data teams, and engineering teams, to gather input and customize the VEDA system to meet the unique needs of stakeholders.

1.3. Contribute to the design and implementation of data and visualization pipelines in collaboration with relevant teams and agencies.

II. Data Ingestion, Publication, and Visualization

2.1. Leverage expertise gained through collaboration with the NASA Earth Information System (EIS) project to establish a comprehensive process for data ingestion, publication, and visualization.

2.2. Collaborate with various distributed active archive centers and the EMIT project to ingest, archive, publish, and visualize relevant datasets from a single, authoritative source.

III. Information Curation and Management

3.1. Establish an information curation and management team to work collaboratively with agencies.

3.2. Ensure the accuracy, completeness, and consistency of metadata and information resources, including documents, FAQs, tutorials, announcements, and discoveries.

IV. Implementation of an Information Management System

4.1. Incorporate an information management system designed to offer innovative search and discovery capabilities.

4.2. Facilitate access to linked resources, including indexed data, metadata, publications, notebooks, code, and more, to provide users with a comprehensive and integrated experience.

V. Feedback Mechanism and Help Desk

5.1. Incorporate a comprehensive feedback mechanism within the GHG Center data system.

5.2. Provide a help desk for support, offering both technical and scientific assistance to users.

Task 6 – NASA’s Commercial Smallsat Data Acquisition (CSDA) Program The CSDA Program focuses on increasing the use and utility of data from commercial constellations of small satellites for Earth science research and application.

The Contractor shall provide for:

● Cloud Data Archive and Distribution o Upgrade the current SpatioTemporal Asset Catalog (STAC) API with enhanced

API features;

o Forward migrate STAC records from the 0.8 to 1.0 specification; and o Enhance the data delivery system

● Onboarding of New Vendors o Support the mirroring of data from new and existing vendors to CSDA archive;

o Incorporate additional data sets as acquired by the CSDA into the data system;

o Develop metadata extraction applications for new vendors; and o Adapt the Smallsat Data Explorer (SDX) to support new vendors based on their unique end user data system needs

● Integration with the Earth Science Data and Information Systems (ESDIS) Program

○ Develop Program Increment (PI) plans to enhance and deploy the ingest and archive system components; and

○ Execute PI plans.

● Deployment and maintenance of all systems

● Provide technical support end user operation of the data system

Task 7 – Algorithm Publication Tool (APT)

APT supports the modernization and standardization of Algorithm Theoretical Basis Documents (ATBDs) and provides an efficient way to publish and manage ATBDs while easing the burden of content discovery for the end-user scientists searching for and reading these documents.

The Contractor shall:

o Add new features requested by the test team to APT;

o Support submission of ATBDs to an American Geophysical Union journal;

o Deploy APT to an operational cloud environment managed by the ESDIS

Program; and o Maintain the tool.

Task 8– Multi-Mission Algorithm and Analysis Platform (MAAP).

MAAP provides a cloud-based, collaborative environment that enables users to easily access, share and process data and code from both the European Space Agency and NASA. The goal of the NASA MAAP data system is to prototype a new and innovative data system solution to enable open science in cloud-based platforms and to ensure the ongoing quality of data, metadata and services provided by the MAAP.

The Contractor shall support the MSFC team for transition from Version 1 MAAP operations to the broader operation of the MAAP system and required to:

o Develop and adapt tools to ingest identified data into the MAAP.

▪ Implement continuous ingest streams; and

▪ Establish links to Earthdata Cloud.

o Enhance the MAAP data systems.

▪ Investigate and implement solutions to make MAAP operationally more efficient;

▪ Improve the MAAP user experience; and

▪ Improve scalability of the system o Provide fast browse, subsetting and other services to support the scientific research process.

o Implement cloud resource allocation and management processes and tracking data system metrics.

o Prototype unified search for data, software and documentation.

o Prototype a graph database to facilitate new discovery pathways for the users.

o Identify and infuse existing capabilities for ingestion, access, and visualization of data into the MAAP platform where possible.

Task 9 – Airborne Data Management Group (ADMG)

ADMG develops recommendations and produces guidelines for improved airborne data management and stewardship. Goals for the project are to facilitate improvements in three primary areas: communication pathways and processes; overall airborne and field data management and stewardship; and discoverability and access of airborne and field data and information.

The Contractor shall:

o Maintain the Catalog of Archived Suborbital Earth Science Investigations (CASEI), an airborne and field investigation inventory of contextual investigation metadata and data information and access;

o Develop additional web pages for additional contents such as instruments, platforms, and focus areas; and o Implement changes requested by the users.

Task 10 – Harmonized Landsat Sentinel (HLS) Global Processing The HLS project is tasked to generate HLS products (global 30-meter surface reflectance every 2-3 days) for a given period of time until the United States Geological Survey (USGS) takes over data production.

The Contractor shall provide subject matter expertise in cloud data processing for the following tasks:

o Complete historical archive for Landsat and Sentinel data products;

o Implement forward processing of Landsat and Sentinel data products;

o Support integration of new sensors within the Landsat/Sentinel-2 fleet as they become available;

o Collaborate with USGS to evaluate replacement atmospheric correction ancillary data once MODIS reaches its end of life;

o Support cost assessments of alterations to HLS processing; and o Deploy, maintain, and periodically modernize the processing pipeline in the

NASA cloud.

Task 11 – Science Discovery Engine

The objective of the Science Discovery Engine (SDE) is to develop a search capability that supports discovery and access to complex scientific data and contextual information across the five Science Mission Directorate divisions.

The Contractor shall:

o Implement indexing workflows to bring metadata and documentation into the

SDE

o Develop, implement and maintain pre- and post- processing governance workflows into the SDE o Develop and implement a scalable preprocessing pipeline for Natural Language Processing techniques o Develop and maintain API services for the SDE o Develop and maintain the user interface for the catalog; and o Deploy and maintain the system in the NASA cloud.

Task 12 – Satellite Needs Working Group (SNWG)

The SNWG Management Office supports the formulation of the biennial federal agency survey disseminated by the SNWG, and then facilitates the NASA-side assessment of the federal agency survey responses culminating in NASA HQ’s selection of recommended solutions to federal agency needs for satellite Earth Observations. The Management Office will instantiate a new tool for the one-stop coordination of the Scheduling, Communication, Reporting, Information, and Process Tracking (SCRIPT) of the SNWG assessment phase.

The Contractor shall:

● Conduct User Interface (UI) development for the SCRIPT assessment phase tool;

● Integrate the SCRIPT assessment phase tool with other IMPACT tools and databases via

API;

● Create SNWG-specific (custom instances) of existing IMPACT applications to facilitate accomplishing the milestones of the SNWG assessment phase; and

● Develop and deploy the SCRIPT assessment phase tool and related IMPACT applications in the NASA cloud as appropriate.

● Maintain User Needs Database refinement in coordination with SNWG stakeholders.

The SNWG Management Office will support product adoption and training of stakeholder communities via a Stakeholder Engagement Program (SEP). A new learning management system (LMS) will also be developed/deployed to coordinate the SNWG Stakeholder Engagement Program (SEP) and centrally manage training on SNWG products.

The Contractor shall:

● Conduct UI development for the SEP LMS;

● Integrate the SEP LMS with other IMPACT and Earth Science Division tools and databases via API; and

● Deploy the SEP LMS solution in the NASA cloud.

Task 13 – NASA Interagency Technical Report Observation (NITRO)

The NITRO project will assess and develop machine learning (ML) tools and services relevant to multiple NASA science divisions. The goal of the project is to adapt existing tools to maximize its reuse across science data archives to advance ML for science.

Furthermore, the project will develop guidelines and best practices for routine ML tasks and train domain experts in developing ML skills.

The Contractor shall:

o Expand the ImageLabeler tool to include other science domains;

o Enhance the active learning pipeline in Amazon Web Services;

o Support training dataset benchmarking tasks; and o Assist the IMPACT team in conducting machine learning trainings and workshops

Task 14 – Machine Learning (ML)

The goal of the ML project is to develop and deploy ML applications to support Earth Science Data Systems providing novel search and discovery of Earth Science data and augmenting data stewardship tasks.

The Contractor shall support the following areas:

● Development of an end-to-end machine learning pipeline:

o Implement a scalable preprocessing pipeline for machine learning; and o Containerize model training and deployment modules

● Phenomena portal o Develop the capability to search within the phenomena portal for different types of phenomena; and o Integrate a self-supervised learning capability.

Anticipated Future New Tasks. Due to the historical growth of IMPACT, new tasks are anticipated. These tasks will be added with a cumulative average expected growth of about 2 new tasks per year at 3 Whole Year Equivalents. Future new tasks will be negotiated as they arise.

IV. Four Option Period (twelve months each) tasks. Option periods are assumed to continue at the same level of effort for each assigned IMPACT project; however, annual requirement refinements will be negotiated resulting in an anticipated minor annual rebaselining of resources.

V. The Contractor is required to travel. Travel shall require prior approval from the Contracting Officer’s Representative attained in writing prior to each travel occurrence. A trip report including a summary of all expenditures shall be provided in the subsequent monthly report provided after conclusion of the travel. Travel costs will be incorporated into the contract via change modification.