SEPPOTools23_SON.pdf

STATEMENT OF NEED

SEPPO software for SAR Flood Mapping Production

Requisition NEED3000-23- 01165

PURPOSE:

The overall goal of this project is to provide support for the National Oceanic and Atmospheric Administration (NOAA) National Environmental Satellite Data, and Information Service (NESDIS), the Center for Satellite Applications and Research (STAR) is to support of SAR based flood mapping using cloud-based production. Currently, developmental algorithms utilize specialized Software for Earth Big Data Processing, Prediction, and Organization (SEPPO) for baseline SAR imagery production and interacts seamlessly with NOAA’s customized SAR flood mapping algorithms. These generated flood products are sent to National Water Center, NWS River Forecast Offices, FEMA, NOAA Flood Proving Ground Test Bed and USGS for evaluation and utility to monitor flooding disasters. The current production has been accomplished without integration into STAR to ensure developers frequent access to alter evolving code during refinement and due to past limited projects supported on STAR Cloud Production on NCCF.

Purchase of SEPPO is required to stage the transition of current NOAA SAR Flood products leveraging multiple satellites from pre-operational external cloud production into NESDIS Common Cloud Framework (NCCF). This is particularly important to support NOAA’s stated goal to improve flood monitoring over Alaskan rivers and coastlines and over other high priority areas as part of supplemental funded advancements. SEPPO is a novel, modern age set of tools for big data processing tasks largely rooted in well-developed open source libraries and add-on modules wrapping SEPPO around COTS software for processing at scale. The philosophy behind SEPPO is to provide a processing solution close to a user’s Linux environment with a modular design, for combining open source components to optimally customize big data processing solutions. As such, SEPPO provides a rich API library with many Python and Bash programs and scripts in a native, scalable Linux processing environment. SEPPO development was rooted in scaling geospatial and remote sensing data processing and offers modules for these application domains where SEPPO also emphasizes modern geospatial data mining and time series analysis techniques. While harnessing the cloud-processing solutions offered by Amazon Web Services (AWS), SEPPO does not use any proprietary scripting and the entire SEPPO source code is delivered to Earth Big Data customers for its open source, non-COTS components. The proprietary nature of SEPPO by Earth Big Data, LLC suggests that a sole source contract is needed to accomplish the goal.

SCOPE OR MISSION:

This procurement requires the purchase of 3 co-related items (SEPPO CORE software tools, SEPPO Geospatial, and SEPPO training and software establishment support). Each of these components is needed to ensure proper instillation of the software and migration of the NESDIS SAR flood monitoring algorithms and production environment into NCCF. SEPPO testing will also involve NOAA STAR NCCF AWS sandbox for storage and system to begin internal STAR SAR Production and delivery to current NWS and other federal customers.

BACKGROUND:

The increasing frequency and intensity of floods pose a significant threat to communities worldwide.

Accurate and timely flood mapping is crucial for effective disaster management, risk assessment, and response planning. SAR-based flood mapping leverages satellite data to provide near real-time information, enabling authorities to make informed decisions and allocate resources efficiently. This project is to begin the testing of previously funded research and development of NESDIS flood mapping using C-Band SAR.

The maturity of this algorithm has reached a point where code updates can be provided as modular code updates. Currently, this is all done using SEPPO in AWS to process the SAR imagery to a sufficient level to have a consistent and normalized radar cross-section. The cloud scaling for this SAR flood mapping has been optimized to generate low cost, efficient, and low latency products. Purchase of SEPPO is required to stage the transition of current NOAA SAR Flood products leveraging multiple satellites from pre-operational external cloud production into NESDIS Common Cloud Framework (NCCF). This is particularly important to support NOAA’s stated goal to improve flood monitoring over Alaskan rivers and coastlines and over other high priority areas as part of supplemental funded advancements. SEPPO is a novel, modern age set of tools for big data processing tasks largely rooted in well-developed open source libraries and add-on modules wrapping SEPPO around COTS software for processing at scale. The philosophy behind SEPPO is to provide a processing solution close to a user’s Linux environment with a modular design, for combining open source components to optimally customize big data processing solutions. The SEPPO application should fit seamlessly with AWS based NCCF environment, though the current SEPPO grabs and distributes independent of other systems. STAR generation will require S3 storage of preprocessed SAR imagery before SEPPO processing can begin. Earth Big Data personnel are currently NESDIS contractors working on flood mapping development and have valid credentials to aid in the SEPPO integration into STAR Cloud Environment. SEPPO has also been successfully integrated into Michigan Tech Research Institute’s cloud remote sensing environment (https://www.mtu.edu/mtri/).

TECHNICAL SPECIFICATIONS:

This Contract requires three procurements from the same vendor:

• SEPPO CORE software tools, this module contains the core functionality to setup and manage a company/institutional cloud infrastructure and handle big data processing jobs at scale. They include:

Administrator tools to manage users and cloud resources, setting of user policies for cloud resource access and quota User management tools to setup and maintain user accounts in a cloud environment Administrator/user tools for secret key management and key rotation Linux OS user management on local and cloud instances with linux style access policies for cloud bucket access Creation and management of amazon machine images for customized compute environments Creation and management of core cloud storage buckets (administration, users, shared) and user/company buckets Multi-region supports for efficient “process next to the data” deployments Creation and management of a cloud database server (POSTGRES) for job management with full control of user queues, queue priority and interdependency in complex multi-stage processing tasks. Queue parameters to include machine image, instance type, and job execution time settings for fine-grained management of processing jobs

Creation and management of autoscaling groups that can tap into the cost-saving Spot market. Tested with deployments of thousands of compute instances in several petabyte scale processing jobs

Job queuing system seppo_recipe_processor.py script with flexible data sourcing from multiple cloud access protocols (s3, http, gcp, scp, sftp, etc.)

Deployment of a cloud-daemon to handle the job queueing system Conda/mambaforge scientific computing environment for python deployment Jupyter Notebook / Jupyter Lab conda environment setup and connection to local/cloud jupyter server scripts

Setup scripts and tools to operate local and cloud-based Jupyter notebook servers Bash scripting Sandbox setup for users to experiment and develop code before production processing Examples and scripts for full automation of routine processing via cronjobs to setup end-to-end processing pipelines GitOps tools and integration with GitHub for code management SEPPO software delivered in source code (python 3 and bash), based on open source software components for full transparency and API availability to users HTML and PDF documentation of programs and API library

• SEPPO Geospatial Extension of the POSTGRES database with POSTGIS handling Access routines to the ASF DAAC for SAR Data inventory management DEM tools to handle a variety of DEM data sets in cloud environments (SRTM, NASADEM, COPERNICUS DEM, NED, USGS DEM resource, custom DEMs) Tiling tool to produce tiled output at Lat/Lon or Military Grid Reference System (MGRS) tiles from Satellite imagery processing time series tools for data stacking and optimized cloud data store format handling

(ZARR, GEOTiffs/BIGTiff, NetCDF, HDF5, etc.)

time series metrics computation for large time series in spatial and temporal dimensions, e.g. mean, median, maximum, minimum, percentiles, skewnessm kurtosis, coefficient of variation, standard deviation, cumulative sums; can be computed in groupings, e.g.

monthly or seasonal) several rastertools to manage raster data in a cloud setting GDAL integration (gdal VSI enhanced tools) RioXarray integration QGIS conda environment setup and connection to geospatial cloud database sample recipe scripts for routine geospatial processing operations Jupyter notebook examples of cloud access to data sets and cloud-based processing and visualization of cloudnative data

• SEPPO training and software establishment support

40 hours (one week) on-site training (travel costs billed separately) to setup the cloud infrastructure with an administrator group and then train users on the core principles of using SEPPO for cloud-scaled operations including customizations of special processing needs.

50 hours email/phone support over the course of one year to support administrators in operation of the software, and address bug fixes. Email support for users on general usage questions and troubleshooting.

This support includes release updates of the modules purchased that become available during year one.

PERIOD OF PERFORMANCE SCHEDULE:

The Period of Performance for this project shall be twelve (12) Months

SPECIAL REQUIREMENTS:

None

GOVERNMENT FURNISHED PROPERTY:

The Government will supply the AWS environment for the deployment of SEPPO. The Government will not supply the any workspaces, additional IT, computers, or equipment necessary to fulfill the contract.

Access to Government Facilities will need to be coordinated NOAA STAR representatives to support any in-person training.

IT INFRASTRUCTURE CONSOLIDATION CONSIDERATIONS:

1. Can the application(s) to be housed on the server(s) be serviced in the “cloud”?

Yes. SEPPO is designed to support cloud based large scaled remote sensing processing.

2. Can the application(s) to be housed on the server(s) be serviced by existing server(s)/storage?

Not applicable.

3. Can the use of virtualization alleviate the need for this new server(s)/storage?

Not applicable.