Attachment_B_-_Sample_Task_Order_B.pdf

USGS Contract #TBD Sample Task Order B

Statement of Work

1. Background

The USGS National Hydrologic Geospatial Fabric (NHGF) team maintains a system of open-source software projects in support of the USGS Integrated Water Prediction and affiliated programs. This task order is intended to provide software development support for modernization of one of those projects, the Geo Data Portal (GDP) and enhanced delivery of the National Water Census (NWC). The GDP system includes processing services, web and script client software, and a co-located data catalog. All of these sub systems are undergoing modernization in an iterative development process. This task order focuses on supporting the NHGF team in operationalizing the core geoprocessing web service framework to house the in-development gdptools python package as a geoprocessing plugin and assist in building out components of a cloud-optimized data catalog and associated data services. These tasks will also support the NWC new API development.

The existing GDP geoprocessing service is a legacy (12 year old) software project that uses outdated web service standards and a Java-based implementation that has become cumbersome to maintain. The body of work that this task order contributes to seeks to operationalize the replacement for the legacy GDP web processing service software, the python-based pygeoapi server software that hosts processing algorithms based on the stand alone gdptools python package. In addition to the OGC-API processes API, the pygeoapi server software supports the OGC-API Environmental Data Retrieval (EDR) that provides data access patterns similar to the GDP but are simpler and more user friendly. Given this, the NHGF team aims to support both EDR collections and the more flexible, but complex to use, GDP geoprocessing service.

Development for this project will be contributed to publicly available repositories (code.usgs.gov, github.com) and work with publicly visible data resources. Internal-USGS infrastructure requirements will be supported by USGS staff collaborating with contract staff.

USGS staff will act as product owner on the project and will make final approval decisions on technologies and merge requests to the publicly available GitLab code repositories.

Outcomes of this work will contribute to an incremental modernization of the system of data and processing services that make up the GDP. As such, outcomes of this project will be designed to facilitate USGS scientists and engineer’s efforts to build, maintain, and evolve the GDP and other similar systems that help deploy data and software developed by USGS projects as services and APIs.

2. Objectives

The primary objective of this task order is to assist in operational configuration and development of a a hardened and high performance pygeoapi instance to support a catalog of hydroclimate / hydrologic model data and access services. The server seeks to modernize the capabilities

Sample Task Order B / XXXXX offered by the USGS Geo Data Portal and associated THREDDS data server with modern OGC- API data services backed by cloud-ready data and more accessible python processing software.

Areas of modernization that USGS requires assistance with include:

1) providing asynchronous and scalable geoprocessing capabilities via OGC-API processes and USGS-authored pygeoapi plugins,

2) establishing, if possible, configuration practices for a replacement OPeNDAP server solution for use with cloud-optimized data,

3) establishing configuration best practices for EDR API collections for selected datasets,

4) evaluating scalability, performance expectations, and cloud costs for the system of data access and processing services,

5) assisting with performance, scalability, and orchestration of the USGS authored gdptools python package, and

6) assisting with establishing additional output formats and functionality for time series services via the EDR API.

All pygeoapi configuration will be contributed to repositories in this code.usgs.gov organization:

https://code.usgs.gov/wma/

All S3 data resources to be used in EDR API and THREDDS replacement configuration will be made available at this requester pays bucket: s3://nhgf-development/ or other publicly available end points.

The work will be conducted iteratively in collaboration with USGS staff. Deliverables will be contributed to publicly available open-source repositories maintained by the USGS or other open-source organizations (such as https://github.com/geopython). Outcomes should adhere to a quality assurance plan to be established at project kickoff using the quality assurance plan described at https://derisking-guide.18f.gov/qasp/ as a guide. Quality assurance guidelines will be evaluated by USGS via code reviews and at regularly scheduled demonstration and planning meetings as appropriate.

3. Statement of Work This statement of work consists of the following tasks to be delivered as milestones in an incrementally planned and delivered process:

Task 1: Asynchronous processing support and scalability for pygeoapi.

Provide configuration and required code modifications to operationalize pygeoapi enabling robust and scalable asynchronous process execution and design a solution for horizontal scaling of running pygeoapi processes.

Task 2: Data collection configuration patterns for diverse data sources in pygeoapi and python-based replacement for the THREDDS data server.

Assist in expanding and operationalizing dataset configuration patterns for selected datasets available via xarray compatible opendap and object stores. Dataset aggregation from many objects or endpoints into EDR collections and OPeNDAP services will be the emphasis of this task. Work on the xpublish-based OPeNDAP server alongside pygeoapi is required under this task.

Task 3: Environmental Data Retrieval scalability, performance, and cost containment.

Provide analysis, benchmarking, and guidelines to ensure known and acceptable quality of service from EDR services.

Task 4: Scalability and performance enhancements for the gdpTools python package.

The gdptools python package does not take advantage of some potential parallel and high-performance python techniques. Support in implementing these and ensuring they work within the pygeoapi context.

Task 5: Environmental Data Retrieval output formats and time series support.

The pygeoapi server software has limited support for output formats and requesting timeseries data as is common in hydrologic modeling. Support in implementing user-friendly output formats (including csv) and timeseries request use cases (including from polygon time series via zarr and possibly other formats) will be required.

4. Period of Performance The period of performance will commence on date of award and shall be completed within 12 months of the date of the award.

5. Deliverables The order of demonstrations laid out here is notional and is subject to confirmation and planning as the project progresses.

Item No.

Description Due Dates

1 Project Kickoff By end of first month of date of award for task order

2 Asynchronous geoprocessing demo. (Task 1) By end of first month of date of award for task order

3 Environmental Data Retrieval demo (Task 2/5)

By end of second month of date of award for task order

4 Asynchronous geoprocessing scalability demo (Task 1/3)

By end of fourth month of date of award for task order

5 Environmental Data Retrieval demo. (Task 3/5)

By end of sixth month of date of award for task order

6 Performance and scalability improvements for gdptools. (Task 4)

By end of eighth month of date of award for task order

7 Iterative improvements and support. (All tasks)

By end of Period of Performance of the task order

6. Inspection and Acceptance See Contract #TBD.

7. Government Furnished Materials None.

8. IT Security Requirements See Contract #TBD and completed FISMA 18-Point Checklist.

9. 508 Compliance See Contract #TBD and completed Accessibility Requirements Tool (ART).

10. Travel Estimates No travel will be needed for this task order.