A04_Performance_Work_Statement-Sediment_Fingerprinting.docx

Statement of Work

Introduction and Background

Sediment is a pollutant that impairs surface water supply and aquatic habitat. In order to manage sediment as a pollutant it is important to understand the sources of this sediment. For example, if the majority of sediment originates from construction sites or agricultural fields, managers would be interested to know this and develop appropriate strategies. An approach known as ‘sediment fingerprinting’ uses the geochemical fingerprints of sediment sources and by analyzing downstream sediment, known as target sediment, the sources can be apportioned.

The USGS-Water Mission Area and the Maryland Water Science Center have been actively involved in advancing sediment fingerprinting science including the development of the Sediment Source Assessment Tool (Sed_SAT). Sed_SAT relied on a Microsoft platform using a parametric regression approach. Because the Microsoft platform is no longer viable, we have the need to produce a new version of Sed_SAT. Furthermore, Sed_SAT used a single regression approach to correct for size and organic differences in source and target sediment, whereas a multiple regression approach may be more robust. We seek to develop a new version of Sed_SAT that can run on a downloadable version or a virtual version run directly on the Internet.

Scope The contractor shall deliver a user-friendly product for performing sediment fingerprinting using a Bayesian (probabilistic) and multivariate (deterministic) approaches. The product should be assessable and downloadable for PCs, or can be a product that is has a user-access interface directly on the WEB. The tool’s main task is to infer the contribution of sediment sources to the target samples both using a deterministically and probabilistically chemical mass balance approach that includes organic and size corrections with robust error analysis. The deterministic approach follows guidelines established by (Gorman Sanisaca et al. (2017) that has the following:

Data entry and correctness, imputation of missing or non-detects, bracket test, correction of source data using grainsize and organic content that uses a multiple regression approach with error analysis, discriminant function analysis, unmixing model with apportionment results, and error analysis using a Monte-Carlo approach and source verification test.

The probabilistic approach will rely on Bayesian inference to estimate the probability distributions of the contributions of each source. The Markov chain Monte Carlo (MCMC) algorithm will be used to sample from the posterior distributions obtained based on the Bayes theory.

The contractor shall deliver a software design, a complete test harness, a continuous delivery pipeline, and the web services compatible with and similar to those in the design document. In keeping with agile principles USGS will have a prioritized backlog of web services and integration tasks that the contractor shall complete during the period of performance. The product shall be delivered incrementally using agile and continuous delivery methods so that the government gains working software throughout each iteration/sprint. Agile methodologies allow for less detail up front and more planning as the work is completed.

The design document shall describes existing services that the new services shall be compatible with in technologies and style. The contractor shall work with the government to build the additional services and functionality. The new services shall be similar in scope and complexity to the existing service. Any new web services shall be compatible with the existing services and databases, or the contractor shall also modify or rewrite that service and database. The contractor shall work closely with USGS developers and product owner to ensure a sustainable product. The software code shall be developed in an open source version control repository. The contractor shall participate in planning, design, development, testing, and documentation throughout the term of the contract.

Requirements A user-friendly, public accessible program, that can work on a PC. The program can be a desktop application or a web-based application. In developing the tool, agile software development principles by Fowler (et al., 2001) must be employed so as the software will be easily extensible. New features and analysis tools can be added easily as the science on the subject advances or as the user community's needs evolve.

The tool must be developed using the C++ language. Qt (www.Qt.io) is needed to create the graphical user interface (GUI). The computational engine of the tool will be provided as a stand-alone Application Programmable Interface (API) library. The stand-alone API will allow the future use of the API in other environments, such as in web-based tools or GIS systems. The computational tool will be executed without the GUI by feeding the inputs and commands using a sequential scripting language. he GSL library (Galassi et al., 2002) will be used for the statistical analysis, OpenMP for shared-memory multi-core processing, and Armadillo (Sanderson and Curtin, 2016) for linear algebra. Qt allows cross-platform compilation for Windows, macOS, and Linux. Downloadable distribution packages will be made available for seamless installation on each operating system.

The software will acquire fingerprint (tracer) data as comma-separated values (CSV), Excel files (.xlxs), and JSON files. The software will also be equipped with a set of features for pre-processing the data. These features will include the ability to graph the data in different ways, fit probability distributions to the data, evaluate and compare the goodness of fit for various probability distributions, and perform correlation analysis between different element contents and also between element contents and other characteristics such as total organic carbon (TOC) and particle size, creating matrix plots between various constituents, performing Principal Component Analysis (PCA) and some other features. These analyses will inform the user about the usefulness of various fingerprinting elements and how to handle the particle size and organic content correction factors.

The program should have a multiple regression approach to correcting for size and organic differences between source and target samples. Tests that examine the significance of the multiple regression results should be included. The program will use the unmixing model in Gellis and Gorman Sanisaca (2018). Final apportionment results will include error analysis that uses a Monte Carlo approach and includes the source-verification test outlined in Gorman Sanisaca et al., (2017).

There will also be a range of post-processing tools available, including the visualization of the deterministic results in various formats, including bar and pie charts, and visualization of the probabilistic results, including posterior distribution of source contribution, posterior distributions of contributions of sources into a specific element of interest, posterior correlations and measure-predicted agreement plots in various forms including agreement plots, box plots, and bean plots. The analysis results can also be exported into Ascii text files plottable using external tools. The graphical results can also be exported in graphic formats such as PDF and png formats.

All the codes and some example input files and datasets will be made available via Github under MIT license (https://en.wikipedia.org/wiki/MIT_License). Also, several instructional YouTube videos will be posted that will show how to use various software features.

The contractor shall work cooperatively with USGS staff throughout the project to ensure compliance and sustainability of the delivered software. This work shall take place via pair programming, and frequent (daily) meetings/discussions with the product owner.

The contractor must work in a pair programming with USGS developers. This ensures viability of the project after contractor involvement.

Technical Capability requirements:

This project work requires a high level of software development skills. The contractor shall have knowledge of Bayesian and multivariate statistics. The applicant should have a proven track record of creating intuitive, user-friendly, GUI-based tools for water resource applications.

The contractor should have a track-record of utilizing third-party C++ scientific libraries such as Armadillo, SuperLU, GNU scientific library (GSL), machine learning libraries such as MLPack, and visualization APIs such as VTK. The contractor should have the ability to develop client/server applications and have the knowledge of HTTP Servers, JavaScript, experience in Database programming (MySQL), and server-side scripting using tools such as Node.js, Python, or PHP.

The contractor shall provide staff who have a minimum of 2 years of test-driven software development experience, including: CI/CD pipeline implementation, software design, testing, programming, and deployment.

The contractor shall have well-developed communication skills as well as an ability to work collaboratively with other professionals.

Deliverables:

The contractor and the government shall agree on a method and approach and shall deliver quality product adhering to agreed upon acceptance criteria. Working software components shall be reviewed and delivered throughout each iteration, ensuring the government will be gaining value throughout the duration of the contract.

The tool will be A user-friendly, public accessible program, that can work on a PC as a desktop application or a web-based application.

The software shall be delivered in an open source repository and will become property of the US Government to be freely distributed. The software shall have standard documentation and tests.

While working in pairs, the contractor shall report on progress and raise any impediments that they need resolved in order to resume work. The USGS staff shall make final determinations in implementation decisions. The USGS shall make any trade-off decisions on which features will be cut or added, and is the final arbiter on decisions regarding functionality and design.

Presentations: There will be a kick-off and/or planning meeting between the USGS and contractor within 3 weeks of contract award commencing work. The contractor shall use an agile methodology of software development. Once development begins, the contractor shall participate in at least bi-weekly status meetings with USGS scientists. The features shall be delivered electronically upon completion to the agreed upon acceptance environment. When the development is completed the contractor shall participate in a review meeting to accept the product and a retrospective meeting to discuss the successes and shortcomings. The retrospective is to give informal feedback and is not a formal review process.

Final Report:

The code, documentation and list of completed issues will serve as a report for this project.

The COR will inspect all materials delivered to the government under this contract and determine that they are complete and that the documentation meets all contract requirements. If deficient, the COR will issue a short memorandum rejecting the materials delivered. If the materials are deemed unacceptable, the COR will provide complete information detailing the shortcomings. The contractor shall be given two weeks from the date of such a memorandum, to correct the shortcoming. All software developed under this contract shall become the property of the U. S. Government.

Delivery Dates Delivery schedule: Using Agile software development methods, software products shall be delivered throughout the contract.

A web-service must be functional and delivered before work on the next one begins. Consistent and sustained progress must be made throughout the life of the project, expecting that development velocity will increase due to practices implemented earlier on in the project. Delivery includes working code, deployment, and documentation.

Quality Assurance Surveillance Plan

INTRODUCTION

This Quality Assurance Surveillance Plan (QASP) has been developed to evaluate contractor actions while implementing the Performance Work Statement (PWS). It is designed to provide an effective surveillance method of monitoring contractor performance for each listed service or deliverable on the Performance Requirements Matrix in the contract.

STANDARD

The contractor is responsible for management and quality control actions to meet the terms of the contract. The role of the COR is quality assurance to ensure contract standards are achieved. The contractor shall perform all work required in a satisfactory manner in accordance with the requirements of the PWS. The COR shall notify the CO for appropriate action if it is likely that the contractor will not achieve successful final delivery of the software code in accordance with the performance objectives and acceptable quality levels (AQLs) identified below.

PERFORMANCE REQUIREMENTS MATRIX

The COR will evaluate the performance objectives through surveillance as reflected below by reviews and acceptance of work products and services. The COR will assess progress towards the final delivered software code. Note that the performance requirements listed below are required for the final deliverable. The delivery of code will be assessed by the Government to ensure that the contractor is completing contract requirements.

PROCEDURES

The COR, along with the USGS Product Lead and the product owner, will inspect all tasks required by the contract to ensure contractor compliance with the contract requirements at the conclusion of each sprint, which shall have a length of two weeks or less. However as indicated in the statement of work, functioning code shall be deployed to a production environment throughout the life of the project.

Delivery shall occur via the CI/CD pipeline. The COR may find the delivery satisfactory even though further work is required, provided that the specific requirements of the sprint are met.

At the conclusion of each sprint, the COR, along with the USGS Product Lead and the product owner, will review the completed user stories and related functionality. Incomplete or inadequate code and user stories will be noted in a mutually agreed-upon issue tracker, such as Trello, Pivotal Tracker, or GitHub Issues. The contractor shall respond in that tracker addressing the accuracy and validity of the defect as well as any planned corrective action (if not already noted). The contractor team will discuss and document actions to prevent recurrence in their sprint retrospectives.

At the conclusion of the period of performance, a similar procedure to assess overall performance.

If any of the services do not conform to the contract requirements, the COR shall require the contractor to perform the services again in conformity with contract requirements. The COR shall not certify satisfactory performance for the contract until all defects have been corrected.

ACCEPTANCE OF SERVICES

Acceptance of services shall be based upon compliance with performance standards described in the PWS and surveillance procedures described in this QASP. Before approving/certifying any contractor invoices, the COR will verify that all invoiced services have been performed in compliance with contract requirements. The COR shall not certify satisfactory performance for the contract until all defects have been corrected.

References

Fowler, Martin, and Jim Highsmith. "The agile manifesto." Software development 9.8 (2001): 28-35.

Galassi, Mark, Jim Davies, James Theiler, Brian Gough, Gerard Jungman, Patrick Alken, Michael Booth, Fabrice Rossi, and Rhys Ulerich., (2002), GNU scientific library. Network Theory Limited.

Gellis, A.C., and L. Gorman Sanisaca. 2018. “Sediment Fingerprinting to Delineate Sources of Sediment in the Agricultural and Forested Smith Creek Watershed, Virginia, USA.” Journal of the American Water Resources Association 54 (6): 1197–1221. https://doi.org/10.1111/1752-1688.12680.

Gorman Sanisaca, L.E., Gellis, A.C., and Lorenz, D.L., 2017, Determining the sources of fine-grained sediment using the Sediment Source Assessment Tool (Sed_SAT): U.S. Geological Survey Open File Report 2017–1062, 104 p., https://doi.org/10.3133/ofr20171062, IP-079059.

Sanderson C. and Curtin, R. (2016). "Armadillo: a template-based C++ library for linear algebra". Journal of Open Source Software. 1 (2): 26. Bibcode:2016JOSS....1...26S. doi:10.21105/joss.00026