Revised SOW_FINAL.pdf

STATEMENT OF WORK

TITLE: Services for the NIST Genomics Privacy-Preserving Federated Learning Platform

I. BACKGROUND INFORMATION

The NIST Information Technology Laboratory (ITL) Privacy Engineering Group (PEP) is seeking a team that will support the NIST Genomics Privacy-Preserving Federated Learning Platform and the NIST Collaborative Research Cycle (CRC). PEP has an interest enabling research for the sharing of datasets and, as such, has initiated a research project to address data privacy concerns regarding sensitive data including demographic and genomic data.

The PEP group has been developing a privacy-preserving federated learning (PPFL) model architecture.

That platform simulates the exchange and modeling of genotype and phenotype data. This project seeks assistance in evaluating the fidelity, utility, and privacy of output models from the PPFL environment.

This project would also develop privacy metrics to support the NIST Collaborative Research Cycle, an on-going community benchmarking project.

II. SCOPE OF WORK

a. OBJECTIVES/PURPOSE The aim of this statement of work is to conduct privacy evaluation exercises on the Genomics PPFL Platform and existing CRC data. This will involve developing empirical privacy evaluation software tools that can be applied to the Genomics PPFL Platform and then integrated into the metrology suite SDNist.

These software tools shall be complemented by a set of community red-teaming exercise that simulates privacy attacks on the Genomics PPFL Platform and the deidentified CRC data.

b. GENERAL The Contractor shall provide all support for technical project oversight, administration, and technical execution of this task order. The Contractor shall maintain accurate records of project activities.

c. SPECIFIC REQUIREMENTS The Contractor shall complete the following tasks:

Task 1: Create a suite of empirical privacy evaluation metrics.

a: Identify an extensive list of candidate empirical privacy metrics from academic literature, existing software repositories, and other relevant sources.

b: Recruit a set of at least five subject matter experts (SMEs) to advise the selection, evaluation, and implementation of privacy metrics.

c: Work with NIST staff to select a set of metrics to implement as original software. Develop a prototype of the metrics as a Python software library with the capability of examining the outputs of the NIST Genomics PPFL Platform.

d: Implement the prototype software from Specification 3 into the SDNist software library and configure the metrics to evaluate existing CRC data.

e: Create a presentation to communicate the project goals and recruit participants, partners, and other stakeholders at workshops, meetings, conferences, etc.

Task 2: Conduct a community red-teaming exercise to simulate privacy attacks on the Genomics PPFL Platform and the CRC deidentified datasets.

a: Work with NIST staff to develop a project plan for a community red-teaming exercise against the Genomics PPFL Platform and the CRC data. The project plan must have clear objectives and milestones that align to the due dates specified in this statement of work. Create a presentation to communicate the project goals and recruit participants, partners, and other stakeholders at workshops, meetings, conferences, etc.

b: Create a Python library to evaluate the fidelity and utility of the Genomics PPFL Platform output models. NIST and its delegates will collaborate to select a set of data fidelity and utility metrics. The test software shall output a standardized report in machine- and human-readable formats suitable for downstream analysis and research.

c: Develop a Python library to serve as a test harness to conduct the red-teaming exercise on the Genomics PPFL Platform. The test harness must be capable of accepting and validating community-created simulated privacy attacks in the form of Python scripts. The test harness must have an exemplar privacy attack script. The test harness must have clear documentation to facilitate community participation. The test harness will use the privacy metrics developed in Task 1 and output a standardized report in machine- and human-readable formats.

d: Develop a Python library to serve as a test harness to conduct the red-teaming exercise on the CRC data archive. The test harness must be capable of accepting and validating community-created simulated privacy attacks in the form of Python scripts. The test harness must have an exemplar privacy attack script. The test harness must have clear documentation to facilitate community participation. The test harness will use privacy metrics developed in Task 1 and output a standardized report in machine- and human-readable formats.

Task 3: Conduct red-teaming exercises on the Genomics PPFL Platform and the CRC data archive.

a: Coordinate a red-teaming exercise for the Genomics PPFL Platform models. The Contractor will develop a method to receive and validate privacy attack Python scripts from the privacy and data science community. The Contractor will run the attack scripts against the Genomics PPFL Platform models and collect the resulting reports. The Contractor shall create a report detailing participation and outcomes.

b Coordinate a red-teaming exercise for the CRC data archive. The Contractor will develop a method to receive and validate privacy attack Python scripts from the privacy and data science community. The Contractor will run the attack scripts against the CRC data archive and collate the resulting reports. The Contractor shall create a report detailing participation and outcomes.

c: Organize the red-teaming exercise submission and output materials into one or more software repositories. The repository must contain all the submitted privacy attack scripts and the reports generated from the test harness. The repository must be well organized and thoroughly documented. The repository must have Python scripts or Python notebooks that demonstrate how to navigate and analyze the contents of the repository. Create a presentation to communicate the project goals and recruit participants, partners, and other stakeholders at workshops, meetings, conferences, etc.

Task 4: Complete a final report that summarizes the major activities of this project.

The report shall include a directory with links to all the major resources developed during the project. The report shall correlate the results of the empirical privacy evaluation and the red-teaming process with the goal of guiding which empirical evaluations best represent privacy attacks. The report shall include detailed summary statistics on the privacy, fidelity, and utility evaluations made during the project.

Task 5: Maintain the content of the Genomics PPFL Platform evaluation library, the CRC website, and the CRC Github repositories.

The contractor shall update content to reflect changes in the program, fix bugs, respond to pull requests, add additional submissions to the CRC data archive, organize the PPFL model outputs, and add features responsive to this project’s needs. All software for this project shall be version controlled with Git. The contractor shall submit links to the public Git changelogs as a monthly status report.

Optional Tasks

The option task, if executed by the government, is intended to add additional functionality to the CRC.

The goal of the optional task is to add a benchmark time series dataset and evaluation metrics responsive to such data to the existing CRC resources.

Option Task A: Identify a benchmark dataset of time series data for the CRC.

The contractor will identify an open-source dataset of time series data suitable for deidentification research and benchmarking exercises consistent with existing CRC resources. The dataset may be synthetic or semi-synthetic if no appropriate public datasets can be identified. This task will be accomplished in close collaboration with NIST staff and delegates.

Option Task B: Develop and/or adapt data fidelity, utility, and privacy metrics to evaluate time series data for the CRC.

The contractor will develop and implement fidelity, utility, and privacy metrics responsive to the evaluation of time series data. These metrics may be drawn from previous CRC work, the NIST deidentification challenges, or other sources. The metrics shall be implemented in Python be integrated into the SDNist library.

Option Task C: Integrate benchmark time series data into existing CRC and SDNist workflows and website

The contractor shall implement the time series data into the CRC data utilities. The contractor shall ensure that the time series data will be well integrated into existing CRC and SDNist workflows.

Option Task D: Configure and tune open-source synthetic generators to run on benchmark time series data.

In collaboration with NIST and its delegates, the contractor shall identify a set of open-source synthetic data generators to adapt and configure to synthesize new records from the benchmark data created in Option Task A. These synthesizers shall be used to create example synthetic data and evaluation reports for the CRC data archive.

III. PERIOD OF PERFORMANCE

The period of performance shall be 12 months, inclusive of Optional Tasks.

IV. PLACE OF PERFORMANCE

The primary place of performance is offsite but implemented through the NIST ITL program in Gaithersburg, Maryland. Offsite work shall be coordinated with the SME prior to execution and incur no additional cost to the government.

V. PAYMENT SCHEDULE

It is anticipated the labor hours for this action will be spread over the tasks not in a linear fashion but as required in response to the needs of the tasks as they unfold.

VI. GOVERNMENT-FURNISHED PROPERTY

None.

All property, data, and information provided by the Government in the performance of this task remains the property of the Government and shall be surrendered to the Government upon completion or termination of this requirement. Likewise, all deliverables generated under this requirement remain the property of the Government.

VII. DELIVERABLES

Base Tasks

Task Description Format Quantity Due Date

1a Generate a list of candidate empirical privacy metrics MS Word doc 1 Award + 30 days

1b Recruit a set of at least five SMEs to advise on fidelity, utility, and privacy evaluation MS Word doc 1 Award + 60 days

1c Create a prototype suite of empirical privacy metrics that operates on the Genomics PPFL Platform and its data

Software repository 1 Award + 60 days

1d Create a suite of empirical privacy metrics integrated into SDNist that operates on the CRC data

Software repository 1 Award + 90 days

1e Create a orientation presentation for stakeholders Powerpoint 1 Award + 100

2a Develop a plan for a community red-teaming exercise against the Genomics PPFL Platform and the CRC data.

MS Word doc and Powerpoint 1 Award + 120 days

2b Create a software tool to evaluate fidelity and utility of the genomics PPFL output models Software repository 1 Award + 150 days

2c Develop a software test harness to host the red-teaming exercise on the Genomics PPFL Platform

Software repository 1 Award + 180 days

2d Develop a software test harness to host the red-teaming exercise on the CRC data Software repository 1 Award + 210 days

3a Coordinate red-teaming exercise for genomics PPFL data

MS Word doc of outcomes 1 Award + 240 days

3b Coordinate red-teaming exercise for CRC data MS Word doc of outcomes 1 Award + 270 days

3c Organize red-teaming exercise submissions and output materials into a software repository

Software repository and Powerpoint 1

Award + 300

4 Final Report MS Word doc 1 Award + 360

5 Maintain the genomics PPFL evaluation tools, the SDNist library, and the CRC website

Contributions to software repositories tracked with a monthly report of Git changelogs

12 Monthly

Optional Tasks – CRC Time Series

Task Description Format Quantity Due Date

A Identify a benchmark dataset of time series data for the CRC Dataset 1 Option Award +

30 days

B Develop and/or adapt data fidelity, utility, and privacy metrics to evaluate time series data for the CRC

Software repository 1 Option Award +

90 days

C Integrate benchmark time series data into existing CRC and SDNist workflows and website

Software repository 1 Option Award +

150 days

D Configure and tune open-source synthetic generators to run on benchmark time series data

Software repository 1 Option Award +

180 days

VIII. SUGGESTED PAYMENT SCHEDULE

Task Requirement Number of payments 1 All Task 1 deliverables accepted 1

2 All Task 2 deliverables accepted 1

3 All Task 3 deliverables accepted 1

4 All Task 4 deliverables accepted 1

5 Git changelogs accepted 12

Optional Tasks

Task Requirement Number of payments A Task A deliverable accepted 1

B Task B deliverable accepted 1

C Task C deliverable accepted 1

D Task D deliverable accepted 1

IX. PERFORMANCE REQUIREMENT SUMMARY

Base Desired Output Required Service Performance Standard Monitoring Method The first desired output is the implementation of a robust set of empirical privacy metrics for the Genomics PPFL Platform and the CRC.

The required service is identification, implementation, and integration of privacy metrics for the Genomics PPFL Platform and the

The performance standard for the contract will be completion of the deliverables, which are outlined in detail in the tasks for this SOW.

The deliverables of this contract define target due dates. The Contractor shall participate in regularly scheduled meetings that will be monitored by the

The second desired output is community-based red-teaming exercise.

The service is to create the tools, infrastructure, and documentation to conduct a successful set of red-teaming exercises.

The performance standard for the contract will be completion of the deliverables, which are outlined in detail in the tasks for this SOW.

The deliverables of this contract define target due dates. The Contractor shall participate in regularly scheduled meetings that will be monitored by the

SME

Optional Tasks The desired output of the option task is the implementation of a time series benchmark data set into the NIST CRC framework and metrics i t th d t

The service is to identify or create a dataset, build a metrology suite, and implement those resources into existing CRC software utilities.

The performance standard for the contract will be completion of the deliverables, which are outlined in detail in the tasks for this SOW.

The deliverables of this contract define target due dates. The Contractor shall participate in regularly scheduled meetings that will be monitored by the

SME

Standards for Acceptance of Deliverables The SME will provide comments on each deliverable within 14 calendar days from receipt of a given deliverable. The Contractor shall make any necessary changes to reports within 14 calendar days from receipt of electronic or written comments from the SME. The Contracting Officer’s Representative (COR) will provide written notification when a deliverable is accepted.

X. TRAVEL

NIST anticipates two domestic trips for workshops, conferences, or focus group meetings to support the work. The Contractor shall not represent NIST. Travel will be reimbursable per Federal Travel Regulations. Each trip is expected to be less than a week in duration. NIST will provide at least three weeks’ notice before each trip. Prior approval from the Contracting Officer is required in the following scenarios: lodging in excess of the government rate, airline/rental car upgrades, medical accommodations, driving instead of flying to destinations outside local travel, multiple destinations in a single trip, or trips longer than a week in duration. Prior approval is requested by the Contractor to the Contracting Officer via the SME. At a minimum, the Contractor shall include the justification and documented cost comparison. Travel locations and dates are being finalized, but the following are likely venues:

• The annual meeting of the Federal Committee of Statistical Methods (FCSM)

• The Theory and Practice of Differential Privacy Workshop (TPDP)

XI. CONTRACTOR’S MINIMUM QUALIFICATIONS

Key Personnel:

Level IV (Senior Scientist/Engineer): Experienced principal scientist or engineer with a minimum of a Ph.D. degree, from an accredited college or university, in a related field appropriate for the work required.

The person must have a minimum of 10 years’ experience in the related field, OR a demonstrated equivalent combination of education, training, and experience can be used to meet the minimum qualifications.

• Ph.D. degree in computer science with focus on differential privacy or related field

• 5 years’ experience in privacy-preserving algorithms

• Strong applied mathematics experience with proficiency in developing efficient algorithms for solving computationally difficult optimization problems

• Experience as a principal contributor to the development of differential and privacy-preserving algorithms

• Record of noteworthy publications, invited talks, and contributions

Support Personnel:

One or more Level III (Senior Scientist/Engineer): Experienced scientist or engineer with a minimum of a master’s degree, from an accredited college or university, in a related field appropriate for the work required. The person must have a minimum of 5 years’ experience in the related field, OR a demonstrated equivalent combination of education, training, and experience can be used to meet the minimum qualifications.

• Experience evaluating synthetic data

• Experience developing code to evaluate and visualize synthetic data

• Experience developing leaderboard or similar visualizations

• 2 years’ experience in privacy-preserving algorithms

• Applied mathematics experience with proficiency in developing efficient algorithms for solving computationally difficult optimization problems

One or more Level II (Senior Scientist/Engineer): Scientist or engineer with a minimum of a bachelor’s degree, from an accredited college or university, in a related field appropriate for the work required. The person must have a minimum of 1 year of experience in the related field, OR a demonstrated equivalent combination of education, training, and experience can be used to meet the minimum qualifications.

• Experience with data management and processing

• Experience managing software repositories

• Experience reviewing and editing technical documents such as technical manuscripts, code, and documentation.

XII. CONTRACT TYPE

Firm Fixed Price