I_1_b Draft PWS.pdf

DRAFT

Internal Revenue Service (IRS) Statistics of Income Division (OS:RAAS)

Performance Work Statement for

Production of New Synthetic Public Use Files & Production of the Validation Methodology

1. BACKGROUND

Federal tax data, derived from individuals' and businesses' tax and information returns, are invaluable resources for broadening our understanding of individuals' and firms' responses to economic incentives through quantitative research. However, full access to these data is only available to select government agencies, a very limited number of researchers working in collaboration with analysts in those agencies, or through highly selective research programs run by these agencies. In addition, the existing process of manually vetting each statistical release for disclosure risks is labor-intensive and offers inconsistent protection since it relies on subjective human review.

The Statistics of Income (SOI) Division of the IRS has collaborated with a think tank for nearly seven years on producing two statistical products that would safely allow researchers to perform statistical analyses using administrative tax data while protecting the confidentiality of taxpayer information. The first is developing a framework to produce synthetic public-use files on individual information derived from income tax returns and information returns filed by third parties. The change from traditional public use file (PUF), based on more ad hoc statistical data privacy methods, to synthetic data is due to the growing disclosure risks associated with producing a traditional PUF. In other words, a malicious actor might be able to use the increasing amounts of publicly available data to match with tax return information and leverage computing power to identify such matches. Additionally, the emergence of modern privacy-preserving methods, such as synthetic data generation, can potentially allow more research access to tax data while also increasing protection of information from attack.

The synthetic data are based on a statistical representation of the distribution of the confidential tax data. Tax records are drawn at random from that statistical model, where the synthetic records are designed to resemble actual tax returns and preserve statistical relationships. More importantly, these synthetic records are not actual tax returns and do not contain confidential data. Synthetic data can be useful for many purposes, such as calculating summary statistics and predicting revenues from proposed changes in the tax code. But because synthetic data protects privacy by obscuring certain features of the data, more complex statistical analyses must be performed on the confidential data.

The second statistical process, a validation server methodology, addresses this concern. A validation process will let SOI-approved researchers conduct more complex analyses by allowing them to develop, test, and debug statistical programs on the synthetic PUF and then submit those programs via a secure portal to receive sanitized versions of the statistical results that are based on the actual data. The estimates will be statistically valid, but somewhat less precise than estimates based on the actual data due to the privacy-preserving alterations.

1.1 PURPOSE

The goal of this project is to continue the expansion of high-quality synthetic tax data and a validation process—two innovative new statistical methodologies designed to allow and expand research access to administrative tax data while protecting privacy. These are both cutting-edge approaches. Over the last seven years the new methods have been adapted to the large scale and complexity of tax datasets and finding ways to improve quality while protecting privacy. Using these methods, a synthetic supplemental PUF was created containing information about people who did not file income tax returns or have an obligation to file. Those data had never been publicly available before, but a synthetic version was produced that was safe to release. In combination with the PUF, these two files provide a more complete picture of the range of households in the US, including those with low incomes. A synthetic version of the 2012 PUF was created and a 2013 synthetic PUF is currently in tested. Evaluation of the 2012 synthetic PUF and the methodological contributions embodied in it were the subject of articles in the National Tax Journal and statistical journals.

1.2 AGENCY MISSION

The federal statistical community views SOI as a leader in the development and deployment of modern privacy methods. The final report of the Advisory Committee on Data for Evidence Building (to which SOI Director Barry Johnson was appointed) cited the approach as a model for how sensitive administrative data produced at the federal, state, and local levels could be safely used for research. This work is an important element of implementing the IRS modernization plan under the Inflation Reduction Act. Moreover, the knowledge learned will help agencies throughout the federal government advance the goals of the Foundations for Evidence-Based Policymaking Act of 2018.

2. SCOPE

The work under this contract will build on the work performed under contract TIRNO-17-C- 00062 and 2032H5-23-P-00106. The work on confidential data will be performed at IRS facilities or on IRS-provided laptops, by cleared vendor staff trained on the use of IRS systems and subject to the laws governing their use. No information derived from confidential data may be released unless cleared by SOI senior staff.

The contractor will continue to refine and improve the process of creating synthetic data files.

The contractor will also begin training data science staff at SOI on the building and evaluation of synthetic data files and will support SOI in producing a public release of a fully synthetic PUF as a replacement for the traditional PUF. The contractor will develop documentation and training materials to support this effort suitable for publication and provide materials to support publication on the SOI website, if necessary, of extensive documentation and analysis of the synthetic PUF.

The contractor will present its vision for deploying a validation process on the National Secure Data Service (NSDS) (or some other entity to be chosen by SOI) to SOI staff and receive feedback on data security plans and any other related issues that could be involved in a future implementation of the NSDS. The contractor, in collaboration with SOI, will continue to research and implement statistical methods for improving synthetic data while protecting privacy. For each of these, the contractor will provide regular updates on work status to SOI staff.

The contractor will provide formal training sessions on how to generate the synthetic data on future PUFs to SOI data science staff and regularly follow up to provide guidance in developing and evaluating one or more synthetic data files. The contractor and SOI staff will make at least one presentation to SOI reporting on progress, and the contractor will report on status to at least one other panel of synthetic data privacy experts. The contractor will fully document all work.

3. CURRENT INFORMATION TECHNOLOGY (IT)/NETWORK ENVIRONMENT

Once granted clearance, the IRS Cleared Individuals (vendor staff with IRS clearance) will use IRS laptops to access the data and use the existing RAS-1 computing environment to run the analyses. All IRS-cleared individuals access the IRS laptops in a separate room to prevent others from viewing the screen. All IRS laptops are locked when not in use.

4. OBJECTIVE

The objective is to generate a fully synthetic PUF, write a white paper that describes and evaluates the synthesis methodology and other technical papers documenting all aspects of the research project, host training sessions and create materials for SOI on how to generate the synthetic data, develop and improve a validation server that could be used to conduct research while protecting privacy, and develop open-source code with accompanying documentation.

5. QUALITY CONTROL PLAN

The Contractor shall submit a Quality Control Plan in response to the Quality Assurance Surveillance Plan for each IDIQ Task Order within ten (10) business days of Task Order award.

The Contractor shall maintain and utilize a quality control plan which correlates to the Performance Requirements Summary and Quality Assurance Surveillance Plan for each IDIQ Task Order and throughout the life of each respective task order. Each quality control plan must contain, at a minimum:

• A description of the inspection system to cover all services in the QASP. The description must include specifics as to the areas to be inspected on both a scheduled and unscheduled basis, frequency of inspection, and the title and organizational placement of the inspectors.

• A description of the methods used for identifying and preventing defects in the quality of service performed.

• A description of the records to be kept to document inspections and corrective or preventive actions taken.

The records must be kept and made available to the contracting officer, when requested, throughout the agreement performance period and for the period after each task order’s completion until final settlement of any claims under this agreement or any of its associated task orders.

6. TASKS

The proposed work includes 17 tasks to safely expand access to administrative tax data. In developing the timeline and costs, the following assumptions were made: (1) the vendor will train SOI staff on data synthesis methodology and collaborate in product development; (2) by task year 5 (or option year 4) SOI staff will be able to generate the synthetic data products independently.

1. Working with SOI staff, prepare the 2016 Synthetic Public Use File (PUF) for SOI release.

a. Conduct user testing of 2015 Synthetic PUF (prepared largely under the 2023 contract).

b. Based on user feedback, build, and test the 2016 Synthetic PUF.

2. Technical paper on the 2016 Synthetic PUF.

a. Draft a technical paper including a description and assessment of the 2016

Synthetic PUF and blueprint for future work and planned improvements.

b. Present white paper internally to SOI and to at least one other panel of synthetic data privacy experts.

c. Revise the paper (and the 2016 Synthetic PUF, if needed) to reflect feedback and publish the paper.

3. Assist SOI staff in development and documentation of a Synthetic Supplemental PUF (nonfiler data) for tax year 2016.

a. Provide training for SOI staff as needed and feedback and advice on work product.

4. Make validation server user-friendly.

a. Perform qualitative user studies to assess the usability of prototype validation server.

b. Based on those studies, develop the validation server interface and functionality to enhance the user experience.

5. Technical paper documenting the validation server infrastructure.

a. The technical paper will include a description and analysis of the user studies and documentation on the enhanced user interface.

6. Develop and test the feasibility of relaxations to differential privacy as the basis for privacy-protecting algorithms in a validation server.

a. Develop and test approaches to privacy protection based on local sensitivity (such as maximum observed sensitivity) to apply to an advanced validation server.

b. Publish the findings in a technical paper.

7. Develop a Synthetic PUF including indicators for race and ethnicity.

a. SOI and Treasury are exploring avenues for matching or imputing race and ethnicity onto individual income tax data. If those files become available, the vendor, in collaboration with SOI staff, will develop a synthetic file including race and ethnicity.

8. In collaboration with SOI staff, publish a paper on the quality of a Synthetic PUF with race and ethnicity and its usefulness for assessing racial disparities in the tax system.

9. In collaboration with SOI staff, produce a clean master file augmented with information from electronically filed returns and the INSOLE (individual and sole proprietorship) data file.

a. Using algorithms to correct math errors and logical inconsistencies as well as AI methods trained on the INSOLE file, improve the integrity of the individual master file.

b. Work with SOI data science staff to create a public use file that is representative at the state level.

c. Working with SOI staff, use the cleaned master file to create synthetic data based on a much larger share of the population data; assess the effect of sample size on the quality of the synthesis.

d. Analyze the effect of using a random sample from the master file as the source for a validation server (rather than a modified version of the INSOLE).

10. Continue to improve the validation server.

a. Modify the prototype validation server to allow more complex multivariate analysis requested by researchers within a privacy preserving framework.

b. Continue to improve the user experience for researchers.

c. Speed processing time.

11. Assist SOI staff in developing one or more additional synthetic data products based on priorities from the user community.

a. Document these new data sets (under task 17).

12. Work with SOI staff to develop a methodology to quickly produce synthetic datasets as companions to selected special studies datasets, both for stand-alone analysis, and to develop queries to execute on the validation server.

13. Improve the measurement and usability of a privacy budget to track and manage privacy loss from statistical queries.

a. Conduct tests on potential ways to better quantify and allocate a privacy budget or another approach to regulate how users submit analyses in a validation server.

b. Document findings (under task 17)

14. Work with IRS staff to assess the data security of the validation server prototype and enhancements.

a. Search for potential vulnerabilities in the validation server infrastructure and suggest necessary improvements.

b. Identify issues for further exploration as well as entities that might be able to help in identifying and mitigating risks.

15. Build and test a user interface for data stewards and owners, such as the IRS, to adjust privacy budgets, inspect researcher analyses and results, and approve and deny requests.

16. Produce annual reports to SOI on progress.

a. Periodically meet with senior SOI staff and data science team to discuss progress and issues.

b. Produce an annual report at the end of each contract period.

17. Write research papers on findings and present them to SOI, at conferences, and in other forums.

18. Provide ongoing consulting, troubleshooting, and user support services related to Tasks 1-17, above.

6. DELIVERABLES TABLE

DEL.

MILESTONE/

DELIVERABLE

PWS

REFERENCE

DATE OF

COMPLETION/

DELIVERY

GOV’T

RIGHTS

1 Conduct user testing of 2015 Synthetic PUF; based on user feedback, complete 2016 Synthetic PUF with goal of public release by SOI.

Task 6 1a.

June 30, 2025 UR

2 White Paper: A description and assessment of the 2016 Synthetic PUF (including blueprint for future work and planned improvements)

Task 6 2b

June 30, 2025 UR

3 Provide training for Task 6 June 30, 2025 UR

SOI staff and assist in development and documentation of 2016 Synthetic Supplemental PUF (nonfiler data)

3a

Perform qualitative user studies and develop user-centered approach to assess usability of prototype validation server.

June 30, 2025 UR

5 White Paper, Documentation on Infrastructure for Prototype

Task 6 5a

June 30, 2025 UR

6 Develop, test, and document approaches based on local sensitivity (such as maximum observed sensitivity) to apply to an advanced validation server.

Task 6 6a

June 30, 2026 (task years 1 & 2)

UR

Planned Work in Option Years 2 - 5 7 Contingent on the availability of data inside SOI, develop a synthetic tax file with race and ethnicity.

June 30, 2026 UR

Write a paper assessing its quality and usefulness for analysis of disparate racial effects of tax policy.

June 30, 2026

UR

9 With SOI staff, clean and access the master file and the electronically filed returns data for three purposes: (1) to facilitate creation of

June 30, 2027 (task years 2 & 3) new synthetic data files, such as files with geographic detail; (2) improve the quality of data synthesis by starting with much more data to train the algorithm;

(3) reduce and improve measurement of the privacy cost of validation server access.

10 Improve validation server at least three ways: allow more complex multivariate analysis requested by researchers within a privacy preserving framework; improve the user experience for researchers; speed processing time. (We expect this to be an ongoing process.)

June 30, 2027 UR

11 Develop one or more additional synthetic data products based on priorities from the user community.

June 30, 2029 (task years 3 – 5)

UR

12 Train SOI staff to produce synthetic datasets as companions to selected special studies datasets, both for stand-alone analysis, and to develop queries to execute on the validation server.

June 30, 2027 UR

13 Conduct tests on Task 6 June 30, 2028 UR potential ways to better quantify and allocate a privacy budget or another approach to regulate how users submit analyses in a validation server.

13a (task years 3 & 4)

14 Work with SOI to assess the data security of the validation server prototype and enhancements and suggest necessary improvements.

June 30, 2027

15 Build and test a user interface for data stewards and owners such as the IRS to adjust privacy budgets, inspect researcher analyses and results, and approve and deny requests

Task 6

June 30, 2028

16 Produce annual reports to SOI on progress.

Task 6

2026 - 2029

17 Report findings and innovations in periodic research papers.

2027 - 2029

18 Consulting, troubleshooting, and user support

Years 1-5 UR

7. PERIOD OF PERFORMANCE

The period of performance for this contract is one year, with four option years. The anticipated period of performance for the base year is 07/01/2024 – 06/30/2025.

8. PLACE OF PERFORMANCE

The primary place of performance will be the vendor’s facilities.

9. GOVERNMENT PROPERTY

Computer and other resources will be provided by the Internal Revenue Service / Statistics of Income, as necessary.

10. GOVERNMENT FURNISHED INFORMATION

Statistics of Income (SOI) will provide all necessary IRS data files. To maintain the security of IRS files, the contractor may not release the results of any studies to anybody outside Statistics of Income Division unless SOI grants prior written approval.

12. KEY PERSONNEL

IRS Program/Project manager:

Name: Victoria Bryant (or designee) Email: Victoria.L.Bryant@irs.gov Telephone Number: 202-803-9388

13. SECURITY REQUIREMENTS

All contractors must pass a standard background investigation and be issued contractor badges.

All contractors must complete mandatory training annually.

14. TRAVEL

Travel costs are not authorized.

15. TYPE OF CONTRACT

This will be a firm, fixed-price award.

16. POINTS OF CONTACT

IRS Contracting Officer Name: Annette Jones-Lumpkin Email: annette.e.jones-lumpkin@irs.gov mailto:Victoria.L.Bryant@irs.gov mailto:annette.e.jones-lumpkin@irs.gov

IRS Contracting Specialist Name: Kenya Bracey Email: kenya.l.bracey@irs.gov

Contracting Officer’s Representative (COR)

Name: Melissa Belvedere Email: Melissa.J.Belvedere@irs.gov Telephone Number: 202-803-9426

Backup Contracting Officer’s Representative (COR) Name: Brian Balkovic Email: Brian.E.Balkovic@irs.gov Telephone Number: 202-803-9425

Backup Contracting Officer’s Representative (COR) Name: Donald Guy Email: Donald.I.Guy@irs.gov Telephone Number: 202-803-9062 mailto:kenya.l.bracey@irs.gov mailto:Melissa.J.Belvedere@irs.gov mailto:Brian.E.Balkovic@irs.gov mailto:Donald.I.Guy@irs.gov