24-022-OITA - IDT Detection_ PWS (001).docx

Performance Work Statement (PWS) Identity Theft (IDT) Detection

1.0 Introduction & Overview

The Internal Revenue Service (IRS) recognizes improving outcomes as a strategic goal to enhance mission performance and expand its enforcement efforts. The Office of Research, Applied Analytics & Statistics (RAAS) provides innovative approaches to address IRS challenges by using cutting-edge technology and advanced data analytics while leveraging public-private partnerships, improved collaboration with state tax agencies, the tax industry, and other stakeholders, to assist with identifying and combating fraud.

To accomplish these strategic changes, RAAS supports the IRS Strategic Goals laid out in the IRS Inflation Reduction Act Strategic Operating Plan addressed below covering FY2022 to FY2026:

· Dramatically improve services to help taxpayers meet their obligations and receive the tax incentives for which they are eligible.

· Quickly resolve taxpayer issues when they arise.

· Focus expanded enforcement on taxpayers with complex tax filings and high-dollar noncompliance to address the tax gap.

· Deliver cutting-edge technology, data, and analytics to operate more effectively.

· Attract, retain, and empower a highly skilled, diverse workforce and develop a culture that is better equipped to deliver results for taxpayers.

The purpose of this requirement is to develop, test, deploy, and enhance IDT detection approaches for both individual and business tax return filings, as well as to support the ongoing collaboration with external partners in private industry and state taxation agencies.

2.0 Background

Identity (ID) thieves are becoming more adept at hacking data sources, placing legitimate taxpayers at greater risk. In 2022, government accounts and benefits, including the IRS, accounted for being the organization with the top number of compromised accounts by identity misuse (Identity Theft Resource Center). Data breaches can reveal confidential information that allows ID thieves to look like legitimate taxpayers making detection more difficult. The increase in promotion of first party fraud (FPF) schemes complicates the landscape in discerning identity theft (IDT) from FPF, and while the impact of both is harmful, they remain dissimilar in an important way: IDT results in loss of private taxpayer data as well as refund dollars with no chance of recovery and risks granting a thief access and legitimacy to commit further malicious acts, while first party fraud has no data loss and the refund dollars may have a path to recovery.

Attempted tax refund identity theft was almost $15B in 2022, which represents an increase of 50% more than the prior year amount, and it burdens taxpayers, creates challenges for revenue protection, diminishes confidence in the tax system, and impacts IRS operations. RAAS partners with other organizations in the IRS to combat IDT and serves as an effective champion of IDT efforts across the organization and helping business units collaborate to achieve their IDT goals. RAAS delivers an unwavering commitment to precision and quality in the IRS by:

1. Identifying and implementing proven frameworks to holistically guide IDT efforts and address risks.

2. Producing in-depth analysis of IDT and IRS efforts to combat IDT, the results of which directly impact decision-making.

3. Working to enhance defenses and putting forth innovative approaches and strategies to get ahead of ID thieves.

The IRS has had success combating IDT, as the known unprotected dollar amount is steadily dropping and is currently estimated at under $250M. Still, the IRS has seen schemes become more sophisticated and include more legitimate taxpayer information. This creates challenges in separating legitimate taxpayers from fraudulent filers increasing the likelihood of unknown undetected IDT.

3.0 Award Objectives

The IRS addresses challenges and responds to changes in the IDT environment by expanding and adapting the services delivered over time. RAAS delivers research and analysis to identify opportunities to improve IDT detection and deterrence through an unwavering commitment to innovation and strategic analysis.

The objectives of this Performance Work Statement (PWS) are to provide contractor technical support, analytics, and subject matter expertise across the RAAS IDT Portfolio by focusing on the following areas of responsibility:

1. Monitoring – The development of predictive fraud models and filters and the subsequent evaluation of the resulting selections.

2. Threat Response – To provide rapid response analytical capabilities to assess breaches and incidents and provide recommendations for threat mitigation and treatment.

3. Partnerships – To collaborate with outside partners like the state revenue departments, software Industry, and financial sector.

4. New Tools and Techniques – To explore and test innovative approaches to IDT challenges.

5. Deterrence – To identify the specific techniques and methods used by ID thieves to defeat IRS defenses and use this knowledge to develop and adjust policies and capabilities to dissuade future IDT attempts.

6. Strategy – To collaborate with IRS stakeholders to assess and develop strategies for IRS business areas and align program areas to key strategic outcomes.

7. Portfolio management – To support the ongoing coordination of IDT programs across RAAS and the IRS.

This requirement encompasses IDT Detection, which works alongside and as part of the entire RAAS IDT Portfolio to improve systemic IDT detection efforts using models and filters, establish and maintain standardized reporting as well as collaborating with outside partners.

The RAAS IDT Portfolio workstreams work closely in strategic partnership with other RAAS portfolios that treat and identify intersecting issues (e.g., identity assurance, promoter issues). Collaboration and knowledge sharing across programs in the portfolio is essential to ensure response to a dynamic IDT environment. RAAS and its contractors must work in partnership to ensure alignment of approaches across programs and to integrate and coordinate responses that address portfolio needs.

Contractors supporting IDT programs must engage in bi-directional feedback through ad hoc meetings and other program and portfolio activities that identify needs or emerging issues that may impact more than one contractor. RAAS and its contractors plan for and execute the integration of activities across a multi-contractor work environment and engage with IRS stakeholders to plan and execute programs and deliver continuous process improvement activities. RAAS contractors report program activities and progress up through their RAAS Initiative Managers who provide portfolio management, set strategic direction, define performance management objectives, goals, measures of success, and key milestones that are planned, monitored, and evaluated across programs in the IDT portfolio.

4.0 Scope of Work and Description of Tasks

This PWS is Performance Based Acquisition, hybrid Firm Fixed Price (FPP) / Labor Hour (LH) with mission oriented and event driven performance measures.

The scope of work encompasses contractor support to identify and deliver:

1. Develop innovative approaches to detect IDT and reduce false detections.

2. Identify policies and procedures to deter the submission of IDT returns.

3. Develop advanced analytic approaches to identify current and future IDT risks.

4. Research and consider new data sources to inform and revise taxpayer protection strategies.

5. Collaborate with stakeholders to assist IRS taxpayers who may be potential victims of IDT.

6. Improve the individual and business taxpayer experience.

7. Protect the IRS from revenue loss due to IDT and fraud.

8. Maintain and improve operational efficiencies.

Contractor support is required to:

1. Perform analysis of datasets used to research IDT and legitimate taxpayer returns.

2. Lead the identification of IDT tactics and schemes needed better prevent IDT.

3. Assess how IDT impacts the IRS and taxpayers; as well as the recommendation of actionable strategies to address strategic capability gaps.

4. Knowledge sharing and coordination across the IDT portfolio and partners.

Contractor support will work alongside business stakeholders across the IRS that:

1. Manage programs to detect IDT,

2. Apply additional protections and response to taxpayers,

3. Make referrals for law enforcement actions, and/or,

4. Monitor and maintain supporting data and operational detection models. IRS has performed this mission for several years and has both mature and evolving processes in place to conduct the above action.

The IRS works with organizations in the federal tax preparation industry through partnerships and best practice sharing to address the identity theft threat. To support the above efforts, the tasks will require the integration of advanced data mining, data-driven strategic analysis, and threat analysis.

As fraudulent actors increase in sophistication, IDT detection efforts must similarly iterate to develop systemic solutions for identifying complex fraudulent patterns. RAAS currently supports four IDT Detection workstreams that share common cycles of research and development throughout the processing year:

1. Return Review Program (RRP): This is the IRS’s most advanced system currently in operation, able to runs sophisticated filters using machine learning ensembles.

2. Dependent Database (DDB): This is one of the first systems deployed to combat identity theft, using more simplified business rule logic, but remains an important tool across many IRS program areas, not just identity theft.

3. Business Identity Theft: While this is a newer program area when compared to Individual Master File (IMF), it has made significant advances in the past few years but presents a host of challenges as the IRS has dozens of Business Master File (BMF) forms requiring attention, versus the singular Form 1040 on the IMF side.

4. Suspect Electronic Filing Identification Number (EFIN) Tool (SET): A more recent tool to the IDT program, this tracks professional preparer EFINs and looks for early signs of compromise or breach, allowing for a prompt mitigation effort, and protection for return preparers.

The above selection systems and processes coordinate with the efforts of additional detection project tasks, which includes several key processes and deliverables:

5. Production Support: IRS data essential to the IDT portfolio comes from a variety of systems, many of which have no systemic data transfers in place. This work is critical to consolidate data among disconnected systems, but also serves as the primary metrics and reporting work and runs many of the non-production offline processes.

6. Neutral Dataset and IDT Taxonomies: The Neutral Dataset remains an important component in our annual model and filter refresh process, providing valuable insights useful in creating the training data. The IDT taxonomies provide important metrics on the overall health and success of the IDT program and are shared with oversight agencies and Congress.

7. Security Summit: The Security Summit is a unique public-private partnership formed in 2015 to protect taxpayers and the tax system against identity theft refund fraud and includes both state tax agencies and tax industry partners.

8. Information Sharing and Analysis Center: An important part of the Security Summit, the ISAC facilitates information exchange for tax administration purposes related to identity theft tax refund fraud and remains a key resource for emerging tax IDT schemes.

4.1 TASK 1, Orientation Briefing (FFP)

Within two weeks of award, the contractor shall conduct an orientation briefing for the Government. The Government does not desire an elaborate orientation briefing, nor does it expect the contractor to expend significant resources in preparation for this briefing. Rather, the intent of the briefing is to initiate the communication process between the Government and contractor by introducing key task award participants, explaining their roles, reviewing communication ground rules, and assuring a common understanding of award requirements and objectives.

The orientation briefing shall be held at the Government's facility, or virtually at the Government’s discretion, and both parties shall mutually agree upon the date and time.

The completion of this briefing shall result in the following:

1. The contractor and Government personnel who will perform work under this award will be introduced.

2. The Government will show the applicable facilities to the contractor and determine if the contractor will be performing work on the Government’s site.

3. The Government may provide any GFP to the contractor at this time.

4. Any issues concerning the contractor clearances for contractor personnel will be discussed.

5. The contractor shall demonstrate confirmation of their understanding of the work to be accomplished under this PWS.

6. The contractor shall provide the accounting period end dates to be used for the term of this award.

Desired Outcomes & Deliverables:

The contractor shall provide:

Deliverable 0002.1: Contractor delivers Orientation Briefing Minutes to COR within 5 business days after conducting the briefing. Contractor delivers Quality Control Plan (QCP) to the COR within 60 business days of award order.

4.2 TASK 2, Ramp-Up Knowledge Transfer / Transition-In (LH) (Optional)

The Contractor shall conduct knowledge transfer and transition-in sessions as needed to ensure uninterrupted performance and service of the IDT program with the IRS stakeholders and the current support contractor.

4.2.1 Ramp-Up Knowledge Transfer (LH)

Ramp-Up Knowledge Transfer is defined as the Contractor acquiring and understanding the necessary documentation, information, and processes related to any work in progress. This task includes meeting with the IRS Subject Matter Experts frequently to gather any knowledge related to the Core Systems, Compliance Document Matching and Solutions Engineering/Enterprise Services tasks or programs in development. The government estimates this task will take approximately ninety (90) calendar days to complete after award.

The Contractor shall appoint a knowledge transfer manager to oversee the transfer of knowledge and ensure all areas are accounted for. If desired, this can be the Project Manager (PM) or Project Scheduler. Prior to the transfer of knowledge, the Contractor shall develop a draft Knowledge Transfer Plan (KTP) and present it to the Government within the first 15 business days after award. The KTP shall identify the Contractor resources and roles involved in the transfer process, provide a list of risks and mitigation strategies for the transfer, and contain a detailed resource balanced project schedule developed with Microsoft Project. The project schedule shall identify tasks, dependencies, deliverables, and milestones. The project schedule shall be at a sufficient level of detail to track progress on a bi-weekly basis, (i.e., all tasks will be detailed to a level such that no task has duration of more than 10 business days).

The Government shall have 10 business days to review the draft KTP plan and provide comments to the Contractor. Government and Contractor representatives will then meet to discuss the comments after which the Contractor shall incorporate the agreed-upon changes and deliver a final KTP within 5 business days of the government’s feedback.

The contractor shall meet with the IRS bi-weekly (twice a month) to discuss the progress of the knowledge transfer. The Contractor shall maintain and share a project schedule and discuss any risks or concerns at this time.

Desired Outcomes & Deliverables:

The contractor shall provide:

Deliverable 0002.1: Knowledge Transfer Plan:

The Contractor shall provide a Knowledge Transfer Plan that is accepted by the Government and successfully completes knowledge transfer in accordance with the plan and no later than three months after Government acceptance of the Knowledge Transfer plan.

4.2.2 Transition-In (LH)

The Transition-In Task permits for an effective transition from incumbent(s) Contractor to a new Contractor. The Government estimates this task will take approximately 90 calendar days to complete from the contract award date. Successful transition can be defined by its capacity to maintain full continuity of operations, high productivity, and exemplary service quality. To ensure a successful, low-risk transition, the Contractor shall follow the approved Transition-In Plan submitted as part of their proposal and keep the Government fully informed of status throughout the transition-in period. During the transition-in period, the incumbent(s) is responsible for performance. This Task would be worked concurrently with Task 2.1, Ramp-Up/ Knowledge Transfer.

Performance Requirements:

In addition to the transition actions in the approved Transition Plan, the Contractor shall perform the following activities during the transition-in period:

1. Coordinate with the COR to facilitate staff assignments, onboarding and offboarding, complete required annual briefings, obtain all required IRS and RAAS permissions and certifications, and process new contractor personnel background investigations, which is essential to receive permissions to access required IRS data and systems.

2. Create a project schedule at a sufficient level of detail to track progress on a weekly basis.

3. Request access to tools and documentation.

4. Review and understand all the technical and process documentation.

5. Review and understand the existing data and system security policies in place.

6. The IRS will ensure the contractor receives:

a. Any relevant planning materials and documentation which may include, but is not limited to, previous briefings, data queries, data models, filters, templates, tools, methodologies, and any other relevant information specific to IRS initiatives.

b. Operations and maintenance processes and procedures from IRS staff or the incumbent contractor.

c. Configurable data, including usernames and passwords and application settings.

d. Documentation on processes for reproducibility by the Contractor’s staff.

Operating Constraints:

1. Work under this Task Order may require a transition support period from the previous Contractor to transfer knowledge to the new Contractor and/or government staff that takes an average of 90 calendar days.

2. The transition phase will start no later than sixty (60) days prior to the incumbent contract end date.

3. All new contractor personnel attend a system security orientation within the first 10 business days following initial assignment to any IRS contract/order and additional IT security awareness training (commensurate with the individual’s duties and responsibilities) within 5 business days of being granted access to an IRS facility or system.

4. The IRS relies on multiple teams working together to deliver planned capabilities for one or more products and services. Teams involved in the transition could be primarily Contractor, the IRS, or any combination thereof (including multiple contractor organizations).

5. All work and products become the property of the government, regardless of the ownership of the automation hardware or software used to create the products or services.

Desired Outcomes & Deliverables:

The contractor shall provide:

Deliverable 0002.2: Transition-In Checklist:

The Transition-in Checklist provides a vehicle to verify completion of the necessary transition-in activities. The Transition-in Checklist shall be developed by the Contractor and shall include all necessary activities to ensure an effective transition from incumbent(s) Contractor to a new Contractor were performed. The Transition-in Checklist shall be signed by both the Contractor and the COR after completion of the task.

4.3 TASK 3, Program Management (FFP)

The contractor shall provide the following services:

1. Initiate, plan, execute, and monitor the work of the Contractor team to achieve the objectives, address the requirements, and provide the deliverables as stated in the PWS.

2. Participate in weekly briefings about the work throughout the period of performance, including periodic quality reviews of in-process work. The contractor shall provide status updates to RAAS management and other appropriate IRS staff, including the RAAS Initiative Manager and COR, which highlight key accomplishments/work activities, risks, and any general concerns with the completion and/or burn rate of the labor hour tasks. Ongoing meetings will be established to go over the progress on projects and deliverables.

3. Provide monthly status reports throughout the period of performance.

4. For each year of the Award Order, conduct a year-end closeout meeting to review and summarize all the work completed during the past year.

5. Create a final work summary report and conduct a close-out meeting to wrap up any items related to the Contractor’s final deliverables and any other necessary matters prior to the final period of performance date.

Operating Constraints:

1. Follow all IRS policies and procedures relevant to data privacy and security risk management throughout the period of performance by following IRS security and data protection protocols identified.

2. Coordinate with the COR to facilitate staff onboarding and offboarding, complete required annual briefings, obtain all required IRS and RAAS permissions and certifications, and process new contractor personnel background investigations.

3. Perform knowledge management activities to ensure interim work product and final deliverable information is current. Ensure all data, analyses, and materials are saved on SharePoint, or on IRS servers such as the Return Review Program (RRP) and the Compliance Data Warehouse (CDW), as appropriate.

4. Transfer knowledge of technical skills and lessons learned to IRS staff upon completion of deliverables and work products, or more frequently as needed.

Desired Outcomes & Deliverables:

The contractor shall provide:

Deliverable 0003.1: Weekly and Ad Hoc Briefings Deliverable 0003.2: Monthly Status Reports The Contractor shall produce a single monthly report in Word, PowerPoint, or any another agreed upon format, that contains the status report information that records the work delivered in the prior month to the COR and IRS Initiative Manager no later than 5 business days after the end of each calendar month. The monthly status report shall include:

1. Description of work performed in the past month, including a compendium of analytic products.

2. Work planned for the coming weeks.

3. Progress of critical upcoming tasks and activities.

4. Warnings to any problems or delays in project timeframes.

5. Attempted solutions of encountered problems, issues, and any risks identified with corresponding mitigation actions and escalation plans.

Deliverable 0003.3: Award / Annual Status Reports The Contractor shall provide complete and accurate Period of Performance Summary Reports, Monthly Status Reports, Cost Performance Reports, and Contractor Status Reports and a list of government furnished property currently and formally possessed by the contractor at the end of each Base and Optional Period of Performance. The Government also reserves the right to request this report at any point during the performance of the award at the discretion of the Contracting Officer or Contracting Officer Representative. This ad hoc report, if desired, will not be requested more than twice per year during the life of the award and is in addition to the regularly scheduled Period of Performance Summary report.

4.4 TASK 4, Return Review Program (RRP) Models and Filters (Individual Master File) (FFP)

RAAS annually develops and refines multiple IMF IDT models which are deployed in the RRP production system. Each of these IDT models is trained, either via machine learning or an advanced statistical model, to effectively detect identity theft returns using innovative approaches that have been refined over several years of research and analytics. In Processing Year 2024, these models included:

1. Filter K-1: Security Summit Data Models

2. Filter N: Schedule C Model

3. Filter O-1: Online Filer (OLF) Autonomous System Number (ASN) Clustering

4. Filter O-2: Online Filer (OLF) Device ID Clustering

5. Filter O-3: Online Filer (OLF) Email Domain Clustering

6. Filter O-4: Online Filter (OLF) Occupation Clustering

7. Filter T: Preparer EFIN Clustering

8. Filter U: IDT Suspicious Conditions

The RRP IMF modeling development timeline follows five key phases:

1. Business & Data Understanding: Leverage RAAS research across multiple workstreams to identify business issues that are likely to impact IDT behavior in the next processing year and develop initial new modeling feature concepts which address these issues.

2. Feature Engineering: Code new modeling features and evaluate feature significance marginal to features currently used in RRP models.

3. Model Development & Evaluation: Specify model parameters and build a training dataset from the return population in the current processing year. Subsequently train the model and evaluate performance relative to prior iterations.

4. Deployment: Support RRP production process with in-database scoring and provide support to RRP during the workload projection process.

5. Monitoring: Track filter performance during the Processing Year and partner with RRP personnel in diagnosing any encountered issue with both the filters that RAAS supports and any other performance issue with the non-RAAS filters.

The contractor shall provide the following services:

1. Develop an understanding of IDT operations across stakeholder business units to identify opportunities for improvement and ensure recommendations are in line with current operations.

2. As early filing season IDT patterns and schemes start to emerge, perform analyses to evaluate emerging business issue applicability for potential new RRP modeling features, filter updates, or new filter proposals (estimated five analyses completed by end of March).

3. Review existing data elements contained in the RRP Operational Data Store (ODS) and propose new elements to be added to ODS for the upcoming Processing Year (estimated five to ten proposed ODS additions to be populated in an Excel spreadsheet by end of April).

4. Specify new features to be included in the RRP modeling dataset for the upcoming Processing Year. The significance of new features will be evaluated marginal to features currently used in RRP models (Up to twenty new modeling features historically due no sooner than the end of May).

5. For the RRP filters maintained by RAAS, upon receiving RAAS approval, submit to RRP personnel a Filter Development Plan and updates to the RRP Filter Implementation Guide (Two Microsoft Word templates provided by RRP and which historically are due no sooner than the end of May).

6. In accordance with the approved Filter Development Plan, refine and test new model specifications and potential model segmentation strategies as applicable to improve effectiveness and enhance detection capabilities. Publish finalized models to RRP Model Production Environment (MPE) by mid-September.

7. Support RRP workload projections analysis by loading DDB filter projections into RRP development database and provide testing assistance for modeling features and reference tables generated by RAAS (September through December).

8. Appropriately document methodologies, work products, programming scripts and other foundational documents in Microsoft Word or other programs as required.

9. A rigorous methodology is implemented to ensure the widest set of potential new modeling features are tested. In addition, the contractor will support the submission to RRP of any new modeling features proposed by the RAAS IDT Innovation project which have been approved by RAAS leadership.

10. All Model development deadlines specified by RRP for various deliverables (e.g., Filter Development Plan, RRP Filter Implementation Guide, etc.) are met.

11. Propose and develop at least 10 new modeling features during feature engineering, leveraging new data and changing scheme dynamics to identify and test those which will add the highest marginal value (feature importance) to the existing models.

12. Provide a summary of the return risk indicators used by the clustering filters which compares their relative importance and effectiveness in identifying high risk clusters of returns and propose and develop at least 2 new indicators to replace non-performant ones.

13. Challenger RRP IMF models developed by the contractor are expected to outperform existing champion models. Success will be evaluated by assessing how well these models distinguish between confirmed IDT and non-IDT returns based on an industry-standard model performance methodology (e.g., ROC curve or Precision-Recall curve).

14. RRP IMF IDT filter metrics are reported during the Processing Year to inform IRS stakeholders about IDT Detection outcomes and facilitate leadership decision making. These metrics include but are not limited to:

a. Selection volumes, both total and marginal. Marginality can vary as we consider the total IDT program, versus subsets like all RAAS RRP, or all RAAS filters in general, which would include DDB.

b. Counts of confirmed IDT and non-IDT, using different criteria for these definitions based upon program needs.

c. Dollars claimed on refunds returns, as well as perfected amounts that would potentially be refunded.

d. The False Detection Rate, using established logic, as well as other appropriate formulas as necessary.

15. Any significant filter performance deviations from workload projection levels will be thoroughly evaluated and model specification refinements will be proposed to RAAS leadership and RRP personnel within one week of issue identification.

16. Provide a timely response (acknowledgement of request within one day) to RAAS and defined stakeholder requests for information about the Contractor’s plans and execution of task activities, proactive and effective resolution of issues, timely communications, and regular status updates that may include verbal and written briefings (estimated ten presentations throughout the year).

Operating Constraints:

1. RRP IMF models are currently developed using SAS Enterprise Miner and deployed in Greenplum. Enterprise Miner can support a variety of modeling approaches (e.g., decision trees, neural networks, or ensemble model specifications), but this software platform does not contain built-in functionality for some machine learning approaches such as random forests.

2. The IRS is moving towards a future state within IT that may transition work to a new platform.

Desired Outcomes & Deliverables:

The contractor shall provide:

Deliverable 0004.1 - Feature Development and Delivery The Contractor shall develop 10 new features to improve existing RRP IMF models and aggregate other proposals across the portfolio. Features shall be summarized using an established format that describes the feature, the rationale for the feature, and demonstrates IDT Detection performance. Once RAAS approves of the generated features, the code and supporting documentation shall be delivered to RRP. The final iteration of proposed features shall be summarized in a PowerPoint document, and formally delivered to RRP in the appropriate technical format by May 31.

Deliverable 0004.2 - Clustering Risk Indicators The Contractor shall develop 2 new risk indicators (features) to improve existing RRP IMF clustering models and evaluate the performance of these against existing indicators. Features shall be summarized using an established format that describes the feature, the rationale for the feature, and demonstrates IDT Detection performance. Once RAAS approves of the generated features, the code and supporting documentation shall be delivered to RRP. The final iteration of proposed features shall be summarized in a PowerPoint document, and formally delivered to RRP in the appropriate technical format by May 31.

Deliverable 0004.3 - Model & Filter Development Outputs The contractor shall develop all required RRP Model Development outputs including:

1. Completion of RRP Filter Development Plan and Filter Implementation Guide.

2. Completion of model and filter development process for all current and newly proposed filters and publishing of completed models to RRP MPE. This shall include the use an industry accepted model and data analytics development process like CRISP-DM and each step in the development lifecycle shall be documented and shared in regularly scheduled meetings.

3. Delivery of workload projection estimates to RRP for the upcoming Processing Year.

RRP Model Development Outputs shall be summarized in a PowerPoint document, and formally delivered to RRP in the appropriate technical format by their specified annual due date but no later than December 15.

Deliverable 0004.4 - AI Governance Reporting In accordance with recent executive orders and policy determinations, any machine learning model or similar advanced analytics tool may have to undergo an annual review and reporting requirement. This will include an internal review process through the AI assurance team, a presentation of the use case to a technical evaluation committee, the annual creation and/or update of a TEC model and data card, and an evaluation of the work to ensure it remains fair and unbiased. These materials should be prepared in a suitable presentation format, like PowerPoint.

4.5 TASK 5, Dependent Database (DDB) Models and Filters (Individual Master File) (FFP)

RAAS partners with business unit stakeholders and information technology to develop and maintain a set of filters deployed in the DDB System for IDT Detection. During the Filing Season, DDB filter performance metrics are monitored, and opportunities are identified to implement in-season programming updates. RAAS also conducts post-Filing Season analysis to recommend larger-scale enhancements to existing business logic, refresh underperforming filters, and identify candidates for filter retirement. Towards the end of the calendar year, RAAS conducts testing of approved DDB filter enhancements and generates projections.

The Coronavirus Aid, Relief, and Economic Security Act (CARES Act) was signed into law on March 27, 2000, and the American Rescue Plan Act (ARPA) was signed into law on March 11, 2021, and the resulting stimulus credits enacted by these bills, resulted in significant increases in the filing of tax returns by identity thieves. When these credits ceased, a massive migration of efforts occurred as thieves pursued other opportunities to file fraudulent returns with the IRS.

The contractor shall provide the following services:

1. As part of the annual model development lifecycle, develop a business understanding of current IDT risks and challenges across stakeholder business units to identify opportunities for improvement and ensure recommendations are in line with current operations.

2. Monitor DDB filter performance metrics throughout the processing year. Generate ad-hoc analyses on high priority filter topics and recommend programming updates as appropriate (generally from January to May).

3. Define and report key metrics regarding the impact of new legislation on DDB filters. These monitoring metrics should include year-over-year changes in many of the variables used in DDB filters including, but not limited to, refund claimed, tax liability, withholding, refundable credit amounts, filing history, etc.

4. Conduct DDB Post-Filing Season Analysis which examines filter marginal selection volume, false detection rate, and historical filter performance trends among other factors. This product will propose recommendations to business unit and IT stakeholders on potential filter changes for the upcoming processing year.

5. Evaluate opportunities to improve the performance of the DDB exclusion model, the logistic regression filter and other DDB filters based on offline statistical modeling and propose recommendations for model enhancements to IRS stakeholders. The production deliverable for this requirement will include a listing of all parameters in the re-specified models, the coefficient estimate of each parameter, a recommended threshold score value for each model and back-testing selection results. These specifications for any agreed upon model updates be delivered to IT by the end of November.

6. In advance of the next Processing Year, generate end-of-year data element reference files and DSLs that support various DDB filters and deliver to IT by the end of November.

7. Maintain a DDB testing environment in SAS Viya or SAS Enterprise Miner or a comparable statistical computing environment that allows for impact assessments of proposed filter changes prior to production deployment.

8. Appropriately document methodologies, work products, programming scripts and other foundational documents in Word or other programs as required.

9. Provide a timely response (acknowledgement of request within one day) to RAAS and defined stakeholder requests for information about the Contractor’s plans and execution of task activities,

10. Appropriately document methodologies, work products, and other foundational documents in an appropriate form in PowerPoint, Word, or a similar form. All developed code should be maintained in a shared environment, like Gitlab or SharePoint.

11. DDB filter performance metrics are monitored during the Filing Season and analytical support is provided to IRS stakeholders as needed.

12. Annual filter performance assessment will contain basic metrics on all active filters and deep dive metrics on a select number, not to exceed 15, according to business needs. These metrics will also include recommendations to update and retire select DDB filters as appropriate to minimize false detection selections while maintaining a high-level of marginal confirmed IDT selections.

13. During the Filing Season, any significant DDB filter performance deviations from workload projections or historical filing rates, will be thoroughly evaluated and recommended programming updates will be delivered within one week of issue identification to minimize revenue protection impact.

14. DDB Tax Reform Impact Report will provide actionable insights regarding taxpayer behavioral changes and recommendations to account for these changes in DDB for the next Processing Year.

15. New specifications for RAAS supported DDB models (e.g., logistic regression filter and exclusion model) to deployed in the next Processing Year will outperform existing models. Success will be evaluated by assessing how well these models distinguish between confirmed IDT and non-IDT returns based on an industry-standard model performance methodology (e.g., ROC curve or Precision-Recall curve).

16. DDB IMF IDT filter metrics are reported during the Processing Year to inform IRS stakeholders about IDT Detection outcomes and facilitate leadership decision making. These metrics include but are not limited to:

a. Selection volumes, both total and marginal. Marginality can vary as we consider the total IDT program, but DDB is often compared to RRP, RAAS RRP, the DSLS, and other offline processes.

b. Counts of confirmed IDT and non-IDT, using different criteria for these definitions based upon program needs.

c. Dollars claimed on refunds returns, as well as perfected amounts that would potentially be refunded.

d. The False Detection Rate, using established logic, as well as other appropriate formulas as necessary.

Operating Constraints:

1. DDB is developed in the programming language COBOL which limits the potential deployment of complex statistical models.

2. DDB filters are primarily discrete business rules which are deployed based on subject matter expertise which comes from the primary business stakeholders in Wage & Investment. These rules may be sensitive to changes in taxpayer behavior, new tax legislation, or IRS policy changes.

3. Direct access to DDB production data is limited and RAAS relies primarily on DDB data obtained from queries in SAP Business Objects Explorer (BOE) and ad-hoc extracts provided by IT.

4. RAAS systems and environments, including the Compliance Data Warehouse (CDW), are undergoing security enhancements, which may delay or restrict data extraction in some cases.

Desired Outcomes & Deliverables:

The contractor shall provide:

Deliverable 0005.1 - Post-Filing Season Analysis The contractor shall perform a post-Filing Season analysis on DDB filters which examines marginal selection volume, false detection rate, and historical filter performance trends among other factors. This product shall be in PowerPoint format and propose recommendations to business unit and IT stakeholders on potential filter changes for the upcoming Processing Year. The DDB Post-Filing Season Analysis should be completed by August 30.

Deliverable 0005.2 - Tax Reform Impact Report The contractor shall develop a formal end-of-year assessment of the impact of legislative changes, both pending and prior, on DDB filters. This product shall include an analysis of key variables used in DDB filters presumed to be impacted by legislative updates, as well more broadly evaluate IDT-related behavioral impacts. This DDB Tax Reform Impact Report should be completed by October 31.

Deliverable 0005.3 - Production Deliverables For any IRS or stakeholder approved updates to the DDB exclusion models, the logistic regression filter, or any other statistical models or business rules, the contractor shall deliver detailed specifications (e.g., a listing of all parameters in the re-specified models, the coefficient estimate of each parameter and recommended model thresholds) and back-testing selection results to IT. In addition, end-of-year data element reference files and Dynamic Selection Lists shall be provided to IT. DDB model specification documentation and reference files should be delivered by the due date set by IT, but no later than November 30 in any case.

Deliverable 0005.4 - English Language Rules The contractor shall maintain on a continual basis a deck, prepared in PowerPoint or any other agreed upon format, that contains simplified descriptions of all DDB statistical models, business rules, the logistic regression filter, exclusion models, dynamic selection lists, and any other work product produced by this task as requested by IRS or stakeholders for inclusion in this deck. A simplified summary version of this deck shall be produced annually in MS Excel no later than January 15.

Deliverable 0005.5 - AI Governance Reporting In accordance with recent executive orders and policy determinations, any machine learning model or similar advanced analytics tool may have to undergo an annual review and reporting requirement. This will include an internal review process through the AI assurance team, a presentation of the use case to a technical evaluation committee, the annual creation and/or update of a TEC model and data card, and an evaluation of the work to ensure it remains fair and unbiased. These materials should be prepared in a suitable presentation format, like PowerPoint.

4.6 TASK 6, Business Identity Theft Detection Analytics (Business Master File) (FFP)

As identity theft activity becomes more aggressive and more complex, fraudulent actors have started targeting the Business Master File (BMF) population. A BMF IDT scheme may focus on obtaining a fraudulent refund directly from a business return, particularly from forms that carry refundable credits, or as scheme could involve submitting business returns which generate income documents or flow through elements to make fraudulent IMF returns appear more legitimate to the IRS. RAAS initiated a BMF IDT analytics program in 2017 to support business unit stakeholders in two key areas: business rules running in the production DDB environment and an offline advanced IDT model.

For the business rules, RAAS collaborates with business unit stakeholders leveraging sprint cycles to identify opportunities to enhance current DDB filters, while also conducting new filter deployment via back-testing analysis. A BMF Dynamic Selection List DSL process is also supported by identifying suspicious IMF IDT elements (e.g., Device ID, EFIN, IP Address or PTIN) which are then provided to business stakeholders for inclusion in BMF DDB DSL filters. An offline BMF model has been deployed which covers several important BMF forms and continues to be improved annually as more and more IDT data becomes available in the BMF IDT program.

The contractor shall provide the following services:

1. As part of the annual model development lifecycle, develop a business understanding of current IDT risks and challenges across stakeholder business units to identify opportunities for improvement and ensure recommendations are in line with current operations.

2. Execute offline processing procedures for BMF IDT models and monitor performance during the Processing Year (January to December). Currently, these models are run weekly.

3. Conduct mentor review sessions with business unit stakeholders where samples of returns scored as high-risk by the BMF IDT models are manually reviewed. Document mentor feedback on the observed risk level of each return in the sample examined (January to December).

4. Support a BMF DDB Business Rule development process that leverages a sprint structure to present back-test results for potential new filters to business unit stakeholders and proposes implementation recommendations as needed. Sprint process shall also be used to evaluate the performance of current BMF DDB Business Rules (January to December).

5. Generate weekly BMF DDB Business Rule and DSL performance reports delivered to business unit stakeholders in Excel format (January to December), or any alternative format as requested by the business unit.

6. Maintain a BMF DSL process which identifies suspicious data element values, both IMF and BMF, and integrates these elements into BMF DDB DSL filters. DSLs based on prior Processing Year data will be delivered to business unit stakeholders prior to the start of the current Filing Season and in-season DSLs shall be generated iteratively beginning in March.

7. Mirroring the IMF modeling timelines in RRP, identify and implement improvements to current BMF IDT models and develop new procedures for production deployment as appropriate. As appropriate, develop new modeling datasets, enhanced modeling techniques, and new models. Prepare for an eventual migration of the offline model into a future IRS production system by ensuring models are properly documented, reviewed, and refreshed as directed, and developed in a platform independent method allowing for straightforward migration.

8. Identify opportunities to incorporate BMF Security Summit Data elements and BMF Leads Reporting information into BMF IDT models and or BMF DDB Business Rules.

9. Appropriately document methodologies, work products, programming scripts and other foundational documents in Microsoft Word or other programs as required.

10. Provide a timely response (written or verbal acknowledgement of request within one day) to the IRS and defined stakeholder requests for information about the Contractor’s plans and execution of task activities, proactive and effective resolution of issues, timely communications, and regular status updates that may include verbal and written briefings (estimated ten presentations throughout the year).

11. The BMF IDT offline modeling process will be executed weekly throughout the BMF processing year and selections are delivered to business unit stakeholders for addition to the BMF IDT case inventory. Model selection mentor review is performed on a reoccurring basis and a Filing Season analysis is delivered by the end of August that allows IRS stakeholders to determine whether to continue this modeling test and whether changes need to be implemented for the next Processing Year.

12. Maintaining a DDB business rule research sprint cycle process which is responsive to emerging BMF IDT schemes detected by stakeholders, despite other priorities. Deliver timely BMF DDB Business Rule and DSL selection reports which satisfy the requirements of IRS stakeholders. Continue to examine opportunities to provide coverage to additional BMF forms annually.

13. BMF IDT model and DDB business rule metrics on selection volume and other factors are reported during the Processing Year to inform IRS stakeholders and facilitate leadership decision making.

14. Technical challenges of scoring returns in an offline environment, and delivering selections timely each week to IRS stakeholders, are successfully navigated.

15. BMF DDB business rules development process generates new DDB filters deployed in production which are projected to select marginal high-risk returns.

Operating Constraints:

1. BMF IDT is an emerging risk area for tax administration, and the volume of confirmed IDT cases on many of the forms is very low compared to IMF IDT. This low level of BMF cases is a constraint for both BMF IDT business rule and model development on many of the BMF forms as the available targets are insufficient for supervised modeling approaches and comprehensive back testing for business rules.

2. Recent spikes in selections related to employment tax forms and their associated stimulus credits have created inventory management challenges as selection volumes on these forms have increased greatly. Additional challenges come from the amended returns, which are currently not processed through the production system, but through a manual offline process.

3. RAAS support for BMF IDT business rule and model development is currently limited to a subset of BMF form types considered highest risk for IDT by the IRS. Contractors will need to be agile with respect to new BMF form types that may present future IDT risks and be prepared to provide analytics support for these emerging BMF risk areas as needed.

4. BMF IDT analytics efforts…