Current State of IEP Systems and Services.pdf

Internal Revenue Service

Current State of the IEP Systems and Services

Version: 1.3

Current State of the IEP Systems and Services ii

Table of Contents 1 Introduction

2 Document Purpose and Organization

3 IEP Background and Overview

4 Core Services

4.1 Portal Service and Program Integration Services

4.1.1 Program Management

4.1.2 IT Service Management

4.2 Portal Infrastructure Services

4.2.1 Network and Firewall

4.2.2 Storage

4.2.3 Computing and Virtualized Infrastructure Management

4.2.4 Data Center Services

4.2.5 Dynamic Traffic-Based Scaling

4.2.6 Internal Interfaces

4.2.7 External Interfaces with Other Government Agencies and Commercial Entities

4.3 Portal Web Hosting Services

4.3.1 Web Hosting Support

4.3.2 Capacity Management

4.3.3 Incident/Problem Management

4.3.4 Monitoring

4.3.4.1 Logging

4.3.4.2 Dashboards

4.3.4.3 Reporting

4.3.5 Application Infrastructure Integration Services

4.3.5.1 Application Deployment

4.3.5.2 Application Styles

4.3.5.3 Standard Technologies

4.3.6 Configuration Management and Change Management

4.3.6.1 Configuration Management

4.3.6.2 Change Management

4.3.7 Service Catalog

4.3.8 Service Desk

4.4 Portal Security Services

4.4.1 Compliance

4.4.2 Operations

4.4.3 Security Reporting

iii

4.4.4 Threat Mitigation and Protection

4.5 Portal Application Services

4.5.1 Web Content Management System Services

4.5.2 Web User Analytic Data Collection Services

4.5.3 Application Development Services

4.6 Portal Website Help Desk Services

5 IEP Workloads

5.1 IEP Application Sizes

5.2 Public User Portal (PUP)

5.2.1 IRS.gov Website Content

5.2.2 Web Applications

5.2.2.1 PUP Environments

5.2.3 Associated Systems

5.2.3.1 Political Organization Filing and Disclosure (POFD)

5.2.3.2 EITC Central Applications

5.2.4 PUP Hits Metrics

5.2.5 IRS.gov Website Monthly Page Views and Visits

5.2.6 IRS.gov Website Monthly Downloads

5.2.7 IRS.gov Website Daily Page Views and Visits for Peak Month of Filing Season

5.2.8 IRS.gov Website Hourly Page Views and Visits for Peak Day of Filing Season

5.2.9 Peak Days Downloads

5.2.10 Associated Systems Volumes and Volume Forecasts

5.3 Registered User Portal (RUP)

5.3.1 RUP Environments

5.3.2 RUP Hits and Bandwidth Metrics

5.4 Transactional Portal Environment (TPE)

5.4.1 TPE Environments

5.4.2 TPE Metrics

5.5 Employee User Portal (EUP)

5.6 Foreign Account Tax Compliance Act (FATCA)

5.7 IRS Cloud Managed Services Provider Support

5.7.1 IRS GFE Cloud Services

5.7.1.1 NetHub Support

5.7.1.2 Enterprise Case Management (ECM) Support

5.7.2 IEP Cloud Hosting Services

6 Current State Volumes and Volume Forecasts

6.1 IRS.gov Website Volumes and Volume Forecasts

iv

6.2 EUP Volumes and Volume Forecasts

6.3 ACA Volumes

6.4 FATCA Volumes

6.5 IEP Service Desk Volumes

Appendix A Summary of Applications

Appendix B IEP Application Details

Appendix C Key Organizations

Appendix D Acronyms v

Table of Tables Table 2-1: Current State Workload Element Definitions Table 3-1: IEP Website Service Model Definitions Table 3-2: IEP Component RACI Table 4-1: IEP Capacity Management RACI Table 4-2: IEP Incident Management RACI Table 4-3: IEP Problem Management RACI Table 4-4: IEP Monitoring RACI Table 5-1: Public User Portal Description Table 5-2: Political Organization Filing and Disclosure Description Table 5-3: EITC Central Applications Description Table 5-4: Public User Portal Hits Metrics Table 5-5: IRS.gov Website Monthly Page Views and Visits Table 5-6: IRS.gov Website Daily Page Views and Visits for Peak Month of Filing Season Table 5-7: IRS.gov Website Hourly Page Views and Visits for Peak Day of Filing Season Table 5-8: IRS.gov Website Total Downloads and Peak Day Downloads Table 5-9: EITC Central Visits and Page Views Table 5-10: EITC Marketing Express Month and Page Views Table 5-11: Registered User Portal Current State Description Table 5-12: Registered User Portal – Total Hits and Bandwidth Metrics Table 5-13: Transactional Portal Environment Description Table 5-14: ACA Open Enrollment Transactions Processed Table 5-15: Employee User Portal Description Table 5-16: Foreign Account Tax Compliance Act Description Table 5-17: Treasury Cloud Work Requests Table 5-18: NetHub Support Description Table 5-19: Enterprise Case Management (ECM) Support Description Table 5-20: IEP Cloud Work Requests Table 5-21: IEP Cloud Hosting Support Description Table 6-1: IRS.gov Volumes and Volume Forecasts Table 6-2: IRS.gov Monthly Page Views Table 6-3: IRS.gov Monthly Sessions Table 6-4: IRS.gov Forecasted Daily Number of Page Views for Peak Month of Filing Season Table 6-5: IRS.gov Forecasted Daily Number of Visits for Peak Month of Filing Season Table 6-6: IRS.gov Hourly Number of Page Views and Visits for the Peak Day of the Filing Season ... 55 Table 6-7: IRS.gov Hourly Number of Visits for the Peak Day of the Filing Season Table 6-8: IRS.gov Hourly Number of Visits for the Peak Day of the Filing Season Table 6-9: ACA Volumes for Open Enrollment period, November 1, 2022 – January 31, 2023 Table 6-10: FATCA Volumes for Filing Season 2021, 2022, and 2023 Table 6-11: Total Number of Incidents Table 6-12: Total Number of P1 and P2 Incidents Table 6-13: Total Number of P3 and P4 Incidents Table 6-14: Total Number of Service Requests Submitted via Service Catalog vi

Table of Figures Figure 4-1: IEP 1.5 and IRS Enterprise Computing Center Interfaces Figure 4-2: IRS Enterprise Computing Center and Treasury Cloud Interfaces Figure 4-3: IEP 1.5 External Interfaces Figure 4-4 : IRS Enterprise Architecture (EA) Target State Continuous Integration/Continuous Deployment (CI/CD) Process Figure 5-1: IRS.gov PUP Weekly Page Views and Visits (2022) Figure 5-2: IRS.gov PUP Weekly Page Views and Visits (2023) Figure 5-3: IRS.gov RUP Weekly Page Views and Visits (2022) Figure 5-4: IRS.gov RUP Weekly Page Views and Visits (2023) Figure 5-5: IRS.gov Monthly Downloads Figure 5-6: RUP Extranet Connections

1 Introduction Internal Revenue Service (IRS) is planning to award a new managed services contract to build and transition the existing IRS Integrated Enterprise Portals (IEP) to meet the evolving future business needs of IRS.

2 Document Purpose and Organization The purpose of this document is to describe the current state of IEP, which is a major program that IRS Information Technology has implemented to enable secure access to IRS business systems by internal and external users. This document describes IEP as a robust virtualized infrastructure platform that is fully scalable, managed privately, and dedicated to IRS as part of a comprehensive set of technology services for internal and external users.

This document also provides information to support IEP transition planning by articulating the scope, complexity, scale, and expected growth of IEP’s integrated core services, including IT infrastructure, operations, security, application, integration and support, and program management services. It includes a description of multiple workloads and application styles that integrate with the IRS processes and the approved technologies to enhance and accelerate the delivery of IRS business applications.

This document is divided into six major sections:

1. Introduction

2. Document Purpose and Organization

3. Integrated Enterprise Portals Background & Overview

4. Core Services

5. IEP Workloads

6. Current State Volumes and Volume Forecasts

The table below defines the elements used to describe each workload in the IEP Workloads section.

Table 2-1: Current State Workload Element Definitions

Table Element Table Element Definition System Name The system’s formal designation in Request for Proposal (RFP) documents.

Target Audience The group or groups the system is intended to serve.

Content and Services The information, functions, and resources available on the system.

Technical Platform A general description of technology components or framework that comprises the system.

System Integrations A general description of the interfaces and connections between the system and other systems that deliver the overarching functionality.

Location The current system locations.

Managed Service Provider (MSP) The current Managed Service Provider (MSP).

This document contains four appendices:

• Appendix A – Summary of Applications. This provides a summary list of all applications that are hosted in IEP Public User Portal (PUP), Registered User Portal (RUP), Employee User Portal (EUP), and Transactional Portal Environment (TPE).

• Appendix B – IEP Application Details. This provides application details and attributes for all IEP-hosted applications.

• Appendix C – Key Organizations. This identifies key current state organizations that have stakeholder roles in the operations and maintenance (O&M) of current IEP infrastructure.

• Appendix D – Acronym List.

3 IEP Background and Overview The mission of IRS is to provide America’s taxpayers with top-quality services by helping them understand and meet their tax responsibilities and enforcing the law with integrity and fairness to all.

In 2021, IRS IEP websites served over 11.4 billion page views to 660 million site visitors globally (during 2 billion sessions).

IEP is a robust virtualized hybrid cloud platform that is fully scalable, managed privately, and dedicated to IRS as part of a comprehensive set of technology services for internal and external users. IEP is the front door to the IRS Enterprise systems/applications and plays a mission-critical role in serving taxpayers, tax preparers, and employees. IEP provides fully integrated core services, including infrastructure, operations, security, application, integration & support, and program management services, to establish a single, flexible, scalable, reliable, robust, and digital platform that supports multiple workloads and application styles. This platform supports a variety of performance and availability requirements with more than 99.99% availability and sub-second responses even in the event of an unexpected disaster during peak tax season.

IEP platform supports:

• Rapid growth of online electronic filings and taxpayer access to information

• Automation of key business processes

• Centralized logging and monitoring that is integrated with IRS end-to-end (E2E) systems

• Balanced system capability to meet demand by dynamic application scaling

• Consistent user experiences for the taxpayer and tax preparer

• Improved data and content sharing between IRS user communities

• Enhanced delivery, accessibility to, and support of modernized IRS business applications for taxpayers and preparers

• Enhanced security posture of web-based applications

The core IEP platform consists of the following types of services that support multiple IRS workloads:

• Portal Service and Program Integration Services – Operates the IRS environment within specified scope, quality, time, and cost objectives, and as described in IRS Program Management, Project Management, Software Development Lifecycle, and Operations Management Framework.

• Portal Infrastructure Services – Manages the physical facilities furnished with hardware, software, middleware, network connectivity, and peripherals required to establish and host the IEP environment, as well as services needed to improve IEP operations in functional areas arising during the period of performance.

• Portal Web Hosting Services – Provides comprehensive application hosting support for application development, testing, integration, and production, as well as provisions for capacity management, incident management, performance monitoring, and application infrastructure integration support.

• Portal Security Services – Ensures security processes and procedures are consistent with industry and government best practices, such as NIST Special Publication (SP) 800-14, Generally Accepted Principles and Practices for Securing Information Technology.

• Portal Application Services – Provides enhancement and break/fix support for Contractor-managed applications on IEP 1.5, including maintenance of all tools/applications, continual optimization of the site based upon user experience, working with customers to identify requirements, as well as the design, development, testing and release of applications.

• Portal Website Help Desk Services – Provides the IRS.gov website service desk to taxpayers as a “first aid station” for IEP 1.5 website questions, such as navigation of IRS content and forms retrieval. This service is complementary to the IRS toll-free tax assistance line and does not respond to tax questions, personal queries, or economic stimulus queries. (Help Desk services’ requirements will not be included within IEP 2.0).

By leveraging core services described above, IEP scales to numerous virtual workloads, isolating each workload as needed to meet varying security requirements. Currently, IEP supports the following eight workloads, each with a different target audience and security:

• Public User Portal (PUP)

• Earned Income Tax Credit (EITC) Central

• Political Organization Filing and Disclosure (POFD)

• Registered User Portal (RUP)

• Transactional Portal Environment (TPE)

• Employee User Portal (EUP)

• Foreign Account Tax Compliance Act (FATCA).

• IRS Cloud Integration Support

A website service model description also is provided to delineate current Managed Service Provider/IRS accountabilities across major service areas. IEP follows a tailored lifecycle. The table below defines IEP website service model.

Table 3-1: IEP Website Service Model Definitions

Table Element Table Element Definition

Lifecycle Service/Process

A major system lifecycle function based on Information Technology Infrastructure Library (ITIL) latest version framework. Note: Selected ITIL services/processes are omitted if they are not currently operational.

MSP Managed Service Provider. An ‘X’ in this column indicates primary service ownership lies with MSP. IRS may play a supportive service role.

IRS Internal Revenue Service. An ‘X’ in this column indicates primary service ownership lies with IRS. MSP may play a supportive service role.

Joint An ‘X’ in this column indicates IRS and MSP share service ownership.

IRS IEP is comprised of the following components and associated accountabilities as outlined in the table below. Most IEP services are considered custom services. Custom services are service components that are not typically found in standard Operations & Maintenance offerings such as Application Development and Maintenance, Program Management, Security, and Middleware/Integration. The table below defines IRS IEP components and their specific accountabilities by using a RACI (Responsible, Accountable, Consulted, Informed) chart.

• Responsible for correct execution of the service or process outcome.

• Accountable for the result and overall quality of the service or process outcome.

• Consulted by those performing the process or delivering the service.

• Informed and kept up to date on the progress by those performing the process or delivering the service.

Table 3-2: IEP Component RACI

Operational Area Lifecycle Service/Process MSP IRS

Program Service and Program Integration Services

Program Support R A Process Support R A Reporting/SLOs R A

Portal Infrastructure Services

Multiple geographically separated data centers R A Telco – Redundant Dedicated Circuits; Vendor Diversity R A

Other - Redundant; Vendor Diversity R A Business Continuity (multiple geographically separated sites capable of handling full workload) R A

Portal Web Hosting Services

Web Hosting Support R A Capacity Management R A Incident/Problem Management R A Monitoring R A Application Infrastructure Integration Services R A Configuration and Change Management R A Service Catalog R A Service Desk R A

Portal Security Services Monitoring, Compliance, Reporting R A

Portal Application Services Enhancement/Optimization of Contractor-managed applications within IEP 1.5 R A

Maintenance of all tools/applications R A Portal Website Help Desk IRS.gov Website Help Desk R A

4 Core Services IEP platforms support multiple workloads and application styles to establish a set of reusable technology and process services that can be configured to meet the unique demands of IRS applications and environments. The core IEP services are tightly coupled and automated to enable a robust and scalable platform that is integrated with IRS processes and technologies to accelerate the delivery of IRS business applications. IEP uses Shared Service components that are used by more than one application.

4.1 Portal Service and Program Integration Services

Portal Services and Program Integration Services provided by IEP enable the MSP to operate and deliver the IEP services within specified scope, quality, time, and cost objectives, and as described in IEP Program Management, Project Management, Software Development Lifecycle, and Operations Management Frameworks.

4.1.1 Program Management

MSP implements the following Program Management activities:

• Manages the program measurement in Quality Assurance Surveillance Plan (QASP), including a process for developing, implementing, and monitoring a set of performance measures to validate that program activities support both business and IT strategies efficiently and effectively.

• Manages the resource on-boarding and clearance process for IEP. This includes working with IRS CORs to complete the Minimum Background Investigation (MBI) for all resources and service providers that support IEP and the resources and subcontractors that support the Help Desk, Service Desk, and Hosting capabilities. The team also works with IRS to request badges and laptops for cleared resources as needed. To support new or ending projects, IEP supported approximately 25 monthly resource-specific MBI actions over the 12-month period ending March 2023.

• Develops and maintains the overall Program Management Plan that clearly delineates the program and project management approach including, at a minimum, the Project Management Institute’s Standard for Program Management.

• Oversees and governs contractual obligations and risks associated with this contract.

• Manages the IRS coordinated review cycle of all work products and deliverables.

• Performs quality assurance of work products and deliverables.

• Manages the Service Strategy process in partnership with the IRS to seek new solutions and initiatives for business problems and provide proactive recommendations and assessment.

• Manages staffing and time tracking of resources.

• Manages Benchmark process with IRS.

• Manages subcontractors, procurement activities, and vendor contractual obligations.

• Manages onboarding of new products and services

• Manages IEP O&M Asset Management including but not limited to:

a) Hardware maintenance agreement with vendors.

b) Software subscription renewals and license entitlement compliance.

c) Manage GFE reporting

d) Manage cloud services agreements

• Performs contract change control through supporting the contractual new services onboarding and fulfillment process by working with stakeholders, CORs and Contracting Officers to create, estimate and coordinate funding of VROMs, Work Requests, and UWRs and participate in review sessions of this documentation. Coordinates agenda, participates in, and supports IEP Technical Review Board (TRB) operation and activities. IEP provided 75 VROMs and supported 150 Work Requests over the 12-month period ending March 2023.

• Manages new work requests from IRS encompassing schedule, cost, risk, funding, billing/invoicing, and contract action statuses and associated communications across Executive, Management, Operational, and Procurement channels.

• Tracks and communicates IRS Automated Front Door (AFD) status of current Work Requests, Change Requests, overdue Deliverables, and Work Products on a weekly and monthly basis.

• Provides quality control and oversight over the IEP O&M Service Level Objective reporting to IRS.

• Operates and maintains the IEP Self Service Catalog for estimating hosting of applications in the

IEP environment.

• Manages the IEP deliverables and work products associated with the task orders and work requests.

IEP completed more than 1,850 deliverables over the 12-month period ending March 2023.

• Manages the IRS AFD Power Automate tool to support the review process of Deliverables, Work

Products, Contractual Change Requests, and Work Request submissions, allowing for sequential and parallel reviews with multiple reviewer groups.

4.1.2 IT Service Management

IEP Operations Management uses several ITIL-compliant tools and ITIL-based processes that work together with established IRS tracking tools, e.g., KISAM, Development and Release Integration and Visibility Engine (DRIVE). The tools and processes have a specific focus on supporting the activities to provide an overarching Operations Management Framework that spans and governs multiple dimensions of the service, including:

• Internal IEP interactions

• Consulting-IRS interactions

• IEP O&M interactions

• IEP enhancement-focused interactions

IEP Service Management practices include the following:

• Change Management

• Release and Deployment Management

• Event Management and Monitoring

• Request Fulfillment

• Incident Management

• Problem Management

• Access Management

• Availability Management

• Capacity Management

• Vulnerability and Patch Management

• Configuration Management

4.2 Portal Infrastructure Services

Portal Infrastructure Services manages a Hybrid Cloud infrastructure spanning Public Government Cloud and Private Cloud environments. The IEP’s hybrid cloud architecture facilitates the build and management of environments required to host applications in FedRAMP and/or FISMA Moderate environments. Portal infrastructure services include architecture support, environment build, infrastructure tuning, patching, hardening, and capability and performance enhancements.

Infrastructure-as-Code (IaC) and automation are leveraged to facilitate rapid and consistent delivery and maintenance of infrastructure services and environments. The IEP Private Cloud is a scalable, virtualized infrastructure housed at vendor managed physical data center facilities furnished with hardware, software, middleware, network connectivity, and peripherals required to establish and host environments, as well as services needed to improve IEP operations in new functional areas arising during the period of performance. As a combined hybrid cloud, both the Public and Private Cloud environments share services and are connected through a highly available Software Defined Wide Area Network (SD-WAN) that also connects to the IRS on-premises data centers. There are 107 IEP applications. 38 are hosted on the Public Government Cloud and 69 are in the Private Cloud.

4.2.1 Network and Firewall

IEP maintains multiple data centers in an active/active state to minimize latency, improve connection speeds for clients, provide fail-over for key toolsets and applications, and meet Service Level Agreement (SLA) requirements. IEP leverages external points of presence distributed across the United States to monitor application health as seen by the service consumers. A robust networking architecture supports the demands and flexibility of IEP. The core of the IEP network provides high-speed (40 Gbps, 10 Gbps, 1 Gbps) layer 2 and layer 3 switching between VLANs, DMZ connections, partner networks, and vendor-diversified outside internet connections. Networks with different security levels are separated via firewalling – virtual, physical, or both. Depending on demand and security requirements, physical and virtual network elements can be used to manage traffic, provide security, and respond to the changing needs of the software-defined data center.

Internet traffic is load balanced across the geographically separated data centers, enabling application availability even during data center outages or maintenance windows. Traffic is also automatically rerouted to secondary locations based on user need and network efficiency, while remaining completely transparent to the end-user. IEP also has the flexibility to configure dynamic or planned fail-over during outages and required maintenance. Configuration and administration are accessible through a web browser interface.

Wide Area Network (WAN) architecture includes interconnection of routers, switches, and firewalls, which help optimize traffic patterns from registered users, MSP data centers, Cloud regions, and IRS data centers that host multiple Enterprise systems/applications. WAN architecture is separated by network zones which include the public internet and the connection between the data centers.

IEP provides the ability to extend network access safely and easily to desktops for remote users, devices, and endpoints after authentication. Remote access to IRS IEP environments is restricted to users connected to the IRS intranet or privileged MSP networks and is specifically designed for operational management as well as for development and testing.

IEP Local Area network (LAN) infrastructure was designed and built with agility and scalability in mind to meet the anticipated future growth demands. The logical LAN design is based on multiple private secured environments that allow ingress and egress traffic through firewalls.

4.2.2 Storage

IEP infrastructure includes storage necessary to meet IEP Service Level Objectives (SLOs). The storage design provides the following services:

• Archival, primary, and transient storage for applications

• High-performance tiered storage for applications with intensive IO requirements

• Applications leverage both Block and Object storage technologies based on requirements

• Fault management, performance reporting, capacity reporting

If the storage requirements grow beyond current capabilities, the design incorporates further deployments to meet application workload requirements.

4.2.3 Computing and Virtualized Infrastructure Management

The IEP hybrid cloud computing approach provides the following:

• Containerized compute and microservice based applications allow IEP to deploy workloads across multiple hardware platforms (cloud or on-premises) with increased automation, orchestration, and on-demand scaling to support next-generation IRS applications.

• Hybrid architecture that allows for automated deployments in both on-premises virtual environments as well as Public Cloud environments leveraging consistent tooling to ensure operational efficiencies, common support models, and consolidated licensing costs.

• Scalable and high-performance computing workloads across a shared virtualized infrastructure and geographically dispersed environments.

• Automation that optimizes available capacity and minimizes unnecessary compute costs by turning compute instances on when they are needed and off when they are not in use.

• Dynamic resource provisioning across compute environments. IEP reallocates compute resources to support peak times and production capacity to development and test environments during seasonal lulls in workload resource requirements. This architecture allows for in-place hardware upgrades without impacting application availability as well as environmental isolation and segmentation.

• Logical environment segmentation (e.g., development, test, production) that enables secure and targeted application environments with varying resource requirements on shared hybrid cloud infrastructure.

4.2.4 Data Center Services

The two data centers, which house IEP infrastructure, are separated geographically across time zones within the United States such that local disasters do not result in a loss of service. Equipment is located within multiple racks and cages within the data centers’ secure FISMA-moderate compliant environments. All data centers use state-of-the-art cabling, wiring, air flow, smoke detection, fire suppression, temperature control, humidity control, and backup power generation. The connection between IEP data centers utilizes a SD-WAN mesh connectivity to the IRS data centers with 1 Gbps bandwidth that is expanding to 10 Gbps.

4.2.5 Dynamic Traffic-Based Scaling

Automation of data center workflows enables rapid provisioning of virtual machines and containers that provide the required services to support IRS applications in the IEP. This serves as the cornerstone of auto-scaling to match demand with resource availability. Additionally, it helps to facilitate patch management, change and configuration management, and disaster recovery.

4.2.6 Internal Interfaces

IEP interfaces with multiple IRS data centers and IRS enterprise cloud environments as depicted in the two diagrams below:

Figure 4-1: IEP 1.5 and IRS Enterprise Computing Center Interfaces

Figure 4-2: IRS Enterprise Computing Center and Treasury Cloud Interfaces

4.2.7 External Interfaces with Other Government Agencies and Commercial

Entities IEP interfaces with the following external entities:

• Department of Health and Human Services (HHS) Center for Medicare and Medicaid Services – administers the Medicare program, while partnering with state governments to administer Medicaid, the Children’s Health Insurance Program (CHIP), health insurance portability standards, and the Affordable Care Act Marketplaces.

• Department of Education – establishes policies on education financial aid and distribution while collecting American school data and disseminating research.

• State Tax Agencies (50+) – multiple departments of revenue, tax boards, finance, administration, etc. across each state that develop or communicate major tax policies, implement actions to increase efficiency of revenue collections, and liaison with legislators, business groups, and professional associations.

• International Data Exchange – electronic delivery point for financial institutions and host country tax authority to transmit Foreign Account Tax Compliance Act (FATCA) reports and data.

• Tax transmitters, e.g., Intuit, H&R Block, and miscellaneous accounting firms.

• Insurance companies and large employers, e.g., Aetna, United Healthcare.

• Information Returns Providers – small business, accounting firm, tax preparation software, health care provider, and individual taxpayer entities holding a Transmitter Control Code (TCC) to file online Information Returns through IRMOD A2A, MeF, and ACA.

• Internal Paper Processing – systems such SCRIPS that transcribe paper transmission and interface with IRMOD and other information return intake applications.

The diagram below depicts IEP 1.5 major external interfaces:

Figure 4-3: IEP 1.5 External Interfaces

4.3 Portal Web Hosting Services

Portal Web Hosting Services provides comprehensive application hosting support for application development, testing, integration, and production, as well as provisions for capacity management, incident management, performance monitoring, and application infrastructure integration support.

4.3.1 Web Hosting Support

The reporting process consists of gathering metrics from multiple data sources and generating reports, which provide a consolidated view into IRS.gov, applications, and IEP infrastructure performance analytics. This process facilitates the gathering of data on an ongoing basis, which provides consistent metrics to IRS on an ad-hoc, daily, weekly, and monthly basis. MSP also provides monthly diagnostic reports detailing the current state of the website. These diagnostic reports provide information regarding broken or inappropriate links, automated section 508 checks, page weight, and searchability.

4.3.2 Capacity Management

Capacity Management is necessary to manage the resources within IEP. MSP manages capacity for licensing, computing, storage, network throughput, and caching bandwidth. MSP and IRS teams meet as needed to review requests for capacity within IEP. Non-production environment requests and Production environment capacity requests are both reviewed and approved.

As part of IEP capacity management, IEP is responsible for development and maintenance of Financial Operations (FinOps) dashboards. For private cloud workloads, IEP provides Capacity and Usage. For Government Cloud workloads, IEP provides monthly Spend and quarterly Capacity, Funding vs. Spend and Cloud Optimization reports. These reports are summarized and tagged at the account, instance type, environment application and services level.

The table below depicts Capacity Management activities and accountabilities using a RACI (Responsible, Accountable, Consulted, Informed) chart.

• Responsible for correct execution of the service or process outcome.

• Accountable for the result and overall quality of the service or process outcome.

• Consulted by those performing the process or delivering the service.

Table 4-1: IEP Capacity Management RACI

Activities M

SP

IR

S

Ap pl ic at io n

O w ne rs a nd

SM

Es

Se rv ic e

Le ve l M an ag em en t

IR

S

Ch an ge

M ag em en t

TI

G

TA

St ak eh ol de rs

IE

P

Ca pa ci ty

G ov er na nc e

IR

S

Collect IEP workload data and characterizations. R C I I A

Perform capacity analysis and planning. R I I I A Develop capacity forecasting techniques and modeling for existing IEP workloads. R C I I A

Establish performance targets and service levels that are obtainable and cost-justified R C C I A

Produce a formal capacity report (e.g., baseline updates, allocation rates, usage/bandwidth trends).

R C I I A

Provide capacity metrics across IEP services and elements (e.g., virtual machine environments, server types, storage, network bandwidth).

R C I I A

Support Treasury Inspector General for Tax Administration (TIGTA) audits targeting impact of changes to IRS data systems and risks.

C C A I R

Track and forecast software licenses and certificates required to operate IEP environments at planned levels.

R C I A

Implement agreed IEP Capacity Governance solutions per the IEP Change Management practice.

R C I C C A

Continuously improve the Capacity Management practice. R C C C A

4.3.3 Incident/Problem Management

Incident Management is supported by IEP Service Desk, which is a single point of contact to IRS’s EOps Enterprise Service Desk and other IRS help desks. This function provides 24x7x365 Incident and Service

Request support, including the identification, triage, troubleshooting, and resolution of both non- Production and Production incidents interfacing with hundreds of IT application developers and testers.

These activities utilize MSP IT Service Management (ITSM) tool and IRS KISAM change management and ticketing tool and interact with existing IRS processes such as IRS Support Restoration Team (SRT) process and Change Request Process for emergencies. IEP’s Service Desk tracks 40+ incident tickets per month across applications and environments.

The tables below depict Incident and Problem Management activities and accountabilities using a RACI (Responsible, Accountable, Consulted, Informed) chart.

• Responsible for correct execution of the service or process outcome.

• Accountable for the result and overall quality of the service or process outcome.

• Consulted by those performing the process or delivering the service.

Table 4-2: IEP Incident Management RACI

Incident Management

Se rv ic e

D es k

Pr ob le m ag em en t

Ch ag em en t

IR

S

Ap p

D ev te am s

Ex te rn al

Ve or s

En te rp ri se

O pe ra tio ns

Su pp or t

Re st or at io n

Te

IT

ec ur ity

Support the operational single point of contact delivered by the IRS' EOps Enterprise Service Desk.

R C A

Identify, register, classify, troubleshoot, triage, resolve, and close IEP incidents related to applications hosted on the IEP platform (MSP-managed components)

R C C C A

Support the triage of non-IEP incidents to specific SMEs per area of expertise and coordinate resolution and recovery actions.

R C I C C C I C A

Manage incidents per IRS Incident Management process, e.g., per defined SLOs, and incident prioritization and ticket resolution times.

R C C A

Detect potential Problems (cause or potential cause of one or more Incidents) and assign to Problem

R C I C A

Incident Management

Se rv ic e

D es k

Pr ob le m ag em en t

Ch ag em en t

IR

S

Ap p s

Ex te or s

En te rp ra

Su pp or t

Re st or at io n ec

Management for formal root cause analysis.

Submit formal change requests in order to implement a workaround or resolution.

R I C C C C C C C A

Develop and maintain an IEP Incident/Problem Management SOP.

R I C C C C C A

Continuously improve the Incident Management practice.

R C I I C C C A

Enable user friendly application maintenance pages as necessary

R I C C I A

Table 4-3: IEP Problem Management RACI

Problem Management

Se rv ic e

D es k

In ci de nt ag em en t

Ch ag em en t

IR

S

Ap p s

Ex te or s

En te rp ra

Su pp or t

Re st or at io n ec

Perform root cause analysis on assigned Problem tickets to derive resolution or identify errors.

R I C C C C I C A

Solicit specialist skills (SMEs) as needed to assist with problem analysis and resolution within their area of expertise.

R I I C C C I C A

Submit formal change requests to implement a workaround or resolution.

R I C C C C C C C A

Develop and maintain an IEP Incident/Problem Management SOP.

R I C C C C C A

Continuously improve the Problem Management practice.

R I C C C C C A

4.3.4 Monitoring

MSP monitors comprehensively the IEP infrastructure 24x7x365.

The table below depicts Monitoring activities and accountabilities using a RACI (Responsible, Accountable, Consulted, Informed) chart.

• Responsible for correct execution of the service or process outcome.

• Accountable for the result and overall quality of the service or process outcome.

• Consulted by those performing the process or delivering the service.

Table 4-4: IEP Monitoring RACI

Activities MSP EOps Incident Management IRS

Perform external monitoring of applications and websites replicating end-user experience. R C A

Perform internal monitoring, alerting, and reporting of infrastructure elements, e.g., applications, networks, servers, bandwidth, databases.

R C A

Monitor resource utilization, performance, availability, and security. R C A

Establish KPIs and thresholds for monitored elements. R C A Evaluate alerts and initiate the incident management process when meaningful and actionable per SLOs. R C A

Send logs to the IRS E2E logging servers for analysis, processing, reporting and auto-ticket generation. R C A

Provide routine and ad hoc IEP 1.5 IT infrastructure and application monitoring reports. R C A

Support IRS authorized third party audit of data, per IRS data storage and retention policies. R A

Provide Application Performance monitoring for analyzing application bottlenecks. R C A

Leverage a data visualization, collection, monitoring, and reporting platform to update system monitoring requirements and thresholds.

R C A

Coordinate end-to-end visibility of platform operations for hosted application with the IRS Monitoring tooling. R C A

4.3.4.1 Logging

IEP is responsible for collecting and storing logs from applications, application platforms, databases, and middleware components. These logs are sent to the IRS’ E2E logging servers to provide application and system level monitoring to IRS E2E monitoring applications for IRS’ analysis, processing, reporting, and auto-ticket generation. IEP initiates Enterprise Security Audit Trails (ESAT) data onboarding for each workload with IRS Cybersecurity and coordinates the transformation and stream of data into the Treasury Cloud hosted ESAT system. IEP performs application-level logging for IRS business analytics, such as IT Reporting tool for the Affordable Care Act (ACA). It also provides IT near real-time log ingestion and search capability as an additional audit log access mechanism for IRS.

4.3.4.2 Dashboards

MSP provides status and reporting dashboards to authorized IRS users and contractors enabling autonomous data analysis and reporting for these capabilities:

• End user statistics

• Website usage

• Application to Application (A2A) transfer health

• Application/infrastructure availability

• Infrastructure and bandwidth usage

• Log analytics

• Website optimization

• Cloud Management

• Capacity Utilization

• Continuous Integration/Continuous Deployment

4.3.4.3 Reporting

MSP performs real-time ad hoc reporting, including generation of metrics from multiple data sources to provide a consolidated view into IRS.gov, applications, and IEP infrastructure performance analytics.

MSP also provides monthly diagnostic reports depicting the current state of the IRS.gov website. The diagnostic reports provide information regarding broken/inappropriate links, Section 508 checks, page weights and search-ability. Formal, planned reports include:

• Senior IT executive report

• Daily Filing Season Report

• Office of Management and Budget (OMB) E-300 Metrics

• Response Time Report

• Help Desk Contact Database Extracts

• Help Desk Operational Metrics

• Help Desk Customer Satisfaction Surveys

• Internet Protocol Version 4/6 (IPV4/V6) Report

• Service Level Objective (SLO) performance report

4.3.5 Application Infrastructure Integration Services

Application Infrastructure Integration Services (AIIS) team provides integration and technical guidance to the IRS application teams. Plans, processes, and procedures for application developer coordination and integration of existing and new applications to the IEP 1.5 environments is found within a “Guide for Application Developers.” This guidance enables the IRS application teams to be able to successfully design, develop, test, and deploy to the IEP 1.5 environments without degrading IEP service performance. More specifically the AIIS team must:

• Provide ongoing integration services for the buildout and testing of all applications regardless of their position in the development lifecycle.

• Perform application architecture and engineering support for new applications.

o Preliminary engineering study o Requirements gathering o Effort-scoping o Serve in an advisory role during the plan, analyze, test and deploy phases of the software development lifecycle o Provide performance testing support.

o Application performance tuning o Performance testing planning and execution o Analysis of performance test results o Standardized performance test environment

• Provide sustained after-hours support for special projects, e.g., supporting application testing in

Enterprise Integration Test Environment (EITE) or Functional Integration Testing FIT environment.

• Support annual disaster recovery (DR)/Alternate processing site/DR performance testing.

• Support onboarding of new applications or redesign/enhancements to the production applications.

• Support creation of the IEP Control Document (CD) and Cloud Based YAML deployment artifacts.

• Maintain the automated deployment process documentation to include any relevant updates or changes.

4.3.5.1 Application Deployment

Application deployments in IEP are streamlined through end-to-end automation and integration of core services described in this section. This enables repeatable and reliable deployment of application changes, while abstracting and automating the detailed integration of services required to support the application execution environment. Deployments can be initiated through the service management tool, or an event-driven interface to alleviate the need for manual processes.

The application interface provides IRS application owners with a mechanism to configure, customize, and deploy fully integrated applications across multiple environments, including Production.

Automation also retains the prior version of an application deployment up to three days after a new deployment is performed in case it is required for emergency rollback. There is also a standardized process to provide an interface for niche application deployments that do not follow one of the three application styles, or new technologies not currently in the standardized platform.

IRS application system owners can submit formal deployment requests via an online service catalog.

IEP Service Catalog helps the IRS customers quickly deploy their application by leveraging Infrastructure as Code (IaC) to provision resources rapidly within the production and non-production environments and maintain the prerequisite synchronization.

The following components are configurable for an application:

• Targeted environments for application deployments

• Application style and components configurations

• Application server configurations

• Webserver configurations

• Messaging server configurations

• Database configurations

• Container configurations

• Queue and network configurations

• Auto-scaling configurations

• API configurations

• Batch configurations

• Micro-services configurations

• Server less configurations

• COTS configurations

4.3.5.2 Application Styles

IEP supports multiple application styles including Application to Application (A2A), Web Application, and Component Application.

4.3.5.2.1 Application to Application (A2A)

IEP platform is designed to largely support web services within a virtual environment. A2A style is generally used for system-to-system interaction. This style enables external applications to connect to IRS web services via HTTPS. This style does not present the user with a graphical user interface because the interaction is primarily system-to-system. The following are potential components within A2A style:

• Application Server

• Relational/NoSQL Database

• Secure File Transfer Protocol (SFTP)

• File/Object Storage

• API/XML Gateway

• Secrets Manager

• Secure Ingress/Egress Gateways

• Service Mesh

• Identify Access Management

• Certificate Management

• Event Streaming/Messaging Services

• Load Balancer

4.3.5.2.2 Web Application

Web Application style provides a web front-end for end user interaction with IEP applications. This style supports applications that require a graphical user interface to interact with end-users. The following are potential components within Web Application style:

• Application Server

• Web Server

• Relational/NoSQL Database

• Content Management System

• File/Object Storage

• Secrets Manager

• Content Caching Service

• Event Streaming/Messaging Services

• Secure Ingress/Egress Gateways

• Service Mesh

• Identity Access Management/Single Sign On

• Event Streaming/Messaging Services

• Load Balancer

4.3.5.2.3 Component Application

Component Application style enables IEP application owners to deploy configurations for a specific technology component instead of a group of technologies like other application styles. This includes the configuration of Commercial-off-the-Shelf (COTS) platforms components such as Forum Sentry API Gateway, McAfee Web Gateway, Layer7 API Gateway, Software AG API Gateway, Istio Service Mesh, Ephesoft Transact, and F5 Load Balancer.

4.3.5.3 Standard Technologies

4.3.5.3.1 Application Servers

Most of the modern applications that IEP hosts are deployed onto a Kubernetes containers platform or onto a serverless compute service. The IEP maintains an assortment of standard base container images that applications may use as part of the deployment (i.e., JBoss, OpenJDK, PHP, etc.). Applications that have not transitioned towards micro-services are hosted on compute instances running JBoss or Internet Information Services (IIS) depending on the application technology stack.

4.3.5.3.2 Web Servers

Apache is the IEP standard web server. A common use of web servers in IEP is to provide separation between the application layer and the public. All Web Application styles utilize a web server for this purpose.

4.3.5.3.3 Authentication and Authorization/Identity and Access Management IEP deploys tools to support authentication and authorization of IRS application services. These tools provide Secure Single Sign-On (SSO) and identity and access management, so that users can be authenticated for access to the deployed applications. Application owners are responsible for populating Enterprise systems/applications policy server information and service credentials to enable the orchestration.

4.3.5.3.4 Database and File/Object Storage

The IEP supports the following database types that are used by IRS Applications:

• Oracle Relational Database

• PostgreSQL Relational Database

• SQL Server Relational Database

• AWS DynamoDB NoSQL Database

• MongoDB NoSQL Database

The IEP additionally supports the following additional storage types used by applications and platforms; all storage offerings provide Data at Rest Encryption support:

• Secure Cloud Object Storage

• Network File Storage

• Windows Server Shared File Storage

• Block Level

4.3.5.3.5 Messaging Services

Messaging Services are used to facilitate communication between the IEP and Enterprise systems/applications. Message Queuing (MQ) technology is currently leveraged to allow multiple IEP applications and services to use a single transport type to transfer messages asynchronously to IRS.

IEP supports the following types of messaging services:

• AWS Kinesis Streams

• AWS Simple Email Service (SES) for internal notification

• AWS Simple Notification Service (SNS)

• AWS Simple Queue Service (SQS)

• IBM MQ

The configuration of the IEP MQ services requires a significant amount of coordination between the shared responsibilities of IEP and IRS. The IRS Middleware Infrastructure Integration Section (MIIS) MQ administrators are responsible for gathering, packaging, and submitting MQ updates and artifacts to the IEP MQ Team. The supplied artifacts from MIIS are needed to fully configure the IEP MQ servers to enable communication to the IRS Enterprise systems/applications. The IEP ITSM tool is utilized to facilitate the transfer of MQ artifacts to the IEP MQ Team via an Application Migration Request ticket.

4.3.5.3.6 Secure File Transfer Protocol (SFTP)

Applications in IEP use a secure file transfer solution and the IRS EFTU system to move attachments between IEP and Enterprise systems/applications.

4.3.5.3.7 Secure Gateway

Secure gateway is a server that scans messages and files sent to IEP for potential threats such as viruses, malware, and intrusions. Secure gateway can validate file types and submission sizes that users upload to verify they comply with defined restrictions.

4.3.5.3.8 API/XML Gateways

API/XML Gateways provide additional security for IRS hosted web services. These Gateways provide a set of capabilities for processing and protecting inbound and outbound transactions, including:

• RESTful service endpoints

• Request validation and transformation

• Anti-virus protection

• Denial-of-service protection

•…