P01 PWS - Application Hosting Compute and Storage.pdf

PERFORMANCE WORK STATEMENT (PWS)

DEPARTMENT OF VETERANS AFFAIRS

Office of Information & Technology

Infrastructure Operations

Application Hosting, Compute, and Storage

Date: January 27, 2026 PWS Version Number: 1.0

Application Hosting Compute & Storage

Contents

1.0 BACKGROUND

2.0 APPLICABLE DOCUMENTS

3.0 SCOPE OF WORK

4.0 PERFORMANCE DETAILS

4.1 PERFORMANCE PERIOD

4.2 PLACE OF PERFORMANCE

4.3 TRAVEL

5.0 SPECIFIC TASKS AND DELIVERABLES

5.1.1 CONTRACTOR PROJECT MANAGEMENT PLAN ... Error! Bookmark not defined.

5.1.2 REPORTING REQUIREMENTS ................ Error! Bookmark not defined.

5.2 SERVER ADMINISTRATION

5.2.1 INFRASTRUCTURE DATABASE ADMINISTRATION (DBA) ............ Error!

Bookmark not defined.

5.3 ENTERPRISE STORAGE OPERATIONS ....... Error! Bookmark not defined.

5.4 CONTAINERIZATION – VA PLATFORM ONE (VAPO)

5.5 ENTERPRISE CLOUD MANAGEMENT AND OPERATIONS

5.6 MAINFRAME SERVICES

5.7 TECHNOLOGY AND PLATFORM APPLICATION SUPPORT

5.7.1 LEAF SUPPORT

5.7.1.1 LEAF TRANISTION PLAN (OPTIONAL TASK)

5.7.2 e-SIGNATURE SUPPORT

5.7.3 SHAREPOINT, NINTEX AND ECFax

5.7.4 ENTERPRISE LOGGING WAREHOUSE (ELW)

5.8 SURGE SUPPORT (OPTIONAL TASK)

6.0 GENERAL REQUIREMENTS

6.1 SECURITY AND PRIVACY REQUIREMENTS

6.2 GOVERNMENT FURNISHED PROPERTY

1.0 BACKGROUND

The Department of Veterans Affairs (VA) Office of Information and Technology (OIT), through its Infrastructure Operations (IO) organization, is responsible for delivering secure, reliable, and modern IT services that enable VA to fulfill its mission of providing healthcare and benefits to Veterans. IO supports a vast and complex enterprise environment spanning on-premises data centers, hybrid cloud platforms, and distributed systems serving millions of users including Veterans, employees, and partners. These services underpin mission-critical systems such as electronic health records, claims processing, benefits administration, cybersecurity, and enterprise-wide communications.

To meet this mission, VA operates within a complex, multi‑vendor IT environment.

The organization requires a comprehensive and integrated approach to hosting and infrastructure services to support mission‑critical applications and data. This includes Application Hosting, Compute and Storage capabilities, as well as specialized services such as Server Administration, Enterprise Storage Operations, Containerization through VA Platform One (VAPO), Enterprise Cloud Management & Operations (VAEC), and Mainframe services. Current operations rely on a hybrid environment of on‑premises and cloud‑based solutions to deliver secure, scalable, and high‑performing compute and storage resources. As workloads grow in complexity and demand, there is an increasing need for modernized platforms that ensure high availability, disaster recovery, and compliance with enterprise security standards.

2.0 APPLICABLE DOCUMENTS

In the performance of the tasks associated with this Performance Work Statement (PWS), the Contractor shall comply with applicable documents identified in the parent contract PWS Section 2.0 Applicable Documents. In addition, the following references are essential to tasks identified in this PWS.

1. NIST SP 800-53 Rev. 5, “Security and Privacy Controls for Federal Information Systems and Organizations,” September 23, 2020 (includes updates as of 12/10/2020)

2. NIST SP 800-37 Risk Management Framework

3. NIST SP 800‑61 (Incident Handling)

4. NIST SP 800‑171 (CUI)

5. VA Directive 6500

6. Trusted Internet Connections (TIC) 3.0 Core Guidance Documents, https://www.cisa.gov/publication/tic-30-core-guidance-documents

7. OMB Memorandum M-19-26, “Update to the Trusted Internet Connections

(TIC) Initiative,” September 12, 2019

8. OMB Memorandum M-22-09, “Moving the U.S. Government Toward Zero

Trust Cybersecurity Principles”, January 26, 2022, https://www.whitehouse.gov/wp-content/uploads/2022/01/M-22-09.pdf.

9. FedRAMP (as applicable)

10. enterprise configuration & change management, https://www.cisa.gov/publication/tic-30-core-guidance-documents https://www.whitehouse.gov/wp-content/uploads/2022/01/M-22-09.pdf

11. Development, Security, and Operations (DevSecOps) policies,

12. data governance,

13. records management, privacy, accessibility (Section 508).

3.0 SCOPE OF WORK

The Contractor shall provide end‑to‑end enterprise technology operations and modernization services that ensure secure, highly available, and performant computing across VA’s on‑premises, hybrid cloud, and mainframe environments. The Contractor shall plan, implement, operate, and continuously improve enterprise hosting services, compute and storage infrastructure, server administration, container orchestration via VAPO, enterprise cloud management and operations, and mainframe operations across the organization’s hybrid environment (on‑premises data centers, commercial cloud(s), and edge locations). This scope encompasses day‑to‑day operations, security, capacity management, performance optimization, disaster recovery, and modernization initiatives, as well as transition‑in/out support, documentation, and knowledge transfer. Services will be automation‑forward and compliance‑driven, with documented architectures, frameworks, and performance reporting that demonstrate availability, resiliency, cost efficiency, and continuous improvement in alignment with VA’s mission.

Objectives:

• Reliability & Availability: Meet or exceed uptime and recovery Service Level

Agreements (SLA) for all in‑scope platforms and services.

• Security & Compliance: Adhere to enterprise security policies and applicable federal standards; maintain continuous Authority to Operate (ATO).

• Scalability & Performance: Provide elastic capacity aligned to workload demand;

optimize performance for mission applications.

• Cost Efficiency: Implement capacity planning, right‑sizing, and automation to reduce Total Cost of Ownership (TCO) and improve utilization.

• Modernization & Agility: Advance containerization (VAPO), refactor workloads where feasible, and integrate legacy mainframe with modern architectures.

• Operational Excellence: Standardize processes (Information Technology

Infrastructure Library [ITIL]‑aligned), automate routine tasks, and provide proactive monitoring and reporting.

4.0 PERFORMANCE DETAILS

4.1 PERFORMANCE PERIOD

The PoP shall 12 months from date of award, with four 12-month option period(s) and Optional Tasks that have the following period of performances:

5.2.2 Storage System Administration – Executed monthly, not to exceed the PoP

5.6.1.1 Leaf Transition Plan – Six months from date of exercise

7.0 Surge Support – Within 10 business days from date of exercise

The overall Period of Performance shall not exceed 60 months.

The Contractor shall ensure resource availability during VA business hours (8:00am Eastern Time through 4:30PM Pacific Time), unless otherwise specified.

4.2 PLACE OF PERFORMANCE

(If place of performance is at a Government facility (Onsite)) Tasks under this PWS shall be performed in VA facilities located in <place(s) of performance>. Work may be performed at remote locations with prior concurrence from the Contracting Officer’s Representative (COR). (Modify as necessary)

Tasks under this PWS shall be performed at Contractor facilities. The Contractor shall identify the Contractor’s place of performance in their proposal submission.

4.3 TRAVEL

(Modify as necessary) The Government anticipates travel under this effort to perform the tasks associated with the effort, as well as to attend program-related meetings or conferences throughout the PoP. Include all estimated travel costs in your firm-fixed price line items. These costs will not be directly reimbursed by the Government.

The total estimated number of trips in support of the program related meetings for this effort is <please enter number of trips>. Anticipated locations include the following, estimated at <please enter duration of trips> days in duration:

1. <List location(s)>

Travel shall be in accordance with the Federal Travel Regulations (FTR) and requires advanced concurrence by the COR. Contractor travel within the local commuting area will not be reimbursed.

5.0 SPECIFIC TASKS AND DELIVERABLES

The Application Hosting, Compute, & Storage (AHC&S) Domain provides enterprise-level hosting and infrastructure services to support VA’s mission-critical applications and data across hybrid environments. Services ensure secure, reliable, and scalable compute and storage capabilities, enabling modernization, operational excellence, and compliance with VA’s technical and cybersecurity standards.

The Contractor shall support the AHC&S domain in accordance with the following functional areas as defined in the parent contract PWS:

• Server Administration

• Enterprise Storage Operations

• Containerization

• Enterprise Cloud Management and Operations

• Mainframe Services

• Technology and Platform Services

• Task Order Program Management

5.1 SERVER ADMINISTRATION

The Contractor shall ensure secure, highly available, and performant server operations across VA’s on-premises and hybrid cloud environments (currently AWS and Azure).

Supported platforms include Linux/Unix (e.g., Red Hat, Oracle Solaris), Windows Server, virtualization technologies (e.g., VMware, vSphere, Red Hat OpenShift Virtualization), and enterprise databases (e.g., Oracle, Microsoft SQL Server).

The Contractor shall deliver reliable, scalable, and compliant operations across all systems.

The Contractor shall design, build, and operate Power BI data models, dashboards, and reports for ~200 IT staff across four Service Lines, integrating sources such as ServiceNow, Apptio, Dell SRM, vCenter, NetApp, SQL Server, Oracle OEM, SharePoint, and Azure. Services include data integration & quality, semantic modeling (DAX), performance optimization, security (RLS/OLS), publishing & refresh operations, documentation, and enduser training, ensuring accurate, performant, and accessible (Section 508) analytics. The Contractor shall provide recommendations on an monthly basis that identify and address opportunities for resiliency, redundancy, and efficiency improvements within the program’s operations, systems, and processes.

The Contractor shall provide comprehensive systems administration, engineering, and modernization support for each security domain defined under this section. Services shall cover full lifecycle management of operating systems and associated infrastructure across VA Information Technology Centers (ITCs), hybrid data centers, and approved cloud platforms. The Contractor shall perform the following:

Operational Coverage: The Contractor shall provide the following continuous operational coverage to ensure systems are operational with 99.9% uptime:

• Provide 24/7/365 operational monitoring.

• Ensure systems are available and operate within capacity.

• Perform triage and diagnostics when issues occur.

• Handle after business hours escalation.

• Maintain infrastructure during rapid releases, emergency changes, or increased multi-stream deployment workloads as required.

• Provide support during normal business hours.

• Ensure continuous after-hours coverage (second/third shifts, weekends, holidays).

• Handle outage monitoring, break/fix services, patching, and escalation.

System Health & Performance: The Contractor shall provide technical support for the following system health requirements:

• Perform proactive health checks, no less than weekly.

• Conduct continuous performance tuning to ensure configurations align with network, industry, and/or best practices and to address changes as they occur.

• Optimize performance tools using telemetry tools (for example, DynaTrace, SAR, Oracle Enterprise Manager, Amazon CloudWatch, Azure Monitor, or equivalents).

Disaster Recovery (DR): The Contractor shall provide the following DR activities to ensure systems meet DR needs:

• Regular frequent testing of systems based on DR plans.

• Validate DR plans to meet Recovery Time Objective (RTO) and Recovery Point

Objective (RPO) targets per system tier.

• Support the documentation of annual DR activities in a DR validation reports and identify corrective actions within 15 business days.

• In the event of a DR and recovery exercises, ensure the implementation of a comprehensive disaster recovery plan, employing VA-approved solutions to guarantee complete data recoverability for all platform-related tools, hosted virtualization clients, and associated servers and components related to the virtualization infrastructure.

Patch and Vulnerability Management:

The Contractor shall install, configure, test, and apply patches, upgrades, updates, and new releases to ensure all systems meet security, performance, and organizational readiness requirements. The Contractor shall operate a standardized patch and vulnerability remediation program that identifies, reports, prioritizes, tests, deploys, and verifies security-relevant updates and compensating mitigations for all in-scope systems. Using current VA security and patching tools (e.g., ACAS, SCAP, HCL BigFix, or equivalents), the Contractor shall identify vulnerabilities, provide remediation recommendations, and track execution of remediation tasks to eliminate vulnerabilities.

Flaw remediation shall be integrated into the configuration management process, with auditable evidence maintained to demonstrate compliance with VA Directive 6500, NIST flaw remediation and vulnerability management expectations (e.g., SI-2/RA-5), and enterprise patch management practices (e.g., NIST SP 800-40). Prioritization shall be risk-based, considering severity, exploitability, asset criticality, exposure, and known exploitation, and shall support emergency changes when required. Post-deployment verification shall confirm patch installation success, vulnerability closure or documented compensating controls, service health, and continued monitoring and logging integrity.

The Contractor shall execute Quarterly Full Stack Patch cycles (QFSDP) and ensure timely patching and compliance in accordance with SLAs.

Security & Compliance: The Contractor shall validate that all systems are secure and compliant with required standards to enforce compliance with Defense Information Systems Agency (DISA) Security Technical Implementation Guides (STIG), National Institute of Standards and Technology (NIST), VA baselines. The Contractor shall identify issues with a remediation recommendation. After approval by the VA, the Contractor shall execute remediations. The contractor shall ensure immutable security logging based on VA standards. The Contractor shall, at a minimum, apply expanded security controls to systems by applying, at a minimum:

• Zero Trust principles

• Identity and Access Management (IAM)

• Security-Enhanced Linux (SELinux) enforcement

• Secure Shell (SSH) hardening.

Release Governance: As releases are executed by VA application owners, the Contractor shall manage release governance per VA Change Management policies by ensuring availability of resources and hardware, addressing system performance issues, and supporting testing. During releases, the Contractor shall ensure 100% pre-validation and rollback readiness. The Contractor shall monitor real-time release telemetry. VA traditionally conducts releases outside of normal business hours.

On-Site Technical Services: The Contractor shall provide onsite hardware support for physical equipment by performing preventive maintenance based on industry standards. The Contractor shall validate and inspect racks, cables, power, and environmental integrity. The Contractor shall identify issues during validation with recommendations. The Contractor shall ensure configurations are optimal.

Automation Services: The Contractor shall implement and operate a governed automation capability that standardizes deployments, configuration enforcement, and repeatable operational workflows across VA-approved platforms while maintaining security, compliance, and service continuity. Automation shall be executed under VA change and configuration governance, with documented security impact considerations, approval controls, and audit-ready evidence in alignment with VA cybersecurity policy and configuration/change management requirements and consistent with NIST security-focused configuration management and change control expectations.

The Contractor shall leverage VA-approved automation tools (or VA-approved equivalents) to reduce manual effort, error rates, and configuration drift. Automation shall be designed to produce stable “Day 1 operations” outcomes for modernized and steady-state systems by ensuring systems are provisioned with validated baselines, enrolled in monitoring and log forwarding, integrated with vulnerability scanning and patch orchestration, and registered in CMDB and automation inventories. The Contractor shall maintain complete documentation and governance evidence, drift detection, patch/vulnerability artifacts, compliance reports, release artifacts, and maintenance logs.

Reporting and Documentation: The Contractor shall maintain accurate documentation of configurations and validated architectures. The Contractor shall recommend process and procedure documentation to capture best practices, improvements, and changes. The Contractor shall document system status and metrics based on governance requirements such as KPIs and SLAs using dashboards or other means approved by VA. By collaborating with VA development, infrastructure, and cybersecurity teams, the Contractor shall deliver reports and system documentation to ensure zero undocumented drift. The Contractor shall register Configuration Items in VA’s system of record (currently CMDB). The Contractor shall recommend improvements for improving registry accuracy and compliance, and implement changes at the VA’s approval. At a minimum, the Contractor shall ensure the following activities are completed and documented:

• System configuration documentation validated architecture diagrams, and drift detection reports.

• Annual Disaster Recovery validation reports, including RTO/RPO compliance evidence.

• Patch deployment logs, vulnerability remediation reports, and surge activation logs.

• Compliance reports (DISA STIG, NIST, VA baselines, Zero Trust, IAM, SELinux, SSH hardening).

• Release governance artifacts (validation, rollback plans, telemetry dashboards, failure analysis reports).

• Preventive maintenance schedules, incident repair logs, and environmental integrity audit reports.

• CMDB audit results, sanitization certificates, and chain-of-custody records (NIST

800-88, VA 6500 compliance).

• Weekly/monthly dashboards, governance packages, and modernization progress reports.

Modernization & Migration: The Contractor shall identify modernization improvements and execute modernization initiatives identified by VA. Support for modernization initiatives shall include migration and transitions (i.e. cloud to on-premise). The Contractor shall track and document modernization and migration activities.

The Contractor shall plan and execute modernization and migration for VA Linux and Unix/Solaris environments to preserve operational stability, security, and, for Solaris, disaster recovery recoverability. The Contractor shall identify modernization opportunities based on lifecycle supportability, patch and vulnerability posture, performance and capacity trends, automation or virtualization maturity, sustainment risk, cost, and Solaris-specific security hardening gaps, and shall provide actionable proposals with scope, dependencies, risks, and implementation approach. The Contractor shall perform discovery and dependency mapping of applications, storage, network or fabric, identity and IAM, DNS, certificates, secrets, integrations, and upstream and downstream services, and shall produce documented migration plans for cloud-to-on-premise, on-premise-to-cloud, intra-data-center moves, and platform transitions or relocations where applicable.

The Contractor shall execute cutovers under VA-approved change governance with pre-cutover readiness validation, documented rollback, and post-cutover verification of service functionality, performance baselines, monitoring and alerting, log forwarding (immutable for Solaris), vulnerability scanning, patch orchestration, access provisioning, and backup and recovery operations. The Contractor shall maintain configuration integrity and prevent undocumented drift by updating CMDB records, baselines, automation or virtualization inventories, and operational runbooks; ensure Day 1 operationalization of monitoring, logging, scanning, patching, backup, and alert routing; and track and store all modernization artifacts, including plans, test results, cutover logs, validation evidence, risks and issues, corrective actions, and completion artifacts, in VA-approved repositories sufficient for audit and operational governance.

For Linux, modernization shall include RHEL and cloud-based Linux in AWS and Azure, preserving operational stability and security posture.

For Unix/Solaris, modernization shall include Oracle Solaris and LDOM/Zone environments, upgrading or transitioning operating environments to supported configurations, validating LDOM/Zone constructs and resource allocations, remediating Solaris-specific hardening gaps, and maintaining platform stability through proactive performance tuning, including InfiniBand or fabric considerations where present.

The Contractor shall provide periodic status reporting (at least weekly or as directed) for all modernization initiatives, including progress against plan, risks/issues and mitigations, completed milestones, upcoming activities, and operational impacts, with escalation of blockers that could jeopardize service availability, security posture, DR recoverability, or delivery timelines.

The Contractor shall also provide specialized engineering support as required to continually maintain VA’s physical compute capabilities, including Oracle SuperCluster engineering, InfiniBand (high-speed interconnect) tuning, high-density compute readiness, and execution of modernization and migration initiatives (including cloud-to-on-premises transitions and Solaris modernization), producing required deliverables such as the System Availability Report, Patch and Vulnerability Management Report, and Performance Table.

Specialized Engineering: The Contractor shall deliver system support specific to the environment as follows:

• Platform Engineering Outcomes (Oracle SuperCluster)

• Daily engineering for Oracle SuperCluster hosting to ensure the platform remains secure, stable, performant, and recoverable in support of VA mission workloads.

• Maintain continuous operational health of SuperCluster components (compute, storage, network fabric, virtualization layers) with proactive detection and remediation of degradation conditions before service-impacting outages occur.

• Sustain 99.9% platform availability (excluding approved maintenance windows) and prevent repeat incidents by completing root-cause analysis and implementing corrective actions for platform defects and systemic issues.

• Maintain validated configuration integrity across SuperCluster engineering baselines (firmware, OS, virtualization constructs, network fabric parameters, storage settings) and ensure no undocumented drift from approved configurations.

• Support break/fix engineering for platform-impacting events, including triage, vendor coordination, workaround implementation, and permanent corrective action, with documented resolution steps and knowledge capture.

• InfiniBand Fabric Tuning

• Maintain and tune InfiniBand to ensure predictable throughput, low latency, and resilience for daily operational loads and peak processing windows.

• Continuously monitor InfiniBand health indicators (link state, error rates, congestion, performance counters) and remediate adverse trends before they cause workload degradation.

• Maintain validated tuning profiles (queue depths, MTU, routing, congestion control parameters as applicable) appropriate to VA workload patterns and document the rationale and rollback method for any tuning changes.

• Ensure no single-point-of-failure conditions are introduced by fabric changes and validate redundancy after changes (e.g., link failover validation and post-change health confirmation).

• Virtualization (LDOMs/Zones)

• Administer and engineer LDOM/Zone environments to ensure secure separation, capacity stability, and consistent performance across hosted workloads.

• Maintain accurate resource allocation (CPU, memory, I/O) aligned to operational demand and prevent oversubscription or resource starvation that degrades production services.

• Ensure virtualization constructs remain compliant with security boundaries and hardening baselines (including privileged access controls, logging requirements, and configuration standards).

• Perform daily operational checks and corrective maintenance of virtualization services (domain/zone status, scheduler behavior, resource pools, service dependencies) to prevent cascading failures.

• High-Density Compute Readiness

• Ensure the specialized platform is capable of supporting high-density compute workloads without stability loss or performance collapse.

• Maintain capacity headroom based on measured utilization and forecasted demand and produce actionable recommendations when thresholds are approached (e.g., scaling, rebalancing, workload placement changes).

• Validate performance readiness using telemetry-driven baselines and implement tuning/corrective actions when degradation is detected.

• Ensure that high-density compute operations do not compromise availability, security controls, logging, or recovery objectives.

• Change and Release Engineering

• Engineer and deploy platform changes so daily operations remain predictable and recoverable.

• Ensure 100% of engineering changes include pre-change validation, documented rollback, and post-change verification that confirms service health and performance stability.

• Support off-hours engineering as required to implement changes, restore service, and monitor release outcomes during elevated-risk periods.

• DR/Recoverability

• Maintain a specialized engineering posture that ensures the platform is recoverable and operationally validated.

• Maintain RTO/RPO supportability for specialized components by validating dependencies, configuration parity (as required), and operational runbooks.

• Participate in DR exercises and produce engineering-specific corrective actions for any identified recoverability gaps.

• Operational Evidence, Documentation, and Knowledge Outcomes

• Produce and maintain the engineering artifacts required for stable daily operations.

• Maintain current runbooks, troubleshooting guides, configuration baselines, and escalation procedures specific to SuperCluster, InfiniBand, and LDOM/Zone operations.

• Capture and retain engineering evidence of monitoring, tuning, incident resolution, and change validation sufficient to support audits and operational governance.

• Provide repeatable procedures that reduce key-person dependency and enable consistent Tier escalation outcomes.

Deliverables System Availability Report Patch and Vulnerability Management Report

Outcome KPI AQL Evidence System Uptime Availability of individual systems within a security boundary is 99.9%.

99.9% System Availability Report

Optimization System performance metrics are maintained within capacity bandwidth (i.e. compute, storage, memory, etc.).

50% - 60% capacity during a reporting period for each security boundary.

Resource monitoring tools in use by VA.

Patch Updates and Compliance

Critical – 30 days;

High – 60 days;

Moderate – 90 days;

Low – per ISO guidance;

Emergent – zero day.

For Internet-accessible systems, mitigate critical

Timelines met 100% of the time.

Maintain 99% compliance within each security boundary.

Patch and Vulnerability Management Report, internal monitoring tools.

findings within 15 days and high findings within 30 days of detection

5.1.1 SERVER ADMINISTRATION SECURITY DOMAINS

Applying the requirements outlined in 5.2, the Contractors shall more specifically support the following:

5.1.1.1 LINUX UNIX ADMINISTRATION

The Contractor shall deliver secure, highly available (99.9% uptime per security boundary), and performance end-to-end Linux/Unix Division (LUD) services to LUD environments including Red Hat Enterprise Linux, Oracle Solaris, and cloud-based Linux systems across on-premises data centers and hybrid cloud environments (AWS and Azure), and approved cloud platforms.

5.1.1.2 WINDOWS SERVER ADMINISTRATION

The Contractor shall provide comprehensive administration, engineering, and modernization support for VA’s Windows Server infrastructure across on-premises data centers and hybrid cloud environments (AWS and Azure). Services shall include full lifecycle management of Windows operating systems, Active Directory, and associated infrastructure components to ensure stability, security, and performance. There are currently 1,707 servers/devices with a projected growth of 10% annually.

5.1.1.3 VIRTUALIZATION SYSTEM ADMINISTRATION

The Contractor shall provide full lifecycle virtualization administration services for VA’s enterprise platforms, including VMware vSphere and Red Hat OpenShift Virtualization, across on-premises data centers and hybrid cloud environments (AWS and Azure).

The Contractor shall also provide supplemental (backup) support for additional virtualization platforms (e.g., Microsoft HyperV) and oversight of cloud resources to ensure continuity of operations.

• Asset & Infrastructure Volume o Virtual Machines (Guest Workloads): Support for a fluctuating volume of

2,200 – 3,500 workloads. This includes provisioning, modification, and decommissioning of legacy VMs and new containerized namespaces.

o Compute Nodes (Hosts): Lifecycle management for 200 – 300 physical hosts (currently Cisco USC, Cisco fabric interconnects, Cisco Intersight solutions). Volume may fluctuate as workloads are consolidated onto higher-density hardware at primary sites.

o Site Topology: Support for 4 Enterprise Data Centers, transitioning to 2 Primary Sites. Scope includes maintaining legacy sites during the transition and assisting in site shutdown activities.

o Disaster Recovery: Management of 1 – 3 Disaster Recovery (DR) clusters, including storage replication and annual failover validation.

• Operational Transaction Volume (Annual Estimates) o Service Requests: 2,000 – 3,500 tickets/year. Includes standard catalog tasks such as VM builds, resource expansion (CPU/RAM), and access management.

o Incidents (Break/Fix): 400 – 800 tickets/year. Includes unplanned outages and performance triage. Performance Target: < 15-minute response for critical incidents.

o Engineering "Work Tasks": 800 – 1,000 tasks/year. These are non-standard engineering/environment/or customer requests requiring manual intervention, architectural review, or custom configuration.

o Change Management: 400 – 750 records/year. Includes preparation and execution of Change Tasks and Change Requests for system modifications, upgrades, and site consolidation activities.

• Dual-Platform Infrastructure Management o VMware Operations: Provide operational support for vSphere (current version 7.x and above), including vCenter administration, ESXi host patching, and cluster resource optimization (DRS/HA).

o OpenShift Operations: Provide enduring support for Red Hat OpenShift, including node lifecycle management, operator updates, and integration with persistent storage and networking.

o Platform Interoperability: Ensure seamless operation between the two virtualization platforms during the multi-year transition, maintaining consistent policies across the hybrid environment.

• Automated Provisioning & Modernization o Infrastructure-as-Code (IaC): Develop, maintain, and refactor automation scripts (e.g., Ansible, Terraform, vRealize Automation) to standardize host and virtual machine provisioning, resource allocations, and configuration management across both platforms.

o Workflow Integration: Maintain API integrations with the IT Service Management (ITSM) system (e.g., ServiceNow) to ensure automated delivery of Service Requests and customer Virtualization requests.

o Modernization Engineering: Support the continuous evolution of the virtualization infrastructure environment by adapting automation to new hardware standards and site topologies.

• Security, Compliance & Audit Defense o Audit Support: Maintain the virtualization environment in a state of "Audit

Readiness" for annual IG and IRS assessments for virtual hosted customers, ensuring 100% compliance with defined security standards (e.g., NIST, HIPAA).

o Access Control: Manage Role-Based Access Control (RBAC) and identity management for the virtualization control plane.

• Specialized Virtualization Engineer Support o Infrastructure & Node Lifecycle: Additional engineering support for rapid virtualization hardware refresh cycles, site evacuation/decommissioning, or parallel virtualization farm build-outs during major migrations or datacenter optimization efforts.

o Workload Migration Services: Dedicated execution of large-scale VM migrations (e.g., moving 500+ VMs to OpenShift or new datacenters) with strict downtime limits (<1%).

o Automation Refactoring: Targeted engineering efforts to rewrite legacy virtualization automation workflows for compatibility with new virtualization platform versions or security mandates.

o Advanced Capacity Planning: Develop forecasting and trend analysis to support virtualization hardware procurement decisions during site consolidation.

5.2 ENTERPRISE STORAGE SUPPORT

The Contractor shall provide comprehensive Enterprise Storage Operational for the Enterprise Backup (EBU) and Enterprise Storage Management (ESM) services under the VA Infrastructure Operations Service's Storage Management Service Line (SMSL).

This support is integral to ensuring the protection, recoverability, and integrity of Veterans' data across Health and Benefits Information Technology systems. The combined EBU and ESM system platforms include but are not limited to greater than 32.67PB+ of raw disk capacity, Open System and Mainframe Storage data hosted across: 10+ CommVault Storage Clusters (5 Cisco HSX, 3 HPE HSX, and 2 Dell HSX), 1 Commvault Commcell, 8 Dell Media Agents, and 4 Dell Commservs, 95+ HSX nodes; 4 Netapp Clusters with 26 AFF-A900 nodes, 2 Dell EMC Clusters with 12 PowerMax 8500 directors, 16 Cisco MDS9700 series switches, 6 Cisco MDS9250i switches, 2 Windows 2019 Standard bare-metal servers, utilizing NetApp Active IQ Unified Manager (AIQUM) and Dell EMC Storage Resource Manager (SRM) monitoring tools that span VA locations across all US time zones.

EBU OVERVIEW

Enterprise Backup (EBU) provides Data Protection, ensuring the recoverability and safeguarding of Veterans' data on Health and Benefits Information Technology (IT) systems. EBU delivers services to aid Data Center consolidation; provide continuous service and cost competitiveness; increase System/Service availability; support cross-organization synergy; and maintain the highest standards by aligning security practices with current and future organizational goals. The EBU team is responsible for the Data Protection for over 27.71 Petabytes (PB) of Franchise Fund supported Product Lines and Information Systems protected data copies. EBU protects our customer data, ensuring secure services for Personal Health & Identifiable Information, Federal Tax Information, VA Federal Records, and other data types in use by the VA. EBU provides data protection services for FF and IOS infrastructure servers and database instances, securely protecting up to 1800 TBs of data daily. To accomplish this, EBU employs the Commvault Backup and Recovery software platform, which is a comprehensive data protection application that provides backup, recovery, and disaster recovery capabilities across on-premises, cloud, and Software as a System (SaaS) environments. The computing environment consists of 3 Dell CommServers (central control) staged in an N+2 configuration, 1 Dell CommServer (test system); 26 Cisco Hyperscale Storage Servers; 29 HP Hyperscale Storage Servers; 15+ Dell Hyperscale Storage Servers and 12 Media Agents (storage interface servers). In addition, the Data

Insights environment consists of 2 Index servers, 1 at AITC and 1 at PITC and 4 threat scan servers: 1 each at each enterprise data center (AITC, CRRC, HITC, and PITC).

These systems are deployed in an enterprise configuration which provides reliable services during daily operations, contingencies, tests, and transitional activities associated with the EBU data protection mission. EBU provides an N+2 data protection via primary data copy (onsite) and secondary data copy (offsite). The data retention for these copies is one month for primary and 3 months for secondary. EBU manages

16.58 PBs of disk storage to accommodate the primary and secondary data copies and all their change data. Change data is the specific data that has been altered, added to, or deleted since the last backup was performed. EBU uses Hyperscale storage systems provided by the VA Infrastructure as a Managed Service (IaaMS) exercised options. The IaaMS storage options allow EBU to rapidly adjust to our customer storage capacity requirements. EBU services use proven enterprise architecture, establishing an agile platform. This platform is deployed to support Information Technology Centers and can fully support new customer locations, including other VA facilities and private company-hosted VA Enterprise Cloud (VAEC) sites. Our enterprise solution is deployed to allow expanded capabilities for increasing speed, flexibility, and capacity. Modern technology allows EBU to implement a "Recover Anywhere" solution, to include supporting a VA Hybrid Cloud solution. A Hybrid Cloud solution allows EBU customers to maintain positive control of government data by having protected data copies in both private company facilities hosted in a government cloud (VAEC) with an additional copy at a government-owned facility. EBU service offering includes Data Protection administration, reporting, troubleshooting, and training for client administrators & operators. Reporting support includes user-friendly dashboards and reports that aid clients during any data protection review or audit of their systems. Our primary mission is the protection of customer data, so we focus on the ability to restore systems to the most current operational status. Through best practices, our retention policies allow for a greater history of recovery images. EBU architecture ensures protected data is maintained at a secure alternate (off-site) location to allow customer recoverability from a large-scale (site-wide) disaster. EBU systems are designed with encrypted disk storage systems that maintain full compliance with all government regulations. These systems are used to support, but are not limited to, the backup and restoration of all versions of Windows, Linux, and UNIX operating systems, SQL, and Oracle database systems. As new technology, software, and updates are released and developed, our Team and partnered Contractors work diligently to add new product support, to ensure full protection for our Veterans' data. EBU data protection services are designed to ensure the utmost confidentiality and security of customer information. Importantly, EBU does not engage in collecting, processing, retaining, or sharing personal information about individuals.

As part of its commitment to data security, EBU creates encrypted copies of customer data. These encrypted copies may contain individual information; however, due to the encryption process, the specific content and type of data within these copies remain inaccessible to anyone except authorized customers. Only supported customers can verify the content and type of their respective data within these encrypted copies. This ensures that while EBU maintains a high level of security through encryption, customers retain control over their data and can attest to its accuracy and relevance. In summary, EBU prioritizes data protection by not collecting or sharing personal information while utilizing encryption methods that safeguard customer data. Only authorized users can access and verify their encrypted data.

EBU MISSION FUNCTIONS

Data Backup Management: One of the primary functions of a data protection and restoration team is to implement and manage comprehensive data backup solutions.

This includes scheduling regular backups, ensuring that all critical data is backed up securely, and verifying the integrity of backup files. The team must also determine the appropriate backup frequency (daily, weekly, etc.) based on the agency’s operational needs and compliance requirements.

Disaster Recovery Planning: The team is responsible for developing and maintaining a DRP for the data protection platform. This plan outlines procedures to follow in the event of data loss due to disasters such as cyberattacks, natural disasters, or hardware failures. It includes strategies for restoring systems and data quickly to minimize downtime and ensure business continuity. Through the actions to ensure continued operations of the data protection platform, the EBU team assures its ability to aid supported customer recovery operations.

Data Restoration Services: In the event of data loss, the team provides restoration services to recover lost or corrupted data from backups. This involves identifying the extent of data loss, selecting appropriate backup versions for restoration, and executing recovery processes while ensuring minimal disruption to ongoing operations.

Security Compliance and Risk Management: The team ensures that all data protection measures comply with relevant regulations (such as FTI, GDPR, HIPAA) and industry standards. This includes conducting risk assessments to identify vulnerabilities in data management practices and implementing necessary controls to mitigate risks.

Monitoring and Reporting: Continuous monitoring of backup systems is essential for identifying potential issues before they escalate into significant problems. The team generates regular reports on backup success rates, storage utilization, Recovery Point Objectives (RPOs), Recovery Time Objectives (RTOs), and compliance status to keep stakeholders informed.

User Training and Awareness Programs: To enhance overall data security, the team conducts training sessions for employees on best practices for data handling, recognizing phishing attempts, secure password management, and other relevant topics that contribute to a culture of security within the organization.

Incident Response Coordination: In case of a security incident or breach affecting data integrity or availability, the team coordinates incident response efforts. This includes assessing damage, communicating with stakeholders, executing recovery plans, and documenting lessons learned for future prevention.

Technology Evaluation and Implementation: The team evaluates new technologies related to data protection—such as advanced encryption methods or alternate solutions—and implements them as needed to enhance existing capabilities. Staying updated with technological advancements ensures that the agency’s data protection strategies remain effective against evolving threats.

Vendor Management: If third-party vendors are involved in providing backup or restoration services (e.g., storage providers), the team manages these relationships by evaluating vendor performance against SLAs and ensuring compliance with security standards.

Documentation Management: Maintaining thorough documentation of all policies, procedures, configurations, incidents handled, recovery tests conducted, etc., is crucial for accountability and continuous improvement within the team’s operations.

On-Site Activity Requirements: For a Data Protection System Administrator (DPSA) at the Department of Veterans Affairs Office of Information Technology, handling systems that contain Controlled Unclassified Information (CUI), several statutory and regulatory requirements necessitate both physical and logical protection measures. These provisions often suggest or necessitate an on-site presence to ensure compliance and optimal security. Here are the relevant statutory and regulatory frameworks: Federal Information Security Modernization Act (FISMA) - Citation: 44 U.S.C. § 3551 on-site duties to meet stringent access control, incident response, and system maintenance requirements; Health Insurance Portability and Accountability Act (HIPAA) - Citation:

45 CFR Parts 160, 162, and 164 with similar requirements to FISMA; Veterans Affairs Information Security Improvement Act - Citation: 38 U.S.C. § 5721 on-site presence at facilities to audit and manage systems directly; VA Directive 6500 – Information Security Program - Citation: VA Handbook 6500 emphasizes the need for both physical and logical security measures, which can involve on-site responsibilities; NIST SP 800- 53: "Security and Privacy Controls for Federal Information Systems and Organizations" outlines safeguarding measures for physical and operational security; IRS Publication 1075 – Tax Information Security Guidelines includes specific requirements for physical and logical security controls that could require physical verification and presence. The Key Activities That Require On-Site Presence: Physical Security Measures; Access Control; Incident Response; System Maintenance and Upgrades; Audit and Compliance; and Data Transfer and Destruction.

ESM OVERVIEW

ESM is a system platform that provides multiple product offerings under VA Infrastructure Operations Service's (IOS) Storage Management Service Line (SMSL).

ESM provides a comprehensive range of mission functions to ensure unmatched data integrity, security, and performance for hosted data. ESM Data Storage Management team excels in configuring, monitoring, and maintaining efficient storage solutions tailored to meet current and future needs. Through our Data Replication products, we safeguard information by creating resilient data copies across multiple locations, minimizing data loss during unforeseen events. ESM specializes in Disaster Recovery Planning, developing robust data storage strategies to swiftly restore operations following any catastrophic failures. ESM experts continuously engage in Performance Monitoring & Optimization, refining system efficiency and removing bottlenecks to enhance overall productivity. With an unwavering focus on Data Security Management, ESM implements stringent measures, including encryption and access controls, ensuring compliance with government regulations and standards. ESM proactive Capacity Planning and adept storage contracts management ensure resources are available to meet growing demands and that vendor relationships are effectively managed. ESM provides meticulous Storage Documentation & Reporting for transparent operations and quick issue resolution. Additionally, ESM’s end user support services guarantee our customers receive prompt assistance and training, empowering them to leverage our cutting-edge storage solutions effectively.

ESM hosts enterprise-class storage platforms within VA Information Technology Centers. ESM provides this enterprise-level storage and optional mission-related replication for applications supporting our nation's Veterans. ESM offers optional replication services to support customer Business Continuity and DR requirements.

ESM platforms’ data-at-rest storage is compliant with National Institute of Standards and Technology (NIST) and Federal Information Processing Standard (FIPS) security requirements. ESM enhances storage efficiency by implementing compression, deduplication, and compaction technologies across the entire storage management process.

ESM assures compatibility and synergy with our customers and their storage technological requirements by providing a multitude of supported storage interconnection types. Supported storage interfaces include Storage Area Network (SAN), Network Attached Storage (NAS), and Direct Attached Storage Device (DASD).

These interfaces allow ESM to support Block Storage, File Storage, Object Storage, Hybrid Storage, and Cloud Storage. The flexibility of the ESM platform enables rapid adaptation to the evolving systems and applications used in the service of our nation’s Veterans. By embracing advanced technology and stringent security standards, ESM stands as a cornerstone in the effort to provide reliable, secure, and efficient data management solutions for VA.

The ESM product offerings include Custom Performance Data Storage, High Performance Data Storage, Medium Performance Data Storage, Low Performance Data Storage, Object-Based Data Storage, and Mainframe Data Storage. The ESM product offerings are also available with Replication (Data Mirroring) to enhance our customers' disaster recovery options. All ESM products are hosted on premium enterprise-class data storage systems, which include 24 hours a day, 7 days a week system monitoring and incident response. Our customers' ESM-hosted storage availability is 99.9% for mission-critical applications, 99.5% for mission-essential applications, and 99% for routine applications.

ESM MISSION FUNCTIONS

Data Storage Management: The primary function of a production storage team is to manage the storage of data across various systems. This includes configuring, monitoring, and maintaining storage solutions that ensure data is stored efficiently and securely. The team must assess current storage needs, forecast future requirements, and implement scalable solutions to accommodate growth.

Data Replication Services: Replication involves creating copies of data across different locations or systems to ensure availability and reliability. The replication team is responsible for setting up and managing replication strategies that minimize data loss in case of hardware failures or disasters.