Justification and Approval Redacted_Redacted.pdf

Enterprise Splunk License Agreement VA-24-00069537

JUSTIFICATION

FOR AN EXCEPTION TO FAIR OPPORTUNITY

1. Contracting Activity: Department of Veterans Affairs (VA) Office of Procurement, Acquisition, and Logistics Technology Acquisition Center 23 Christopher Way Eatontown, NJ 07724

2. Description of Action: The proposed action is for a Firm-Fixed-Price delivery order to be issued under the National Aeronautics and Space Administration (NASA) Solutions for Enterprise-Wide Procurement (SEWP) V Government-Wide Acquisition Contract (GWAC) for brand name Splunk term (workload) software license maintenance support, Splunk Enterprise Security (ES) software license maintenance support, Splunk Information Technology Service Intelligence (ITSI) software and associated maintenance support, as well as professional services support for migration, and setup and configuration of Splunk VA Enterprise Cloud (VAEC).

3. Description of the Supplies or Services: VA, Office of Information and Technology (OIT), Office Information Security, Cybersecurity Operations Center (CSOC), Enterprise Command Operations (ECO) and Infrastructure Operations (IO) have a requirement for brand name Splunk term (workload) software license maintenance support, Splunk ES software license maintenance support, as well as Splunk ITSI and associated maintenance support. Splunk shall provide professional services support for migration, setup and configuration of Splunk VAEC, along with training. Optional tasks are included for additional Splunk term software license maintenance support; Splunk Enterprise Security software license maintenance support; and Splunk Professional Services to support the existing Splunk VA platform.

VA requires maintenance and support for a minimum of 20,000 virtual central processing units (vCPUs) for 51.572 Terabytes of existing Splunk Enterprise perpetual licenses. VA CSOC manages, protects, and monitors the cyber and security posture of the VA and currently uses Splunk on-premises perpetual licenses to perform Security and Compliance analysis along with Infrastructure and Operations Management. Splunk serves as the log aggregation engine for machine data generated by VA Information Technology (IT) systems within physical, virtual, and cloud infrastructures, and provides record of all user transactions, customer behavior, machine behavior, security threats, fraudulent activity, and other activity in the raw data form that can be stored indefinitely.

VA CSOC uses this data to provide rapid incident response, correlation, and in-depth monitoring across all data sources, monitor equipment uptime, and rapidly pinpoint and resolve problems with security and performance. VA ECO will utilize the Splunk data in a shared construct with VA CSOC configured in the VAEC to provide enterprise level analysis teams with a definitive record of all key performance indicators (KPIs) for the VA IT infrastructure and application environment and other activity in a raw data form that can be stored indefinitely.

The term licenses will allow VA ECO the ability to continue monitoring the availability and performance of the telehealth and VA applications. Another component, Splunk ITSI module, provides the VA ECO the capability to develop glass tables and dashboards to display the services and KPIs within the VA environment in support of the VA and Telehealth programs. Glass tables are unique dashboard styles which allows VA to surface one or more values derived from searches and place the values anywhere on the table. VA ECO will utilize these glass tables and dashboards to provide rapid incident response, event correlation, and in-depth monitoring across all data sources and external partners, monitor equipment uptime, and proactively identify, pinpoint, and resolve performance problems across the VA and Telehealth program.

Additionally, Splunk ITSI will provide VA with the capability to respond to urgent unplanned events like the COVID-19 pandemic and/or natural disasters, by rapidly implementing monitoring dashboards requiring immediate visibility into the performance of critical systems.

The term license will allow for growth in capacity to meet compliance to the Office of Management and Budget (OMB) issued Memorandum 21-31, Improving the Federal

Capabilities Related to Cybersecurity Incidents,1 on August 27, 2021, in accordance with Executive Order 14028, Improving ring, and after a cybersecurity emorandum 21-31 describes logs that agencies must capture as well as any required retention times. It also establishes a maturity model to track agency implementation. This maturity model of Event Logging levels 1-3 will exponentially increase the license capacity requirement between 120TB and 140TB.

VA implemented Splunk Enterprise to support Telehealth for the Electronic Health Record Modernization-Integration Office (EHRM-IO) program, and for CSOC to have the capability to view data for cybersecurity operations from (Federal Information Security Modernization Act) FISMA Low to FISMA High systems that reside within the

-premises, Amazon Web Services Government Cloud Service Provider and the Microsoft Azure Government Cloud Service provider environments. The Security Incident Event Management (SIEM) is required to operate in a FISMA/FedRAMP High security environment; therefore, all the data which is analyzed in the SIEM must also reside in a FISMA/FedRAMP High environment.

Further, VA requires Splunk Professional Support Services required for configuration of the shared data structure, as well as on-site and off-sites support services for the management, training, and administrative services related to the configuration and maintenance of the VAEC Splunk Cloud.

The period of performance is 12 months from date of contract award with 2 12-month option periods, as well as optional tasks for maintenance and support for existing Splunk Enterprise term licenses, additional vCPUs, and additional Splunk Professional

Services for additional management, training, and administrative services. The total estimated price of the proposed action, inclusive of all option periods and optional tasks, is .

4. Statutory Authority: Pursuant to Federal Acquisition Regulation (FAR) 16.505(a)(4), the following brand name requirement is being justified and approved as an exception to fair opportunity under the statutory authority of Section 41 U.S.C. 4106(c)(2) as implemented by FAR 16.505(b)(2)(i)(B), entitled, providing the supplies or services required at the level of quality required because the supplies or services ordered are unique or highly specialized.

5. Rationale Supporting Use of Authority Cited Above: Based on market research, as described in Section 8 of this Justification, it has been determined that limited competition is available among authorized resellers for the required brand name Splunk term license maintenance and support and professional services. Only Splunk brand name products and services are capable of supporting the CSOC, ECO, VA, and Telehealth monitoring requirements and the requirement for rapid response to emergent issues.

In fiscal year 2015, VA competitively awarded a contract for the deployment of the Splunk Enterprise solution in order to satisfy VA CSOC monitoring requirements. Due to

COVID-

of this pandemic, VA began a transition to cloud based solutions. In order to meet capacity and ongoing monitoring requirements, the follow-on contract was awarded in fiscal year 2021. Functionality across cloud and on-premises environments was required that only Splunk could provide. The current VA environment still exists on-premises and in VA Private clouds. Supporting Executive Order 14028 requires a logging solution that can function and provide a single pane of glass across all environments which only Splunk can do. As an ongoing effort to migrate from on-premises to the VAEC, the existing hybrid solution utilizes perpetual ingest model licenses (51.572 Terabytes for daily ingest). While this licensing model was previously able to provide these monitoring requirements, the new Executive Order 14028, (ZTA), VA is currently implementing the OMB memorandum 21-31 and thus cannot continue to properly function without significant enhancement of the existing infrastructure to accommodate this additional strain to the system. The additional strain will increase logging significantly. VA estimates upwards of triple the log capacity required to be retained. VA estimates that these licensing costs would exceed $28 million in hardware/software augmentations per year not including the additional required VA labor and configuration, for a total of $84 million over three years.

Additionally, this would take a minimum of 24 months to implement if using the current perpetual ingest licensing model. These figures are based on historical information for previous implementations. Therefore, VA has determined a term workload licensing model is more cost effective than a perpetual ingest licensing model in which VA currently maintains. Further, based on extensive technical analysis it was determined that the current VA perpetual ingest licensing could be transitioned and hosted in the secure VAEC utilizing term workload (vCPU) licensing. Hosting perpetual ingest licensing in the VAEC provides limited scalability to implement full OMB 21-31 and ZTA requirements that is required for Splunk Enterprise currently owned by VA at the FedRAMP High security level required for SIEM which will be alleviated by transitioning to the term workload licensing. No other solution, other than the Splunk solution, can provide the required security and functional requirements for cyber investigations along with an intuitive single pane of glass. Specifically, this unique feature allows cybersecurity to utilize the same data sources in a shared environment to combat cyber threats while providing VA Leadership from different organizations such as the Veterans Health Administration (VHA), OIT, and EHRM-IO the ability to view the functionality of the system in real time, which will directly provide leadership data, increasing its ability to make informed decisions and report more accurately on potential system downtime.

For instance, this function would allow VHA the ability to verify status of the functionality of its telehealth systems, which are critical to providing direct care for Veterans and reducing strain on the VA Medical Centers while cybersecurity simultaneously monitors for threats which could jeopardize the systems. Failure to obtain this function would

, and access to mental health resources for suicide prevention and/or mental health issues. ITSI can link the Splunk Core instance with other monitoring tools as well as Splunk Core products from disparate partners in the EHRM-IO environment such as Department of Defense (DoD) and Oracle Health Government Services (formerly known as Cerner). Further, it correlates data feeds from the current monitoring tools (Computer Associates Intrascope, Solarwinds, Dynatrace and AppDynamics, OpenSearch, Sentinel and Varonis) into a single presentation of data. The Splunk product provides End-to-End visibil point and respond to issues within the network. No other product can collectively provide such visibility across the VA cloud and on-premises. Failure to provide this functional requirement will produce inefficiencies for VA leadership by requiring the review of multiple webpages from multiple tools to manually review and analyze the systems in order to assess overall operational health which effectively decreases visibility and increases time to resolution.

Additionally, EHRM-IO has strict compatibility requirements to integrate with external partners. Specifically, partners such as the DoD and Oracle Health Government Services are both utilizing the Splunk Core log aggregation tool in its environments. The proprietary language utilized by Splunk is not compatible with any other known brand

Electronic Health Record Modernization effort, as VA positions itself to have a single EHR with DoD. Any other potential product would therefore limit the data feeds VA could receive from DoD and Oracle Health Government Services partners thereby preventing full End-to-End visibility of the shared Federal EHR system. Furthermore, the on-premises Splunk Enterprise version currently in use by Cyber Security is only compatible with brand name Splunk products. A pilot with any other product would not yield like-for-like results for determining the viability of transitioning all the cyber security Splunk products to the cloud offering.