NNG15SD27B 36C10B21F0152.docx

Control Number XX-XXX Enterprise Splunk License Agreement VA-21-00012193

Contract Opportunity Award Notice

PRODUCT SERVICE CODE

SUBJECT

CONTRACTING OFFICE'S

ZIP-CODE

SOLICITATION NUMBER

BASE NOTICE TYPE

ARCHIVE

DAYS AFTER THE AWARD DATE

RECOVERY ACT FUNDS

NAICS CODE

SET-ASIDE

CONTRACTING OFFICE

ADDRESS

DESCRIPTION

See Attachment

POINT OF CONTACT

(POC Information Automatically Filled from User Profile Unless Entered)

AWARD NUMBER

AWARD AMOUNT

LINE ITEM NUMBER

AWARD DATE (MM-DD-YYYY)

AWARDEE NAME

CONTRACTOR'S DUNS NUMBER

AWARDEE ADDRESS LINE 1

AWARDEE ADDRESS LINE 2

AWARDEE ADDRESS LINE 3

AWARDEE ADDRESS LINE 4

AWARDEE CITY

AWARDEE STATE

AWARDEE ZIP CODE

AGENCY'S URL

URL DESCRIPTION

AGENCY CONTACT'S EMAIL

ADDRESS

EMAIL DESCRIPTION

AWARD INFORMATION

ADDITIONAL INFORMATION

GENERAL INFORMATION

* = Required Field Contract Opportunity Award Notice 7A21 Enterprise Splunk License (VA-21-00012193) 07724 36C10B21Q0264 N 541519 Department of Veterans Affairs Technology Acquisition Center 23 Christopher Way Eatontown NJ 07724 heera,.polanco@va.gov

NNG15SD27B36C10B21F0152

$4,250,145.92 05-27-2021

V3GATE, LLC

808862598

555 MIDDLE CREEK PKWY STE 120

COLORADO SPRINGS

CO

80921 heera.polanco@va.gov

JUSTIFICATION

FOR AN EXCEPTION TO FAIR OPPORTUNITY

2. Description of Action: The proposed action is for a firm-fixed-price delivery order to be issued under the National Aeronautics and Space Administration (NASA) Solutions for Enterprise-Wide Procurement (SEWP) V Government-Wide Acquisition Contract (GWAC) for brand name Splunk Cloud Software as a Service (SaaS) subscriptions, Splunk Information Technology Service Intelligence (ITSI) subscriptions which include maintenance support, and Splunk Professional Services. Optional tasks are included for the addition and/or renewal of Splunk SaaS subscriptions, as needed; renewal of Splunk perpetual software license maintenance support; Splunk Enterprise Security software license maintenance support; Splunk Professional Services; and Training, as required to support the existing Splunk VA platform.

3. Description of the Supplies or Services: VA, Office of Information and Technology (OIT), Office of Information Security, Cybersecurity Operations Center (CSOC), Enterprise Command Operations and Office of Electronic Health Record Modernization (OEHRM) each have a requirement for brand name Splunk Cloud SaaS subscriptions, Splunk ITSI subscriptions which include maintenance support, and Splunk Professional Services to support the migration, setup, and configuration of Splunk licensing in the VA Enterprise Cloud (VAEC) environment. Additionally, VA requires optional tasks for Splunk Cloud SaaS subscriptions; renewal of Splunk perpetual software license maintenance support; Splunk Professional Services; and Training. These optional tasks provide VA the flexibility to efficiently optimize and manage its existing Splunk platform portfolio; accommodate future changes regarding VA’s need and ability to transmit data within its Information Technology (IT) infrastructure; to read data from, and transmit requisite data to, the Department of Homeland Security (DHS) Continuous Diagnostics and Mitigation (CDM) and Department of Defense (DoD) Electronic Health Record Modernization (EHRM) Programs; and efficiently utilize existing investments in Splunk perpetual licensing.

The Splunk Cloud SaaS subscriptions and Splunk perpetual licensing required herein not only comprise the existing Splunk Enterprise logging solution which provides VA the ability to monitor the availability and performance of the VA infrastructure and applications that read and analyze data feeds from application performance monitoring products, but will ensure the continuation of that enterprise-wide monitoring capability critical to the security health of VA, without interruption. Splunk serves as the log aggregation engine for machine data generated by VA IT systems within physical, virtual, and cloud infrastructures, and provides the only means to aggregate the required data generated by VA’s IT infrastructure and applications. VA utilizes the Splunk-aggregated data in a shared construct with Splunk Cloud SaaS subscriptions to provide enterprise level performance monitoring of all applications within VA’s expansive network. Splunk’s Security Event Incident Management (SEIM) logging tool provides VA CSOC’s security and analysis teams with a definitive record of all user transactions, customer behavior, machine behavior, security threats, fraudulent activity, and other activity in a raw data form that can be stored indefinitely.

The Splunk ITSI module provides VA with the requisite capability to develop single view pane of glass dashboards that display the services and key performance indicators within the disparate VA environment which spans across, and requires aggregated information from, healthcare, benefits processing, cemetery services and cybersecurity. Single pane of glass dashboards are critical for rapid incident response, event correlation, and in-depth monitoring across both internal data sources and external partners such as third party healthcare providers, the DoD/EHRM Program, and the DHS/CDM Program. These dashboards allow VA to monitor equipment uptime and proactively identify, pinpoint, and resolve performance problems across the VA enterprise, including telehealth and OEHRM applications that are currently configured in the VA Splunk SaaS Cloud. These applications, such as VA Video Connect, Phillips TeleICU, Veterans Data Integration and Federation Enterprise Platform and Caregiver Record Management Application, are essential for providing crucial health care services to Veterans. The monitoring capabilities allow for rapid detection and troubleshooting of performance degradation and system issues, enabling VA to detect problems and resolve them before they impact Veteran-care.

Further, VA requires Splunk Professional Services and Training necessary for configuration of the shared data structure, as well as on-site and off-site support services for the management, training, and administrative services related to the configuration and maintenance of the VAEC Splunk Cloud and VA Splunk SaaS subscriptions.

Finally, VA requires renewal of maintenance and support for up to 51.572 Terabytes of VA’s existing Splunk Enterprise perpetual licenses; this includes, support and maintenance, upgrades and patches, access to Splunk documentation, access to “Splunk Answers” website, live product roadmaps, online case submission, telephone support for routine and priority one (critical) cases, and email and website support. These maintenance and support activities serve to maintain compliance and optimization of the enterprise Splunk platform.

The period of performance is 12 months from date of contract award with four 12-month option periods for continuation of the Splunk Cloud SaaS subscriptions and continued maintenance and support of the Splunk Enterprise perpetual licenses. Optional tasks are required for additional Splunk Cloud SaaS subscriptions, Splunk Professional Services and Training that may be exercised at any time during contract performance and as the need arises. VA anticipates continued usage of the Splunk platform over the next five years to maintain continued compliance with the DoD/EHRM and DHS/CDM Program requirements which are expected to continue through 2024 and beyond. The total estimated value of the proposed action, inclusive of all option periods and optional tasks, is

4. Statutory Authority: Pursuant to Federal Acquisition Regulation (FAR) 16.505(a)(4), the following brand name requirement is being justified and approved as an exception to fair opportunity under the statutory authority of Section 41 U.S.C. 4106(c)(2) as implemented by FAR 16.505(b)(2)(i)(B), entitled, “Only one awardee is capable of providing the supplies or services required at the level of quality required because the supplies or services ordered are unique or highly specialized.”

5. Rationale Supporting Use of Authority Cited Above: Based on market research, as described in Section 8 of this Justification, it has been determined that only the Splunk brand name SaaS solution is capable of complying, integrating, and interoperating with the required DHS/CDM and DoD EHRM Programs, and providing the single pane of glass functionality required across the VA enterprise.

In 2016, DHS/CDM, which has been congressionally charged with fortifying the security posture of federal government networks through the administration of policies and practices for information systems, announced that Splunk Enterprise was selected for use at the 24 largest civilian Departments and Agencies covering 97 percent of the Federal Government civilian workforce to meet the Federal Information Security Modernization Act (FISMA) of 2014 and National Institute of Standards and Technology (NIST) standards. DHS/CDM procured the first year of Splunk CDM licensing and integration services for all agencies. Since VA was already utilizing the Splunk platform for SEIM, the integration with DHS/CDM core functional requirements was seamless, thereby accelerating VA’s ability to come into statutory compliance. FISMA and NIST standards require Federal Government agencies to have the ability to effectively respond to cybersecurity incidents by analyzing massive amounts of data from large network and IT infrastructures. DHS/CDM aggregates this massive amount of data in a Federal dashboard with log aggregation, correlation, visualization and alerting data from each of the Agency level dashboards, including the VA Agency level dashboard, to form a picture of cybersecurity health across civilian agencies. In order for VA to transmit its data in a usable format to the required Federal dashboard, which utilizes Splunk Enterprise software, and maintain compliance with Federal reporting requirements VA must continue the use of the Splunk solution. Splunk software is a proprietary product with which no other aggregation software is capable of interoperating and/or communicating; specifically, no other software is capable of interpreting and/or displaying the ingested data. The encrypted data transmission required from the Agency level dashboards to the Federal dashboard needs to be in the same format in order to be readable and useable. No other brand name software solution can decode Splunk’s proprietary encrypted data feed sent from Agency level dashboards. Due to the proprietary constraints within the Splunk software, the use of any other software to support and feed the VA Agency level dashboard to the DHS/CDM-operated Federal dashboard would cause the data in these transmissions to be jumbled and wholly unusable. As such, VA would be able to neither meet the DHS/CDM and Federal reporting requirements nor comply with FISMA and NIST standards. Specifically, the DHS Cybersecurity & Infrastructure Security Agency (CISA) CDM Program has broken FISMA reporting requirements into various functional areas. Only through continued use of Splunk Enterprise can VA meet reporting requirements on the following functional areas: Hardware Asset Management, Software Asset Management, Configuration Management, Vulnerability Management, Manage Trust in People Granted Access, Manage Security Related Behavior, Prepare for Contingencies and Incidents, Respond to Contingencies and Incidents, Manage Audit Information and Manage Operations Security. Without Splunk, agency data relative to the foregoing functional areas could not be aggregated into the Agency dashboard and therefore, could not be transmitted to or read by the Federal dashboard. The impact of not meeting these requirements would cause serious degradation in the ability of the Government to respond effectively to critical incidents as there would be a gap in the ability to analyze data necessary to respond appropriately.

Additionally, VA OEHRM has strict compatibility requirements to integrate and interoperate with DoD and Cerner's Splunk Core log aggregation tools. Accordingly, it is only through utilizing Splunk ITSI that VA can meet the requirements for security and performance monitoring of a single Federal Electronic Health Record jointly between DoD and VA. Since DoD is utilizing Splunk, and VA must share data with and receive data from DoD, VA must also continue to use Splunk software, a proprietary product with which no other software is capable of interoperating and/or communicating. The proprietary language utilized by Splunk is not compatible with any other brand name products; therefore, the requirement and ability to integrate, interoperate, and communicate with VA’s EHRM partners is only available through the use of Splunk. This is due to the inability of any other product to decode Splunk’s proprietary encrypted data feed received from DoD. Due to these proprietary constraints within the Splunk software, the encoded data transmission required between VA and DoD needs to be in the same format to be readable and useable. VA’s use of any other software would cause the data in these transmissions with DoD to be sent and received as jumbled and rendered useless. As such, VA would be left unable to utilize critical Veteran data from DoD necessary to provide essential and quality care to our Nation’s Veterans.

Many Splunk monitoring tools are currently combined to provide the single pane dashboarding capability needed to rapidly identify and diagnose issues and provide VHA, OIT, and OEHRM leadership the visibility required to manage the OEHRM program. Only through continued use of Splunk can ITSI link the Splunk SaaS subscription with other monitoring tools, as well as Splunk Core products from disparate partners in the OEHRM environment such as DoD and Cerner. Specifically, OEHRM requires a single pane of glass for monitoring all the applications integrated with the new Electronic Health Record in a single dashboard. This single pane of glass gives VA the ability to view the functionality of multiple systems and applications comprising the OEHRM environment, in real time, on a single screen. This capability enables two or three individuals to monitor the health of OEHRM systems which would otherwise take at least 10 individuals to provide the same coverage by viewing the various systems individually; this results in a reduction of over 13,000 man-hours per year in order to monitor the OEHRM environment. Importantly, the aggregated health of multiple systems in a single pane reduces the mean time to detect anomalies and/or failures which directly results in a shorter time to resolve issues which are preventing the provision of essential services to our Veterans. For instance, this function provides VHA the ability to verify status of the functionality of its telehealth systems, which are critical to providing direct care to the Veteran and reducing strain on the VA Medical Centers. Utilization of ITSI single pane of glass for telehealth has directly reduced the average time of resolution from 16 hours down to 4 hours. This provides leadership direct access to streamlined focused data, increasing ability to make informed decisions, and reporting more accurately on potential system downtime. Failure to provide this functional requirement has proven to exacerbate inefficiencies, resulting in VA leadership having to review excessive amounts of data from multiple tools in order to gauge overall system health; this not only decreases visibility but needlessly increases time to resolution. Specifically, VA’s increased visibility has reduced application performance error rates from 1,000 per day to four per day resulting in an increase in preventing system outages that allow for the continuation of healthcare and benefits to Veterans. Since the Splunk solution provides End-to-End visibility of VA’s environment from a single point, VA’s ability to rapidly respond to issues within the network is dramatically increased.

Extensive market research has revealed that no other product has like for like capabilities necessary for VA to read data from, and transmit data to, both the DHS/CDM and DoD/EHRM programs. Likewise, no other product has the ability to meet VA’s functional requirement to provide an intuitive single pane of glass view across the enterprise. VA conducted a proof of concept for a single medical center with the Elasticsearch, Logstash, Kibana (ELK) Suite prior to procuring Splunk Cloud SaaS. The result was 4,800 man-hours expended at a cost of over $600k in labor alone for a product that did not meet the requirements to integrate with DoD due to its inability to translate the data received in Splunk proprietary code format into a readable one. o Even if there were another solution that could meet VA requirements to comply, interoperate, and integrate with DHS/CDM and DoD/EHRM, or possessed the required single pane of glass functionality, VA has, over the past several years, made significant financial investments in non-aggregation cybersecurity tools and technology procurements at an annual cost in excess of $300 Million. A key requirement of each of those investments was the ability of the procured product solutions to seamlessly integrate with VA’s existing Splunk Enterprise platform. The ability for those VA procured, non-aggregation cybersecurity tool platforms to integrate with Splunk is perhaps the most significant requirement as to why the products were procured by VA. The use of any other solution would require significant efforts to replace or reconfigure infrastructure in place.

meet EHRM requirements for a single pane of meet this require Furthermore, Splunk is the only solution that currently has a FedRAMP Moderate certified SaaS Cloud environment that meets the VA Authority to Operate (ATO) security requirements. The VA Splunk SaaS Cloud has an approved ATO and no other solution would be able to meet the time-to-market requirements of providing a rapid SaaS Cloud based solution in order to provide immediate monitoring capabilities necessary to provide continuing support during times of a national emergency.

The required Splunk Professional Services will provide VA with the implementation and configuration of Splunk, as well as the training of the solution. Only Splunk authorized resellers can provide the certified Splunk engineering services due to the requirement of access to the proprietary code of the solution. Splunk owns the proprietary Intellectual Property rights to the enterprise software and only Splunk or a Splunk reseller has the required access to the source code and other proprietary data of the software in order to meet the Government’s needs. Access to this code is also needed to ensure all services provided are properly configured. Additionally, only Splunk authorized resellers have the requisite and intimate knowledge of the product to provide VA the required support and ensure optimal operation of the solution. The intricacy of VA’s requirements including the medical, business, integration of third-party applications and security and monitoring of all events increases the need for Splunk Professional Services to ensure operational integrity of the Splunk software products and meet mission objectives.

Additionally, VA requires renewal of maintenance and support for up to 51.572 Terabytes of VA’s existing Splunk Enterprise perpetual licenses. Due to the proprietary nature of the Splunk source code, only Splunk or an authorized reseller can provide the necessary bug fixes, software patches necessary to keep the software operating at an optimal level. The use of any other vendor for maintenance and support would require a replacement of the existing licensing in place which would lead to estimated duplicative costs of approximately $200 million dollars over this five-year effort. These estimated costs, based on previous acquisition history and current market research, would not be recovered through competition.

Failure to maintain the Splunk SaaS subscriptions, maintenance and support of the perpetual licensing, and the associated services increases the risk to VA’s cybersecurity ecosystems and degrades VA’s ability to monitor all applications that are currently being reported through Splunk capabilities; this would directly and negatively impact VA’s mission to care for Veteran. Specifically, the inability to provide real time status of the infrastructure and view the system from End-to-End, would negatively impact VA’s daily ability to continue providing expanded telehealth services, which would particularly and significantly limit access to care for remote Veterans in rural locations. Without the proactive monitoring that is being provided, small system degradations would be compounded resulting in overall system failure and causing the halting and cancellation of over 45,000 doctor/patient sessions per day until the systems are restored. This would prove especially harmful during a national emergency or future pandemic in which telehealth is the only means for doctor/patient interactions.

Continuation of utilizing Splunk is the only way to ensure VA maintains a solution that can aggregate disparate data sources, including other monitoring tools, into single pane of glass; perform log collection and analysis, schema-less data ingestion and searching; has long term detailed and summary data storage; has highly customizable application of machine learning algorithms; can meet VA’s needs for Application Performance Monitoring (via Splunk ITSI module); and both integrate and interoperate with the DHS/CDM and DoD/EHRM Programs, while preserving the performance and cybersecurity integrity of VA’s disparate systems.

6. Efforts to Obtain Competition: Market research was conducted, details of which are in Section 8 of this Justification. This effort did not yield any additional sources that can meet the Government's requirements. It was determined, however, that limited competition is available among authorized resellers for the required brand name products and services. In accordance with FAR 16.505(a)(4)(iii)(A)(1), this Justification will be provided with the solicitation to all appropriate NASA SEWP V GWAC holders. Furthermore, in accordance with FAR 5.301, 16.505(b)(2)(ii)(D), and 16.505(a)(4)(iii), the award notice for this action will be synopsized on the Contract Opportunities website and this Justification will be made publicly available within 14 days of award.

7. Actions to Increase Competition: In order to remove or overcome barriers to competition in future acquisitions for this requirement, the Government will continue to perform market research to determine if there are any new products or maintenance services available in the marketplace that would enable future requirements to be competed.

8. Market Research: The Government technical experts conducted market research from February 2020 through October 2020 to evaluate the industry leading log file and data aggregation products. Two potential alternative solutions were identified: XpoLog; and a tool suite comprised of ELK. As a preliminary matter, neither solution was found to be FedRAMP high or moderate certified and, therefore, neither are viable SaaS based offerings. Although XpoLog is currently being utilized by VA for log aggregation on-premise, it was determined not feasible on an enterprise level because it is not capable of providing the functional requirement of an intuitive single pane of glass view. Furthermore, it is incapable of reading data from, and transmitting requisite VA data to, the DHS/CDM Federal Dashboard and DoD/VA EHRM Program due to the proprietary constraints within the Splunk software currently utilized by both DHS/CDM and DoD. Therefore, XpoLog is not capable of meeting the Government’s unique needs. Likewise, although the ELK Tool Suite is a cloud-based technology, it was determined that it also does not meet the Government’s requirements on an enterprise level since it does not provide the functional requirement of an intuitive single pane of glass view. Furthermore, it is incapable of reading data from, and transmitting requisite VA data to, the DHS/CDM Federal Dashboard and DoD/VA EHRM Program due to the proprietary constraints within the Splunk software currently utilized by both DHS/CDM and DoD. Therefore, ELK is equally incapable of meeting the Government’s unique needs. Neither of these potential sources identified can integrate or interoperate with the required VA partners as detailed above in Section 5. Based on the market research above, including the proof of concept relative to ELK’s solution as discussed in Section 5, VA technical experts have determined that Splunk is the only product that can meet each of the Government’s requirements; no other product meets, or is capable of being modified to meet, the Government’s unique and integrated needs.

Additional market research was conducted in April 2021 utilizing the NASA SEWP V GWAC Provider Lookup Tool to identify NASA SEWP resellers that could provide the required brand name Splunk license subscriptions and support. The search yielded 76 Splunk resellers capable of providing this requirement, including 16 that are contract holders in Group B (2) Service-Disabled Veteran Owned Small Businesses (SDVOSB) and registered and verified in the Vendor Information Pages (VIP) database. There are seven SDVOSB Splunk authorized resellers; therefore, limited competition is expected among SDVOSBs for this proposed action. Furthermore, market research was conducted on the VIP database which found that two or more VIP verified vendors are capable of meeting the Government’s requirements under North American Industry Classification System code 541519.

9. Other Facts: None

10. Technical and Requirements Certification: I certify that the supporting data under my cognizance, which are included in this justification, are accurate and complete to the best of my knowledge and belief.

Date: ___________________________

Director, Cybersecurity Management Signature: _______________________

11. Fair and Reasonable Cost Determination: I hereby determine that the anticipated price to the Government for this order action will be fair and reasonable based on anticipated limited competition. Additionally, NASA SEWP V has already determined the prices on contract to be fair and reasonable and more competitive pricing may result from competition. Finally, the successful quote will be compared to the Independent Government Cost Estimate.

Date:___________________________

Procuring Contracting Officer Signature: _______________________

12. Procuring Contracting Officer Certification: I certify that this justification is accurate and complete to the best of my knowledge and belief.

Date: ___________________________

Procuring Contracting Officer Signature: _______________________

Concurrence In my role as Head of Contracting Activity, based on the foregoing justification, I hereby concur with the acquisition of brand name Splunk Cloud SaaS subscriptions, Splunk perpetual software license maintenance support, Splunk Professional Services and Training on an other than fair opportunity basis pursuant to the statutory authority cited in Section 4 above, subject to the availability of funds, and provided that the property and services herein described have otherwise been authorized for acquisition.

Date: ________ Signature: ____________________________________

Head of the Contracting Activity Technology Acquisition Center Office of Procurement, Acquisition, and Logistics

Approval In my role as Agency Senior Procurement Executive, based on the foregoing justification, I hereby approve the acquisition of Splunk Cloud SaaS subscriptions, Splunk perpetual software license maintenance support, Splunk Professional Services and Training on an other than fair opportunity basis pursuant to the authority cited in Section 4 above, subject to availability of funds, and provided that the property and services herein described have otherwise been authorized for acquisition.

Date: ________ Signature: ___________________________________

Executive Director and Senior Procurement Executive Office of Acquisition and Logistics Department of Veterans Affairs

Page 1 of