NNG15SD25B 36C10B20F0366_1.docx

Red Hat Enterprise Linux Software Subscription VA-20-00050658 Renewal and Technical Support

Red Hat Enterprise Linux Software Subscription VA-20-00050658 Renewal and Technical Support Contract Opportunity Award Notice

PRODUCT SERVICE CODE

SUBJECT

CONTRACTING OFFICE'S

ZIP-CODE

SOLICITATION NUMBER

BASE NOTICE TYPE

ARCHIVE

DAYS AFTER THE AWARD DATE

RECOVERY ACT FUNDS

NAICS CODE

SET-ASIDE

CONTRACTING OFFICE

ADDRESS

DESCRIPTION

See Attachment

POINT OF CONTACT

(POC Information Automatically Filled from User Profile Unless Entered)

AWARD NUMBER

AWARD AMOUNT

LINE ITEM NUMBER

AWARD DATE (MM-DD-YYYY)

AWARDEE NAME

CONTRACTOR'S DUNS NUMBER

AWARDEE ADDRESS LINE 1

AWARDEE ADDRESS LINE 2

AWARDEE ADDRESS LINE 3

AWARDEE ADDRESS LINE 4

AWARDEE CITY

AWARDEE STATE

AWARDEE ZIP CODE

AGENCY'S URL

URL DESCRIPTION

AGENCY CONTACT'S EMAIL

ADDRESS

EMAIL DESCRIPTION

AWARD INFORMATION

ADDITIONAL INFORMATION

GENERAL INFORMATION

* = Required Field Contract Opportunity Award Notice D319 Recompete of Red Hat ELA (VA-20-00050658) 07724 36C10B20Q0575 N 541519 Department of Veterans Affairs Technology Acquisition Center 23 Christopher Way Eatontown NJ 07724 heera.polanco@va.gov kathryn.pantages@va.gov

NNG15SD25B36C10B20F0366

$11,849,154.67 09-23-2020

THREE WIRE SYSTEMS, LLC

783193175

3130 FAIRVIEW PARK DR STE 425

FALLS CHURCH

VA

22042 heera.polanco@va.gov heera.polanco@va.gov

JUSTIFICATION

FOR AN EXCEPTION TO FAIR OPPORTUNITY

2. Description of Action: The proposed action is for a firm-fixed priced delivery order to be issued under the National Aeronautics and Space Administration (NASA) Solutions for Enterprise-Wide Procurement (SEWP) V Government-Wide Acquisition Contract (GWAC) for the renewal of existing enterprise-wide brand name Red Hat software subscriptions which include renewal of software and associated maintenance and technical support.

3. Description of the Supplies or Services: VA, Office of Information and Technology, Service Delivery and Engineering, Infrastructure Operations has a requirement for the renewal of 10,653 existing enterprise-wide Red Hat software subscriptions which includes renewal of software as well as associated maintenance and technical support that is used throughout VA’s Enterprise Linux operating systems (OS). The existing Red Hat software architecture and infrastructure allows for VA to monitor the performance of the current computers controlled by the Enterprise Linux operating system to ensure reliable operations and provides VA the capability to satisfy ongoing maintenance, security, and operational requirements for existing development, test, validation, and production systems executing on Linux-controlled computers. The required Red Hat software maintenance shall consist of signed Red Hat source code and binaries version upgrades, patches, fixes, updates, and security resolutions. The technical support shall provide for Red Hat Technical Account Management, which shall be the focal point for Red Hat Platforms support, consisting of access to Red Hat technical support advisors/specialists during normal business hours. Technical support includes the following: early identification of deployment issues, periodic risk assessments, quarterly reporting of service performance metrics, and provides for the analysis, troubleshooting, and resolution of any other Red Hat software product-related problems or questions encountered by Government users with the specified software products. Together, the software maintenance and technical support ensures the Red Hat software remains fully operational and is continually updated with signed Red Hat source code, software, and update binaries. The software subscriptions consist of: Red Hat Enterprise Linux (REHL) for Virtual Datacenters with Smart Management + Satellite, Premium; Red Hat Enterprise Linux Server with Smart Management + Satellite, Premium (Physical or Virtual Nodes); Red Hat Enterprise Linux Server with Smart Management + Satellite, Premium, Confirmed Stateside Support (Physical or Virtual Nodes), RHEL servers running in GovCloud; Red Hat Directory Server; Renewal ‐ Red Hat Directory Server (Replica); Renewal ‐ Red Hat Enterprise Linux for IBM System Z, Premium with Smart; Red Hat Enterprise Linux Workstation Premium Support (24x7 Phone and Web support; Resilient Storage For Red Hat Enterprise Linux Server; Red Hat Virtualization (2-sockets), Premium 24X7 Phone and Web Support; Red Hat Certificate System (Starter Pack); Technical Account Management Services for Red Hat Platforms; and Red Hat Learning Subscription Basic.

This requirement includes seven optional tasks for the base period and seven optional tasks for the option period, for the purchase of additional new Red Hat software subscriptions to meet the growing needs of VA on premise and in the cloud. The additional new Red Hat software subscriptions include: Red Hat Enterprise Linux Server with Smart Management + Satellite, Premium (Physical or Virtual Nodes); Red Hat Enterprise Linux Server with Smart Management + Satellite, Premium, Confirmed Stateside Support (Physical or Virtual Nodes), RHEL servers running in GovCloud; Red Hat Ansible Automation, Premium (24X7 Phone and Web Support), (100 Managed Nodes); Red Hat Quay, Premium (1 Deployment); Red Hat OpenShift Container Platform with Application Services (Portfolio), Premium, (64 Cores or 128 vCPUs); Red Hat OpenShift Container Platform with Application Services (Core), Premium, (64 Cores or 128 vCPUs); and Red Hat OpenShift Container Storage, Premium (2 Core).

The Period of Performance for this effort shall be 12 months from September 29, 2020 through September 28, 2021, with one 12-month option period and optional tasks for additional software subscriptions. Optional tasks may be exercised at the discretion of the Government at any time during performance of the delivery order for a period of 12 months. The overall period of performance of the order shall not exceed 36 months to allow VA maximum flexibility to exercise the optional tasks up to the end of the option period. The total estimated value of the proposed action is $----, inclusive of the option period and optional tasks.

4. Statutory Authority: Pursuant to Federal Acquisition Regulation (FAR) 16.505(a)(4), the following brand name requirement is being justified and approved as an exception to fair opportunity under the statutory authority of Section 41 U.S.C. 4106(c)(2) as implemented by FAR 16.505(b)(2)(i)(B), entitled “Only one awardee is capable of providing the supplies or services required at the level of quality required because the supplies or services ordered are unique or highly specialized.”

5. Rationale Supporting Use of Authority Cited Above: Based on the market research as described in Section 8 of this justification, it was determined that limited competition is available among resellers for the required Red Hat software subscriptions and associated maintenance and technical support. Only signed Red Hat source code and binary software subscriptions, provided by Red Hat or an authorized reseller, can meet all of VA’s requirements, specifically, its compatibility and interoperability with VA’s existing Linux Server infrastructure that is solely based on Red Hat Linux OS as well as overarching VA baseline security requirements.

There are approximately 200 different programs within VA using applications hosted only on Red Hat such as Enterprise Veterans Self Service (eBenefits), Benefit Discount Card/Veterans Identification Card, Loan Guaranty, Joint Legacy Viewer (JLV). In addition to these local VA programs, a collaboration between VA and Department of Defense (DoD) exists for HealthShare Enterprise Platform, VA/DoD Identity Repository, and Community Care Reimbursement. These applications have already been developed, implemented, tested, validated, and certified using the VA Assessment and Accreditation process. No other brand name product can work with the existing 200 programs. Specifically, use of any other source would require VA technical experts to re-code all the applications used by these existing programs. VA would have to modify all existing code of these 200 programs, resulting in extensive delivery delays and extensive duplicated costs. VA would have two options, both of which would result in harm to VA, if a source other than RedHat was used. The first option would be to completely shut down the legacy environment, stand up the new OS on the existing servers, reprogram and compile each application under the new OS and then stand up the newly developed application under the new OS source. This would result in a complete stoppage of critical VA programs that provide necessary services to Veterans. These services include, but are not limited to, access to Veterans’ records and Veteran data as well as payments to Veterans. A total stoppage of these services would be catastrophic to Veterans’ livelihood, patient care and wellbeing. For example, JLV is a clinical application that provides an integrated, read-only display of health data from the DoD, VA, and private sector partners in a common data viewer. This program is critical to patient care as DoD and VA clinicians use JLV daily to access real-time medical data in the treatment of their patients. Another program that uses Red Hat is eBenefits, which is a joint VA/DoD web portal that provides resources and self-service capabilities to Veterans, Service members, and their families to research, access and manage their VA and military benefits and personal information. This program allows Veterans to apply for benefits online, submit new claims, order prescription medications, securely message their physicians, and obtain official military personnel documents. Family members and Active Service members also rely on the eBenefits program for important services and information. As previously stated, all of these programs support VA Mission Critical business applications which must remain operational 24/7/365 even if and during a transition, and each would be required to undergo validation testing to determine if it could be executed properly on an alternate source OS. VA has performed similar OS migrations, with cost estimates of approximately $166,600 per program or $33 million for the approximate 200 programs. This cost would be in addition to the millions of dollars VA would have to invest in the new physical/virtual server infrastructure costs and associated VA employees’ hours. The second option would be for VA to buy a completely new server environment; load the prospective “new” OS, replicate servers from each of the 200 programs, reprogram each application under the new OS, completely migrate each program’s associated archived data, and instantiate each database, associated scripts, and Disaster Recovery sites. As stated in the first option, this OS migration would cost approximately $166,600 per program or $33 million for just the migration of the approximate 200 programs, millions of dollars in physical/virtual server infrastructure costs and associated VA employees’ hours. In addition, VA would have to continue to pay for the old OS subscriptions while the new OS development is completed. This duplication of effort would cost VA approximately $12 million annually to run the current Red Hat OS until the migration is completed which would take approximately two years, totaling an overall investment of $24 million. The overall cost impact of this option would be approximately $57 million. In addition to the monetary costs, this effort would require all 200 programs to undergo an application reprogramming and compilation, to incorporate the unique OS nuances which involves the entire application’s staff of system engineers, database developers, and program manager, etc. to focus their time and attention on their specific new application instantiation and take them away from their normal schedules properly maintaining Veteran programs and data. These efforts would also introduce the risk of delays and interruption of existing services currently provided to the Government and its customers, including Veterans. Due to the labor hours required and the number of programs involved, if VA were to choose an alternate source OS at this time, a delivery delay of a minimum of two years is anticipated. Given the delays and duplication of costs associated with both options, VA would not recoup costs by moving to an alternate source OS.

In addition to the above, use of any other brand name software subscription would require a complete redesign, re-implementation, retesting, and Authority to Operate (ATO) certification of the configuration to the alternate source OS for each of the 200 programs. Based upon past VA Enterprise Operations ATO certification experience, new certification cost estimates are approximately $75,000.00 per program or $15 million for the approximate 200 programs that are currently hosting Red Hat software. The current RedHat subscription has already been granted ATO certification and thus would not result in duplicated costs. Based on this, VA could potentially incur costs up to $72 million by moving to alternate source subscriptions and OS.

Additionally, no other vendor can deliver non-publicly available REHL software subscriptions that are capable of producing signed Red Hat Linux OS binary updates, patches, or trademarks with these OS requirements because only Red Hat or its authorized resellers obtain signed source software and updates directly from Red Hat as per Red Hat’s authorized reseller agreements. Only Red Hat or an authorized reseller can provide the required signed Red Hat Linux binary, non-public Red Hat Linux source OS compatibility, immediate Red Hat Linux binary signed patches for severity one bugs, signed Red Hat Linux source code binaries for vulnerability updates, and major and minor release and management capability, allowing VA to continue its ongoing Red Hat Linux infrastructure support while also meeting the growing needs within the RHEL infrastructure. Using publicly available Linux source code, to support VA’s Red Hat Linux instances with publicly available source code to provide security updates and patches would put VA at risk of a material breach of the Red Hat Enterprise Agreement as other vendors’ only source of software is the open source upstream community project, CentOS, which is a public repository. This results in no other brand name software subscription, including RedHat’s open source version, being able to meet all of VA's security requirements. Specifically, only the certified version of Red Hat meets VA OS requirements of OS versioning control, vulnerability management, and data mining protection with respect to National Institute of Standards and Technology (NIST) Special Publication 800-53. These security measures restrict data obtained from data stores and data that is available as open source information residing on external sites because that data has the potential for code modification. By using open source RedHat, the chain of custody is broken resulting in a non-certified version. VA requires OS certifications that come with signed software binary which are invalid with and do not transfer to the upstream CentOS public repository. This means any updates provided by the open source upstream community project CentOS do not meet this requirement and would fail the following security requirements:

1. Common Criteria International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) 15408 certification

1. Federal Information Processing Standards (FIPS) 140-2 certification -- from NIST for FIPS 140-2 validation

1. VA Handbook 6500 and NIST Security Technical Implementation Guide (STIG) compliant – these requirements are only provided via the Red Hat provided and non-upstream CentOS available Security Content Automation Protocol (SCAP) -security-guide software, thus publicly available Red Hat source systems cannot attest their current technical control implementations using vendor provided and NIST National Vulnerability Database (NVD) validated tools

1. Section 508 compliance, Red Hat only validates their products and software via Voluntary Product Accessibility Template, not CentOS Finally, no other brand name maintenance and technical support can meet the Government’s requirements. RHEL OS requires Premium 24X7 Phone and Web Support, and Technical Account Management Services for Red Hat Platforms. RHEL has proprietary rights to the OS and associated technical support. Therefore, the required associated technical support must be compatible with the existing OS. Only RHEL, or an RHEL reseller, has the rights to perform the associated technical support. Any other source must have access to the proprietary rights to ensure all the associated technical support provided is properly configured to both the OS and infrastructure. Without access to the proprietary rights, another source would not be able to push through signed RHEL source code and binaries updates. Having access to RHEL proprietary rights makes the signed binary compliant with VA’s overarching regulations.

Any layered products offered by non-authorized Red Hat resellers are invalidated because non-authorized Red Hat resellers would not have a cooperative support agreement. Without a cooperative support agreement, the vendor would be without access to Red Hat’s engineering teams and would not be able to escalate requests to Red Hat, use Red Hat engineering tools, or gain insight into Red Hat’s roadmap to provide advanced hot fixes and non-public patches for VA systems. Only Red Hat or an authorized Red Hat reseller would have access to Red Hat’s Customer Portal, ticket history, knowledge bases, support tools. If issues arise where non-authorized vendors provide problem escalation and they could not resolve the problem, VA would be at a standstill because non-authorized resellers do not have access to Red Hat’s Technical engineering teams or staff, and VA would have to purchase an additional support contract from Red Hat to obtain escalated solutions.

6. Efforts to Obtain Competition: Market research was conducted, details of which are in Section 8 of this justification. This effort did not yield any additional sources that can meet the Government's requirements. It was determined, however, that limited competition is anticipated among resellers for brand name RHEL products and the associated software maintenance and technical support. In accordance with FAR 16.505(a)(4)(iii)(A)(1), this justification will be provided with the solicitation to all appropriate NASA SEWP V GWAC holders. Furthermore, in accordance with FAR 5.301, 16.505(b)(2)(ii)(D), and 16.505(a)(4)(iii), the award notice for this action will be synopsized on the Contract Opportunities website and this justification will be made publicly available within 14 days of award.

7. Actions to Increase Competition: In order to remove or overcome barriers to competition in future acquisitions for the requirements described herein, the Government’s technical experts will continue to perform market research to determine if there are any products which become available that will enable future requirements to be competed.

8. Market Research: The Government technical experts conducted market research from March to July 2020 to ascertain the ability of any other sources besides Red Hat, a reseller or a different OS, to provide the licensing, associated software maintenance and technical support. Market research consisted of conducting web-based research including reviewing technical descriptions and product sheets of similar brands including CentOS, Amazon Linux, Debian, FreeBSD, SusE, and Ubuntu as well as other approved OSs such as Windows and Solaris Unix including software maintenance and technical support associated with each. Specifically, Government technical experts reviewed Enterprise Linux vendor websites for specifications on Red Hat Enterprise Linux, Oracle Linux and Software and Systems Development Linux. No other software products were found to be interoperable and compatible with the existing Red Hat software. As described in Section 5 of this document, VA’s Linux infrastructure is designed with an underlying proprietary OS from Red Hat and will only integrate with the same proprietary OS. No other manufacturer has the capability of providing the signed Red Hat source code and binary software subscriptions. Each product distribution has its own calibrations and many of the configuration files are in different places and in a different format. Use of software that is not interoperable or compatible with the existing Red Hat software would lead to system outages to the detriment of patient care and additional software costs associated with recoding 200 programs in use that are not certified on Red Hat software. An alternative Enterprise Linux OS software solution would also require proprietary software maintenance and technical support which would not resolve the inability to obtain signed Red Hat source code and binary software maintenance and technical support from vendors other than RedHat or an authorized reseller through the respective software providers.

Additionally, these vendors cannot provide the required Red Hat published list of support application program interfaces for major and minor releases, nor signed Red Hat source code and binary that is required to provide the required software maintenance and technical support for the Red Hat software subscriptions because these vendors cannot access the source code and technical data of VA’s current proprietary signed Red Hat source code and binary software applications. Each software developer offers a proprietary system and associated services that will only operate with its respective hardware and software. Based on the market research, only signed Red Hat source code and binary software maintenance and technical support subscriptions can meet VA’s technical requirements due to security requirements as described in Section 5 of this document.

In addition, on July 20, 2020, VA was contacted via email by SuSE who expressed concerns with VA’s Justification for An Exception to Fair Opportunity associated with VA’s RedHat procurement from Fiscal Year 2019. SuSE stated that it could provide commercial support for Red Hat software maintenance consisting of version upgrades, patches, fixes, updates, and security resolutions. VA’s technical experts reviewed SuSE’s product offerings and advised the Contracting Officer (CO) that SuSE products did not meet VA’s requirements. As a result, the CO informed SuSE of this on July 27, 2020. The CO followed up with SuSE on July 29, 2020, to ensure SuSE understood that the Justification for An Exception to Fair Opportunity they continued to reference in their communications were related to an award from FY19. Since SuSE’s communications claimed that the justification contained factually incorrect and erroneous statements, the CO requested further clarification from SuSE. SuSE provided a response on August 4, 2020, with clarifications. The Government technical experts again reviewed SuSE’s claims and again determined that only RedHat software subscriptions and associated maintenance and technical support will meet VA’s requirements based on the rationale provided in Section 5 of this document. On August 19, 2020, the CO provided a response to SuSE. SuSE responded to the CO on August 24, 2020 and stated that they are confident SuSE can provide a solution that will meet VA’s needs. However, SuSE’s response does not address the extensive re-coding effort that would occur and is predicated on running the 200 programs identified in Section 5 of this document on SuSE Linux Enterprise Server and not Red Hat. This does not meet VA’s requirements due to recoding and migration efforts which are further detailed in Section 5 of this document.

Further, a Request for Information (RFI) was issued on May 22, 2020, via the NASA SEWP V website to ascertain if there were sources that could meet the Government’s technical requirements for brand name RHEL products and the associated software maintenance and technical support. The RFI closed on May 26, 2020 with 12 responses received. Four of the 12 contract holders who responded to the RFI are Service-Disabled Veteran-Owned Small Businesses (SDVOSBs). Based on the information provided, the Government Program Manager determined three vendors (BahFed Corp, Fedstore Corporation and Three Wire Systems, LLC) to be technically capable of performing the effort. Two of the vendors found capable of meeting the requirements are SDVOSBs (Fedstore Corporation and Three Wire Systems, LLC).

Additional market research was conducted in July 2020 utilizing the NASA SEWP V GWAC Provider Lookup Tool to identify NASA SEWP resellers that could provide the required software maintenance and technical support subscriptions. The search yielded 23 Red Hat resellers capable of providing this requirement that are contract holders in Group B (2) SDVOSBs and registered and verified in the Vendor Information Pages (VIP) database; therefore, limited competition is expected among SDVOSBs for this proposed action. Furthermore, market research was conducted on the VIP database which found that two or more VIP verified vendors are capable of meeting the Government’s requirements under North American Industry Classification System code 541519.

9. Other Facts: None.

10. Technical and Requirements Certification: I certify that the supporting data under my cognizance, which are included in this justification, are accurate and complete to the best of my knowledge and belief.

Date: ______________________

Manager, Linux/Unix Signature: ______________________ Service Line

11. Fair and Reasonable Cost Determination: I hereby determine that the anticipated price to the Government for this contract action will be fair and reasonable based on anticipated limited competition. Contracts awarded on NASA SEWP V contract have already been determined to be fair and reasonable. As there are multiple resellers on the NASA SEWP V GWAC capable providing brand name brand name RedHat software subscriptions which include renewal of software and associated maintenance and technical support described herein, price competition is anticipated. Finally, if necessary, the successful quote will be compared with the Independent Government Cost Estimate.

Date: ______________________

Procuring Contracting Officer Signature: ______________________

12. Procuring Contracting Officer Certification: I certify that this justification is accurate and complete to the best of my knowledge and belief.

Date: ______________________

Procuring Contracting Officer Signature: ______________________

13. Legal Sufficiency Certification: I have reviewed this justification and find it adequate to support an exception to fair opportunity and deem it legally sufficient.

Date: ________________________________

Legal Counsel Signature: _____________________________

Approval

In my role as Head of the Contracting Activity, based on the foregoing justification, I hereby approve the acquisition for the renewal of existing enterprise-wide brand name Red Hat software subscriptions which include renewal of software and associated maintenance and technical support on an other than fair opportunity basis pursuant to the authority cited in Section 4 above, subject to availability of funds, and provided that the property and services herein described have otherwise been authorized for acquisition.

Date: __________ Signature: ___________________________________________

Head of the Contracting Activity Technology Acquisition Center Office of Procurement, Acquisition, and Logistics