BIP Platform Confluent JA to post.pdf

Enterprise Confluent subscription node licenses and associated maintenance support Control Number: VA-21-00032257

JUSTIFICATION FOR AN EXCEPTION TO FAIR

OPPORTUNITY

1. Contracting Activity: Department of Veterans Affairs (VA)

Office of Procurement Acquisition and Logistics

Technology Acquisition Center 23 Christopher Way Eatontown, NJ 07724

2. Description of Action: The proposed action is for a firm-fixed-price delivery order issued under the National Aeronautics and Space Administration (NASA) Solutions for Enterprise-Wide Procurement (SEWP) V Governmentwide Acquisition Contract (GWAC), for brand name Enterprise Confluent subscription node licenses and associated maintenance support as well as four optional tasks that can be leveraged to increase the quantity of Confluent Enterprise subscription licenses, enable additional features for each licensed node, add additional connectors and solution acceleration support.

3. Description of the Supplies or Services: VA, Office of Information and Technology, Enterprise Project Management Division, Benefits, Appeals and Memorials (BAM) Portfolio has a requirement for brand name Enterprise Confluent subscription node licenses and associated maintenance support. Specifically, BAM requires 15 brand name Gold Level Confluent Enterprise subscription node licenses for the production environments and a quantity of 15 brand name Gold Level Confluent Enterprise subscription node licenses for Pre-Production environments on the Benefits Integration

Platform (BIP) as well as one Confluent Oracle Change Data Capture (CDC) Source Connector license for Production and one Confluent Oracle CDC Source Connector License for Non-Production environments.

The BAM portfolio requires a data-streaming service to support the modernization of its services that will result in more accurate and faster delivery of benefits to Veterans.

This solution will enable VA services by reducing reliance on legacy applications, improve the timelines and quality of claims processing, offer solutions that are reusable and accessible to other product lines to reduce rework done at VA, and modernize VA software architectures through the adoption of real-time data streaming, eventing and topic-based technologies.

The brand-name Enterprise Confluent subscription node licenses and associated maintenance support is the only product that meets all of VA’s and BIP Platform requirements. Specifically, only the brand-name Enterprise Confluent subscription node licenses and associated maintenance support can meet the Government requirement to provide an out of the box solution that provides enterprise support, data source integration, multi-tenanted security, and data retention. VA requires 24x7x365 enterprise support for a data-streaming and eventing solution, as adoption is anticipated to rapidly scale to an enterprise level. This functionality is already slated for adoption by mission critical applications such as Veterans Benefits Management System.

Enterprise support ensures that if anything goes wrong with the data-streaming solution such as abnormal behavior, data loss, or an outage occurs that impacts mission critical applications, dedicated vendor support is available to quickly resolve the issue. This support and resulting accelerated recovery times will reduce the negative impact to productivity for VA employees and enable timely and continuous delivery of services and benefits to our Veterans. Additionally, VA requires the data source integration requirements in order to provide the ability for BAM services/applications to consume real-time data from disparate data sources. This solution must have the ability to integrate with and perform CDC on Oracle databases such as the Corporate Database (CRP) and other VA data sources including Microsoft and Cloud Service Provider managed databases as well as non-database sources. Typically, the data comes in the form of change sets to data or characteristics about the data is updated and other systems require notifications of the change and what the change is. VA BAM portfolio is required to provide a continuous flow of data from various sources into a single channel for other applications to consume. The requirement to use pre-built data source connectors will allow for the aggregation of data from disparate data sources into a single location (a topic) based on service/application needs. For database connectors to perform properly without negatively effecting the database, the code used to create the connector is specific to the version of the database and database schema, and it requires pre-built, lifecycle-supported connectors.

The multi-tenanted security is required as the BIP Platform holds a Federal Information Security Management Act High authority to operate (ATO) categorization and is required to meet the appliable policies to High systems outlined in the VA 6500 handbook, to include requirements around access control, encryption (Federal Information Processing Standard (FIPS) 140-2 compliance) and auditing. The BIP Platform architecture standards also require that multi-tenant tools integrate with the existing Identity Manager (IdM) access management solution and has the ability to manage external consumer access. This capability will allow for access control to be applied to streams/topics and their contained data to particular users, groups or services both internal to the BIP Platform as well as for external consumers.

Compliance with the VA 6500 aims to ensure that Veteran and other VA related data, including Personal Identifiable Information and Personal Health Information is properly safeguarded and only accessed when necessary in accordance with VA missions and business functions.

VA also requires infinite data retention. One of the benefits of a data-streaming and eventing is that the data is an immutable point in time artifact which some applications and business processes require for functionality. The BIP Platform requires support of infinite data retention on data streams so that a historical view of the data is available to understand what has changed and triggers business logic based on that change in support of application use cases. This also triggers a BIP architecture compliance for use of tiered storage to ensure a cost-efficient means of storing infinite data. Having the data moved to less expensive storage within the application facilitates the effectiveness of the solution and adoptability for other products and projects.

The Period of Performance for this licensing shall be 12 months, with two 12-month option periods, and four optional tasks. Optional Task 1 is for up to 90 Kafka Streams which supports the continuous processing of real time data directly as it is produced or received. This task can be exercised at the Government’s discretion in the amount of the Confluent Gold Licenses at the time of award of this optional task. Optional task 3 is for up to 45 additional Confluent software subscription licenses as demands of the solution scale. Optional Task 2 in a quantity up to three, for the Oracle CDC Connector for adding connections to additional databases and Optional Task 4 is for Oracle CDC Deployment and Configuration Accelerator Engagement in support of Optional Task 3 awards and will consist of a team of Confluent Experts to help accelerate the streaming platform adoption and achieve successful business outcomes, particularly as related to the installation, configuration, and testing of the Oracle CDC connector(s). This engagement will be three weeks, four days per week. This task will include a Resident Field Engineer to work alongside VA team in apps, streams, and connector development to ensure success and an Engagement Manager to unblock advisory and assurance activity impediments, contributes implementation and planning expertise to customer management. The total estimated value of the proposed action is (inclusive of options and optional tasks).

4. Statutory Authority: Pursuant to Federal Acquisition Regulation (FAR) Subpart 16.505(a)(4), the following brand name requirement is being justified and approved as an exception to fair opportunity under the statutory authority of Section 41 U.S.C.

4106(c)(2) as implemented by FAR Subpart 16.505(b)(2)(i)(B), entitled “Only one awardee is capable of providing the supplies or services required at the level of quality required because the supplies or services ordered are unique or highly specialized”.

5. Rationale Supporting Use of Authority Cited Above: Based on extensive market research as described in Section 8 of this document, it was determined that limited competition is viable among resellers of the brand name Enterprise Confluent subscription node licenses and associated maintenance support. Use of the brand name Enterprise Confluent subscription node licenses and associated maintenance support with the full enterprise feature set described in section three becomes necessary to provide a data streaming capability to existing BIP Platform (multi-tenanted Kubernetes platform) hosted services to enable BAM and Veteran Experience

Services Portfolio services to modernize their architectures, while keeping security and performance at the forefront of the solution.

Based on the Market Research in Section 8, only Enterprise Confluent subscription node licenses and associated maintenance support meets all of the Government’s requirements listed in Section 3 and will help enable VA to achieve a goal of reducing the burden on development teams to create custom solutions and will result in common adoption and patterns making it easier for operations to manage an enterprise data streaming solution. The

Enterprise Confluent subscription node licenses and associated maintenance support provides the 24x7x365 enterprise support. It also supports infinite data retention on data streams, as required. Additionally, no other brand name product other than Enterprise Confluent subscription node licenses and associated maintenance meets all of the multi-tenanted security requirements to include providing VA with the ability to manage external consumer access for the data streams and eventing solution, integration with the BIP Platform’s IdM solution to manage access to the Veteran data in the data streams and events, and FIPS 140- 2 compliance. If the requirements for multi-tenanted security are not met, it would put the BIP

Platform ATO and all hosted services at risk for non-compliance and put the BIP Platform and hosted applications at risk to be shut down by VA.

Further, the Enterprise Confluent subscription node licenses and associated maintenance support is the only brand name software that meets all of the data source integration requirements. VA requires an Oracle CDC, without which negative impacts would occur to the source database. Other Oracle connectors provide a single connection and would use a process of polling or scanning the database for and changes to the data which would degrade the data source to the point where it is unusable by any project. Additionally, for VA to build its own CDC connector, it would take multiple teams, months of work to create with all of the features necessary for the solution and then would need to maintain the connector to stay compatible with each version upgrade or schema change of Oracle. Enterprise Confluent subscription node licenses and associated maintenance support offers a pre-built, lifecycle supported Oracle CDC connector which will enable VA products/projects to consume aggregated

Veteran data from disparate sources without negatively impacting mission critical databases such as CRP, which utilizes the Oracle database software. By also offering a variety of out of the box, lifecycle supported and maintained, integration solutions, the Confluent connectors becomes even more value added and cost effective by licensing the cluster where Confluent is installed instead licensing every connected database.

Without access to the Confluent Oracle CDC connector, VA will need to spend additional time, money and resources on development of the initial data source integration with CRP which could result in taking up as much as multiple scrum teams an entire program increment of effort, resulting in potentially $500K+ in duplicative resource costs and delay the delivery of direct Veteran benefits delivered from solutions planning to leverage this data streaming and eventing solution and does not account for the long-term versioning and maintenance costs associated with producing an in house

Oracle connector.

Finally, the aforementioned support for Oracle CDC Deployment and Configuration as part of the Enterprise Confluent solution can only be performed through Confluent or its authorized resellers as the access to the proprietary data and source code necessary to provide support is required. Failure to procure the brand name support would hamper VA’s ability to maintain the operational status of the software. In summary, if the Enterprise Confluent subscription node licenses and associated maintenance support are not procured, it will immediately prevent the BIP Platform from being able to offer a data streaming solution to the BAM Portfolio. This will result in numerous downstream negative impacts on the delivery schedules and architectural improvements of Veterans Benefits Administration services such as Data Sync (NWQ), VA Profile and Records

Resource Center aimed to increase claim accuracy, reduce claim cycle time through the ability to leverage real-time data to inform decisions regarding enrollment, eligibility, patient care and billing which will have direct and negative impact on Veterans and their families.

6. Efforts to Obtain Competition: Market research was conducted, details of which are in section 8 of this justification. This effort did not yield any additional sources that can meet the Government's requirements. However, the undersigned VA technical representative determined that limited competition is available amongst multiple resellers of the brand name Confluent software subscription licenses and associated maintenance support. In accordance with FAR 16.505(a)(4)(iii)(A)(1), this justification will be provided with the solicitation to all appropriate NASA SEWP V GWAC holders.

Furthermore, in accordance with FAR 5.301, 16.505(b)(2)(ii)(D), and 16.505(a)(4)(iii), the award notice for this action will be synopsized on the Contracting Opportunities website and this justification will be made publicly available within 14 days of award.

7. Actions to Increase Competition: The Government will continue to conduct market research to ascertain if there are changes in the marketplace that would enable future actions to be competed.

8. Market Research: The Government’s technical experts conducted market research from August through December of 2020 to identify a software solution available in the marketplace to meet the Government’s requirements. The Government’s technical experts reviewed software products including Confluent Community Edition, Confluent

Enterprise, IBM MQ, Rabbit MQ, Apache Kafka, Amazon Kinesis, Amazon Simple Queue Service (SQS), Amazon Managed Streaming for Apache Kafka (MSK), Azure Streams analytics and Google Cloud Dataflow.

During the Government’s market research of software options available, Azure Streams Analytics and Google Cloud Dataflow were immediately eliminated from consideration as they are solutions based on the named cloud provider (Azure and Google) while the BIP Platform is hosted in VA’s Enterprise Cloud (VAEC) Amazon Web Services (AWS)

GovCloud. Amazon Managed Streaming for Apache Kafka (MSK) was not further pursued because it is not Federal Risk and Authorization Management Program High approved thus it is not available in the VAEC AWS GovCloud.

It was determined from the Government’s market research that RabbitMQ, IBM MQ and Amazon SQS are unable to meet the Government’s requirement because they are classic messaging queues that do not fulfill the complex requirements that BAM services and applications seek from an enterprise data streaming and eventing solution.

Further market research was conducted on Apache Kafka, Community Edition Confluent, Amazon Kinesis, and Confluent Enterprise. As a result of the market research, it was discovered that these products only met some and not all of the requirements of the VA which is explained in greater detail below.

Apache Kafka is an open-source distributed event streaming platform. Apache Kafka is an open-source project and does not meet the Government’s requirement for 24x7x365 enterprise support. Apache Kafka also does not meet the Government’s requirement for data source integration. Specifically, Apache Kafka is an open-source project and all connectors built for the Kafka product are community driven for open-source data sources. There is no open-source or community-built connector for Oracle CDC, which is a requirement for the solution so as to not negatively impact the source database.

Other Oracle connectors provide a single connection and would use a process of polling or scanning the database for and changes to the data which would degrade the data source to the point where it is unusable by any project. For VA to build their own CDC connector, it would take multiple teams, months of work to create with all of the features necessary for the solution and then would need to maintain the connector to stay compatible with each version upgrade or schema change of Oracle. Oracle does make a connector for Kafka but the license is done at the data source level and not at the cluster which means for every Oracle instance the solution needs to connect to, VA would need to purchase and maintain an Oracle CDC license. These community driven connectors pose a risk to the overall solution as they are not required to keep pace with the upgrades and changes of data sources, nor do they meet the required level of support and released version parity required for enterprise-based support. Apache Kafka does not meet the Government’s requirement for multi-tenanted security. Open-source Apache Kafka is able to achieve the access control requirements through the use of Access Control Lists (ACL) for the application and for the data streams but would take extensive custom code and solutioning to achieve the requirements. Additionally, it will consume more development time as every new user, application, etc. would require a new ACL per topic, drastically increasing the complexity to manage access as adoption of the solution scales. Mapping ACLs to the BIP Platform’s IdM solution for access is also possible but would require additional development times and long-term maintenance to meet the architecture requirement. Apache Kafka is able to support the 140-2 FIPS encryption requirement but is not able to enforce encryption. This setting must be configured on a stream-by-stream basis and there is no one policy or configuration to control the setting for all data and streams, presenting a potential risk for non-compliance. Open-source Apache Kafka is able to meet the requirements for data retention and for long term storage and utilizing multiple tiered storage to save costs long term. In summation, Apache Kafka is not capable of meeting the enterprise support, pre-built Oracle database connector needs, connector lifecycle support, enterprise support for data source integration or the FIPS 140-2 requirements of the software solution. Other limitations of the software include concerns about the ability to meet disaster recovery requirements and the increased need for custom developed solutions and high level of complexity for managing the overall solution for a multi-tenant platform.

The Government’s market research of Confluent Community Edition did not meet the

Government’s requirement for enterprise support. The Confluent Community Edition does not come with or offer 24x7x365 enterprise support and therefore does not meet the Government’s requirements. Confluent Community Edition does not meet the Government’s requirement for integration. Although Confluent Community Edition has the ability to use the community supported data connectors, it does not meet the requirements for enterprise support for the data connectors. Confluent Community Edition also does not include access to the Oracle CDC connector and therefore does not meet the requirements for Oracle CDC support. Confluent Community Edition does not meet the Government’s requirements for multi-tenanted security because it does not have an out-of-the-box access management integration into IdM but mapping between ACLs and IdM could be achieved through a custom developed solution. Confluent Community Edition is also not FIPS 140-2 compliant as required by VA6500. Confluent

Community Edition is not configured to support infinite data retention or tiered storage but through custom developed solutions has the ability to meet the Government’s requirements by making configuration changes and pairing with a custom solution paired for using local and remote long-term, tiered storage. In summation, Confluent

Community Edition is not capable of meeting the enterprise support, pre-built Oracle

Database connector solution nor FIPS 140-2 requirements of the software solution. In addition, there is a lack of high-availability options for the community edition version around throughput options, partitions and tasks.

Amazon Kinesis offers key capabilities to cost-effectively process streaming data at any scale, along with the flexibility to choose the tools that best suit the requirements of an application. Amazon Kinesis is an AWS provided product and has a shared support model. Support requests would be processed through the normal VAEC AWS model of opening tickets and getting support. However, Kinesis support would only be available for the limited scope of its messages. Similar to the open-source Kafka solution, Amazon Kinesis does have community driven data connectors but does not meet the

Government’s data source integration requirement as it does not have an Oracle CDC connector required to support the offering of a data streaming and eventing solution while not negatively effecting the data source. Custom development for an Oracle CDC connector for AWS Kinesis would result in similar timelines as developing a custom

Oracle CDC connector for Apache Kafka. Amazon Kinesis can meet some but not all of the Government’s requirements for multi-tenanted security. Amazon Kinesis has the VA6500 access control requirements and through custom workarounds can meet the BIP Architecture requirement to integrate with IdM but is unable to meet the

Government’s requirement to manage external users/consumers outside of the AWS accounts due to current VA policies. Server-side encryption is accomplished through additional configurations making the total solution more complicated. Additional development time, AWS services and costs are necessary for the data to be encrypted in flight and to enable FIPS 140-2 for the service and data. AWS Kinesis does not meet the Government’s data retention requirements. Kinesis has a maximum data retention period of 365 days with an extra charge for data that is retained past one day.

Additionally, the charge is increased for data retained past seven days and there is another additional charge to retrieve data that has been retained past seven days. This limitation on data retention and additional costs for retaining data does not meet the Government’s requirement for data retention. In summation, Amazon Kinesis is not capable of meeting a pre-built Oracle database connector solution, connector lifecycle support, infinite storage, nor management of external consumer access requirements of the software solution. Other concerns include issues with the performance of the service which experienced latency even with testing data and other published documentation around quotas and limitations.

Based on the Government’s market research, Enterprise Confluent subscription node licenses and associated maintenance support is the only product that meets the Government’s requirements of enterprise support, data source integration, multi-tenanted security, and data retention. Enterprise Confluent subscription node licenses and associated maintenance support meets the Government’s requirement for enterprise support and is able to meet this requirement by offering 24x7x365 dedicated support for both non-production and production deployments through a single point of contact for the operations team to use in the need of a critical issue. This support can also be used to review deployment strategies, assist with improving performance and with troubleshooting non-outage needs. Enterprise Confluent subscription node licenses and associated maintenance support meets the Government’s requirements for data source integration because it builds on the community edition version with an https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html additional library of premiere lifecycle supported data connectors. Enterprise Confluent subscription node licenses and associated maintenance support is the only product to have a supported Oracle CDC connector that will be upgraded and managed with the different versions of Confluent Enterprise and Oracle over time. Enterprise Confluent subscription node licenses and associated maintenance support is the only product that meets all of the multi-tenanted security requirements of supporting FIPS 140-2 encryption protecting VA owned Veteran data, auditing, fine grained access control through Role Based Access Control permissions for using the application and for consuming the data streams, integration with the existing Identity Access Management solution. Enterprise Confluent subscription node licenses and associated maintenance support meets the Government’s requirement for data retention as it supports infinite retention and in application options to support multi-tier data storage to reduces costs over time and without the need for custom development or additional services to complicate the solution. Additional Enterprise Confluent licenses can be provisioned through exercising optional tasks that will allow the solution to scale as needed.

Optional tasks will also allow for the VA to extend the features support for each licensed node, add licensing for additional data integration connectors and receive support for accelerated solution delivery if needed, ensuring that the VA is able to meet the requirements of tenants and consumers.

Additional market research was conducted by Government representatives in February 2021 utilizing the NASA SEWP V Provider Lookup Tool. The search identified multiple Service-Disabled Veteran-Owned Small Business (SDVOSB)

Confluent resellers in Group B(2) on NASA SEWP V GWAC. Therefore, based on the results of the market research, there is reasonable expectation that limited competition exists among verified SDVOSB contract holders in Group B(2) on the

NASA SEWP GWAC.

9. Other Facts: None.