JA - Datadog_FINAL_Redacted.pdf

Control Number: VA-24-00087440

JUSTIFICATION

FOR AN EXCEPTION TO FAIR OPPORTUNITY

1. Contracting Activity: Department of Veterans Affairs (VA)

Office of Procurement, Acquisition, and Logistics Technology Acquisition Center 23 Christopher Way Eatontown, NJ 07724

2. Description of Action: The proposed action is for a firm-fixed-price order to be issued under the National Aeronautics and Space Administration (NASA) Solutions for Enterprise-Wide Procurement (SEWP) V Governmentwide Acquisition Contract (GWAC) for Datadog application performance monitoring (APM) tool subscription.

3. Description of the Supplies or Services: VA, Office of the Chief Technology Officer (OCTO), Architecture and Engineering Service (AES) has a requirement to renew subscription for its current APM tool, Datadog, that supports VA.gov, the Lighthouse Application Programming Interface (API) platform, Vetext, Appeals, VA notify, Loan Guarantee, Enterprise Precision Scanning and Indexing (EPSI), Standardized Episodes of Care (SEOC), Consult Toolbox (CTB), and Data and Analytical Support for Healthcare (DASH). VA requires a tool for monitoring its performance and security that interoperates with Ruby on Rails, the coding language VA.gov is built in. This is to monitoring platform is able to fully understand Ruby-on-Rails traces and provide the necessary level of insight, down to the individual endpoint level, of its security and performance to ensure its operational for Veterans to manage their well-earned care and benefits. Datadog integrates natively with Ruby on Rails providing in-depth information about errors that occur in each of these systems, down to the individual endpoint level. Monitoring down to the endpoint level is necessary to indicate where in the source code the system is encountering errors, as well as how that error may relate to other services (such as your database(s) or caching service(s)). This information is required to ensure the applications are working as designed, securely, and if not, provide notification in near real time to VA staff or contractors who can address and remediate the issue(s). Datadog subscription includes access to twenty functions within the tool including, but not limited to, metrics, archives, indexing, logging, auditing, and database monitoring. Lastly, VA requires Datadog to provide two weekly monitoring meetings, one with VA technical staff on the health and usage of the tool within the Agency, and another with budget and finance (as well as representatives from each technical team) to monitor usage and provide support to VA to ensure the tool is being used efficiently and within budgets. The period of performance for this effort is one, 12-month Base Period, with one 12-month Option Period, and Optional Tasks for additional quantities of Datadog monitoring support, additional infrastructure, monitoring, indexes, logging, and auditing for anticipated growth in the use of the tools listed above, for example as a result of the expansion of Veteran benefits under the Sergeant First Class Heath Robinson Honoring our Promise to Address Comprehensive Toxics Act of 2022 (PACT Act). The total estimated value of the proposed action is

, inclusive of all options and optional tasks.

Datadog Control Number: VA-24-00087440

4. Statutory Authority: Pursuant to Federal Acquisition Regulation (FAR) 16.505(a)(4), the following brand name requirement is being justified and approved as an exception to fair opportunity under the statutory authority of Section 41 U.S.C. 4106(c)(2) as providing the supplies or services required at the level of quality required because the supplies or services ordered are unique or highly specialized.

5. Rationale Supporting Use of Authority Cited Above: The Datatog APM solution provides the Office of Information Technology (OIT) in-depth information about errors that occur in the VA systems listed in paragraph 3. Datadog shows specifically where in the source code the system is encountering errors, and how that error may relate to other services (such as other database(s) or caching service(s)). It provides a much higher level of detail about specific cases (server-side errors) than standard data sources (logging and metrics) which is the common default option.

configuration provides detailed and real time performance information down to the individual endpoint level and has been integral to several technical issues encountered on VA.gov over the past year, specifically the Code Yellow effort.

Datadog is also integrated with the new VA.gov Watchtower, which provides VA Leadership real time information on the performance and availability of VA.gov, allowing for resolution of issues in near real time. If Datadog was not available, 472 current performance dashboards with 1,015 monitors and 402 synthetic tests that rely on it would fail, and VA would need to quickly develop an alternative way to collect the same information, which would divert a significant level of development resources to replace it at a substantial cost and delay. This is because the dashboards and monitors are can provide continued access or use of those dashboards other than Datadog. Any near term replacement would provide significantly of VA.gov, requiring a lot of manual work arounds and making performance or security concerns very challenging to resolve and understand.

Given how integrated Datadog is into the applications it supports, replacing it would require up to two years effort, at a total estimated cost for all of the applications of approximately $6,500,000.00. Additionally, to ensure continued monitoring of the aforementioned systems while a new solution was integrated, both systems would still need to be maintained creating a duplicated cost of $9,000,000.00 over the two years.

Specifically, a shift to another tool would require: 1) a gap analysis for features that aren't available in the new solution to ensure no degradation of support; 2) develop a way to send existing sources of information (logs, events from sources like Sentry, Amazon Web Service metrics, etc.) to the new solution; 3) test the new instrumenting APM on the current vets-downtime; 4) ensure the new solution supports Ruby on Rails, including all current integrations with VA.gov, and ensure it is configured to provide the same level of visibility as the current Datadog solution; 5) migrate existing user access levels to a new platform, then develop a process to add existing users to the new platform; 6) update the existing Watch Officer system (the team that staffs the watchtower mentioned above) to accept data from the new system; 7) update the current periodic processes for operations based on S3 log in the VA Enterprise cloud, access to use the a new platform; 8) determine a way to replace existing Service implementation in vets-api with the new solution, then update code for the vets-api applications to support it; 9) Update the vets-website source code to support the new real user monitoring (RUM) platform; 10) trace the current API access requirements so equivalents can be configured on the new platform; document the current Shared reports (accessible via email link for VA business partners) and determine the equivalent approach for the new platform. Additionally, since the new system would take at least two years to fully implement, the current Datadog solution will still need to remain operational until a full cutover is completed. It is essential VA maintain full visibility into the performance of its applications, specifically those that are Veteran facing like VA.gov, to ensure when any degradation of functionality is encountered VA can quickly find the cause of the incident, triage it, and resolve it as quickly as possible. Given VA.gov is the main digital pathway for Veterans to access and manage their health and benefits, any degradation of support reduces their access to these well-earned benefits and healthcare. Thus, in addition to the work to configure a new solution, existing staff would need to continue to maintain the current application, ensure collaboration between the technical and business team members that consume the information from the new monitoring solution, and coordinate with the current technical and business team members to help migrate functionality to a new platform, for the duration of a migration project. This would require them to focus almost exclusively on this transition, at the expense of any other projects that they would otherwise work on during the period.

Lastly, VA currently has two approved application monitoring solutions, Datadog and Dynatrace. It is a best practice in system monitoring to maintain as few separate solutions as possible to ensure visibility of the status of all VA systems by VA leadership which is made significantly more difficult with many multiple monitoring tools. Datadog was chosen through a competitive effort, under the DevOps Tool Suite contract award, because it works natively with Ruby on Rails down to the individual end point level, experience and the performance of these websites. Datadog ensures the integrations between their APM tool and VA.gov are updated in real time any time a new version of Ruby on Rails is released. Without Datadog, VA would have to internally recreate the integration between VA.gov and the APM tool, and provide on-going support and updates, each time Ruby on Rails is updated, throughout the life of the product. Dynatrace was not meant to support Ruby on Rails natively and thus only has a stable implementation of one for the three sources of information VA needs to ensure access to specific details about errors and other minutiae that are part of Ruby on Rails and which are part of Veteran facing functions on the websites. These websites are Veteran facing and thus require constant monitoring to ensure the ability of Veterans to successfully access and manage their VA services.

requirements. If Datadog is not renewed, VA would lose necessary insight into the performance and security of the aforementioned applications, specifically VA.gov the

. Should a performance outage or security incident occur, VA would not be able to determine what part of the website was malfunctioning, how Veteran s access to that website was affected, or whether the incident was caused by a security failure or malicious actor.

(s) and resolve them.

6. Efforts to Obtain Competition: Market research was conducted, details of which are in Section 8 of this justification. This effort did not yield any additional sources that can meet the Government's requirements. However, the undersigned VA technical representative determined that limited competition is available among resellers for brand name Datadog solution. In accordance with FAR 16.505(a)(4)(iii)(A)(1), this justification will be provided with the solicitation to all appropriate NASA SEWP V GWAC holders. Furthermore, in accordance with FAR 5.301, 16.505(b)(2)(ii)(D), and 16.505(a)(4)(iii), the award notice for this action will be synopsized on the Contracting Opportunities website and this justification will be made publicly available within 14 days of award.

7. Actions to Increase Competition: The Government will continue to conduct market research to ascertain if there are changes in the marketplace that would enable future actions to be competed. Specifically, an analysis is currently underway to determine if the other VA approved monitoring tool, Dynatrace, can be updated to support the Ruby on Rails integrations with VA.gov, or augmented with external tools, which it currently does not do. The analysis is expected to take at least a year.

8. Market Research:

in July 2024 conducted an internet search of available APM solutions including Dynatrace and AppDyanmics. Neither solution was determined to meet the technical requirements listed in Section 3 as neither can natively support the Ruby integrations and provide monitoring down to the endpoint level within Ruby as required by VA.gov, as only Datadog currently provides. Further, OCTO met with Dynatrace in July 2024 to discuss how Dynatrace could be augmented, through either government only customizations and/or additional software tools, to provide the required level of integration but to date this is not something Dynatrace can provide.

The aforementioned internal VA review is looking for how VA internally could provide this capability, but this will take several years to confirm and then implement. During the recent conversations Dynatrace noted that they do not have many Ruby on Rail integration requests and thus it is doubtful they would be adding this to their commercial offering. Thus, none of the alternative solutions can provide access tary APM tool. Based on this analysis, the current Datadog solution is the only one that can meet the technical requirements for this effort as listed above.

Additional market research was conducted utilizing the NASA SEWP V Market Research Tool and it was determined that there are five Service-Disabled Veteran- Owned Small Business (SDVOSB) resellers of the brand name Datadog solution. The NASA SEWP V GWAC has a dynamic catalog where line items can be added based on customer requests as long as the required item or service is within scope of the NASA SEWP V GWAC. Based on the above, there is a reasonable expectation that limited competition exists on the NASA SEWP V GWAC. As a result of the aforementioned market research, VA intends to issue a Request for Quotation on the NASA SEWP V

GWAC.

9. Other Facts: None.

11. Fair and Reasonable Cost Determination: I hereby determine that the anticipated price to the Government for this contract action will be fair and reasonable. Price analysis shall be conducted by comparing the quote to the published list pricing for the required Datadog solution. Finally, if necessary, the successful quote will be compared to the Independent Government Cost Estimate.

12. Procuring Contracting Officer Certification: I certify that this justification is accurate and complete to the best of my knowledge and belief.

13. Office of General Counsel Review

I have reviewed this document and find it legally sufficient, based on the representations set forth in the documents and, if any, atta approval is an independently supported decision not wholly reliant on the legal sufficiency determination.

Approval

In my role as procuring activity Advocate for Competition, based on the foregoing justification, I hereby approve the acquisition of Datadog on an other than fair opportunity basis pursuant to the authority cited in Section 4 above, subject to availability of funds, and provided that the property and services herein described have otherwise been authorized for acquisition.