Data Streaming Processing and Analytics Platform_RFI.docx

U.S. DEPARTMENT OF STATE

BUREAU OF DIPLOMATIC TECHNOLOGY (DT)

OFFICE OF INFORMATION TECHNOLOGY ACQUISITIONS (ITA)

REQUEST FOR INFORMATION (RFI)

for

DATA STREAMING, PROCESSING, AND ANALYTICS PLATFORM

SECTION 1: RFI BACKGROUND AND OBJECTIVES

1.1 PURPOSE

This Request for Information (RFI) is issued as market research to determine industry’s capabilities to provide a proven commercial-off-the-shelf (COTS) data streaming, processing, and analytics solution in support of a range of data management, observability, and security requirements at the Department of State (State). The solution shall provide a commercially proven base platform to extract data from systems of generation, load data in native form into various systems of storage and analysis, and transform, where needed, data for efficient analytic queries. The solution shall also provide a data streaming platform to support these features and provide accessibility and reliability across the Department’s global footprint.

The U.S. Government (USG) does not commit to award a contract on the basis of this RFI or reimburse any costs associated with the preparation of responses. This RFI is issued solely for information and planning purposes and does not constitute a solicitation. All information received in response to this RFI that is marked “Proprietary” will be handled accordingly. Responses to the RFI will not be returned. In accordance with FAR 15.201(e), responses to this RFI are not offers and cannot be accepted by the USG to form a binding contract. Responses to this RFI will assist State in determining the potential level of interest, competition adequacy, and technical capabilities of commercial vendors to provide the required products and/or services. State does not guarantee any action beyond this RFI.

1.2 BACKGROUND

The Department’s mission is to advance the interests of the American people, their safety, and economic prosperity by leading America’s foreign policy through diplomacy, advocacy, and assistance. The mission of the Bureau of Diplomatic Technology (DT) is to rapidly and securely deliver, anytime and anywhere, the knowledge resources and IT services needed to support the over 300 sites and 275 missions it services worldwide.

DT is seeking input from industry regarding their capability to provide market-leading solutions capable of meeting the full breadth of functional requirements outlined in this RFI. State requires access to a platform that is compliant with current Federal security requirements, is easy to use, flexible, scalable, customizable, and configurable to meet specific State requirements as further detailed below.

1.3 RFI OVERVIEW

As further detailed in Section 2, State is seeking to identify capable and interested partners able to provide a solution able to meet the full range of functional capabilities outlined in this RFI. Viable partners shall have demonstrated experience deploying the solution and delivering the ongoing support required to maintain, expand, and upgrade the solution over time for an organization of similar size, global scope, and mission complexity as State.

Information requested in this RFI includes the following categories:

· Company Information (Section 1) - Identification of interested vendors, socio-economic status, and potential acquisition vehicles to inform State’s potential acquisition strategy options.

· Capability to meet the Functional Requirements (Section 2) - Description of vendor’s technical solution to fully meet State’s functional requirements.

· Past Experience (Section 3) - Past project example(s) demonstrating vendor’s experience implementing their proposed solution to address the needs of an organization of similar size, global scope, and mission complexity to State.

1.4 SUBMISSION INSTRUCTIONS:

All written responses should be submitted in Microsoft compatible formats (e.g., Word) or PDF.

Responses shall use 12-point font and 1-inch margins, including all text, tables, and graphics. Additional attachments, brochures, or other marketing materials are not requested and will not be considered as part of the response. Response page limit(s) by section are noted below.

· Section 1 - 1 page

· Section 2 - 20 pages

· Section 3 - 5 pages Please submit all materials to John Warner at WarnerJ1@state.gov no later than August 29, 2025 The Government will not accept questions after Monday, August 25 5:00PM ET. Please submit questions to RothsteinAT@state.gov and WarnerJ1@state.gov.

The Government will post the questions and responses on SAM.gov with sufficient time for vendors to complete their responses. Please be advised that all submissions become Government property and will not be returned.

UNDER NO CIRCUMSTANCES SHALL ANY RESPONDENT CONTACT ANY DEPARTMENT PERSONNEL REGARDING THIS NOTICE.

SECTION 2: REQUESTED INFORMATION

Respondents are asked to provide the information requested below per the instructions outlined in Section 1.4 above.

Section 1 - Company Information Please provide the following details regarding your organization:

1. Company name.

2. Company’s point of contact (POC) name.

3. Company’s POC phone number.

4. Company’s POC email address.

5. Company’s URL/web address.

6. Company’s Cage Code.

7. Company’s Unique Entity Identifier (UEI) Number.

8. Company federal socioeconomic classification(s), if any, based on North American Industry Classification (NAICS) Code that applies to the proposed solution.

9. List of the General Services Administration (GSA) schedule, Government wide acquisition contracts (GWACs), or other federal Indefinite Delivery, Indefinite Quantity (IDIQs) contracts your company holds that are accessible by the Department and where the full proposed solution can be procured, including corresponding schedule number or GWAC number.

Section 2 - Functional Requirements State’s functional requirements are outlined below. Respondents are asked to provide a structured response that meets the following format requirements:

a. Response shall be structured and annotated to clearly map the detailed description of the proposed solution’s capabilities to the numbered requirements captured in the sections below. Responses should clearly map to the information in State’s requirements below.

b. Response narrative content should provide sufficient details regarding the proposed solution’s specific technical capabilities to demonstrate that the solution is capable of fully satisfying State’s requirements.

2.1 Mandatory Qualifications

The following qualifications and capabilities are mandatory requirements for any potential solution addressing the needs outlined in this notice. Vendors should assess these requirements carefully and fully describe their ability to meet them.

Key requirements are as follows:

a. The solution shall be based on a commercially proven Commercial Off-The-Shelf (COTS) software platform (custom-built solutions are unacceptable) with demonstrated experience successfully deploying and operating in the federal environment.

i. State does not seek custom-developed solutions that have not yet been successfully deployed at other USG agencies similar in size and scope to State.

b. The solution shall be a single-Original Equipment Manufacturer (OEM), integrated platform with end-to-end solutions leveraging seamless integration across capabilities and tool sets.

c. The solution shall possess significant out-of-the-box functionality, while also supporting custom configuration within expedited timelines.

d. The solution shall offer comprehensive in-house professional and configuration services, supported by a cohesive and specialized team of experts, available for on-premises implementations.

e. The solution shall be capable of encrypting data, both at rest and in transit, via mechanisms compliant with Federal Information Processing Standard (FIPS) Publication 140-2. Please describe the extent to which the proposed solution is also compliant with FIPS 140-3 and any aspects of the solution that are currently not able to meet FIPS 140-3 requirements.

f. The solution shall comply with the Department's Cybersecurity Supply Chain Risk Management (C-SCRM) policy. The vendor must describe its ability to keep its software and product supply chains secure in accordance with National Institute of Standards and Technology (NIST) Special Publication (SP) 800-53.

g. The solution shall be able to route data to Databricks and low-cost object storage options such as Amazon S3, Azure Blob, and Google Cloud Storage, as well as low-cost file storage options. The data must be able to be written in a raw format, in a vendor-neutral non-proprietary format such as JavaScript Object Notation (JSON) or Parquet, and into a schema such as Open Cybersecurity Schema Framework (OCSF).

h. The solution shall support M-21-31 compliance by being capable of writing full fidelity copies of data to any cost-effective storage option of choice that can be recalled on demand for processing within the Department’s analytical systems.

i. The solution shall minimize data loss during outages, disconnected operations, maintenance windows, and backpressure events by providing destination persistent queueing (dPQ) that writes outbound events to disk whenever target systems are slow or offline.

j. The solution shall facilitate consistent and efficient data processing configuration by providing a repository for OEM-authored and community-built configurations complete with routes, pipelines, functions, sample data, and knowledge objects that can be built and tested in development, exported, and re-deployed across worker groups and agents throughout the Department.

k. The solution shall integrate with Okta and support authentication methods including Lightweight Directory Access Protocol (LDAP) integration, Personal Identification Verification/Common Access Card (PIV/CAC) Authentication, and Security Assertion Markup Language (SAML) 2.0 or OpenID Connect Single Sign-On (OIDC SSO).

l. The solution shall provide administrators with Role-Based Access Control (RBAC) to define and manage access-control roles, policies, and user permissions.

m. The solution shall provide a single, intuitive user interface (UI) to visualize and control the entire observability pipeline including data collection, routing, processing, deployment, configuration, and management as well as upgrades for worker and edge nodes without the need for extensive coding knowledge.

n. The solution shall provide a RESTful (Representational State Transfer) API (Application Programming Interface) for programmatic configurations, troubleshooting, and integrations.

o. The solution shall provide seamless integration with existing Continuous Integration/Continuous Delivery (CI/CD) workflows and simple rollbacks for configuration changes. Whether an administrator commits changes via the UI or API, leader nodes must notify worker and edge nodes to pull new configurations via an internal Git repository. The solution must also be compatible with external Git repositories.

p. The solution shall provide a previewing tool that shows an instant, side-by-side view of “In” vs “Out” data and performs pipeline diagnostics as an administrator modifies functions, so that parsing and transformation of events can be verified prior to deploying configurations.

q. The solution shall support push senders including but not limited to Syslog, Splunk Hypertext Transfer Protocol (HTTP) Event Collector (HEC) and Transmission Control Protocol (TCP), Amazon Data Firehose, HTTP/S (HTTP Secure) Bulk API and Raw HTTP/S , OpenTelemetry, Prometheus Remote Write, Loki, Grafana, Simple Network Management Protocol (SNMP) Trap, NetFlow/Internet Protocol Flow Information Export (IPFIX), Zscaler Nanolog Streaming Service (NSS), and raw TCP/User Datagram Protocol (UDP). Please describe the solution’s ability to support these and other push senders.

r. The solution shall pull data from multiple sources including but not limited to Amazon Kinesis Data Streams, Amazon SQS, Amazon S3, Amazon Security Lake, Google Cloud Pub/Sub, Azure Event Hubs, Azure Blob Storage, Confluent Cloud, CrowdStrike FDR, Office 365 Services & Activity, Prometheus Scraper, Kafka, Amazon MSK, Splunk Search, and Wiz. Please describe the solution’s ability to pull data from these and other data sources.

2.2 Ingestion

State requires a universal collector of IT, security, and observability data to natively integrate with Department data sources, modernize ingestion pipelines, and replace legacy systems. Key capabilities include:

a. The solution shall natively receive data from endpoint logging agents including Splunk Universal Forwarder (UF), Elastic Beats/Agent, and Fluent Bit, and must not be limited solely to endpoint agents within its own ecosystem. Please provide details on the breadth and depth of the solution’s ability to natively receive data from various end point logging agents.

b. The solution shall natively receive WEF (Windows Event Forwarder) events and serve as a WEC (Windows Event Collector) instance negating the need to install logging agents.

c. The solution shall support multiple collector-based sources including Amazon S3, Azure Blob Storage, Database, File System/Network File System (NFS), Google Cloud Storage, REST/API Endpoint, Script (shell scripts), and Splunk Search eliminating the need for custom scripts.

d. The solution shall be able to monitor local log files and delete them after ingestion. Application logs written to a filesystem must be easily captured and viewable through a live directory tree for an administrator to browse. Administrators must be able to select any application log within the directory and instantly create a pre-populated file monitor source for streaming eliminating any need for manual path edits.

2.3 Data Processing

State requires the solution to ensure only high-value data is sent downstream by optimizing data volumes through reduction and compression techniques, enriching data in the stream before routing to any destination, and enhancing security while data is in transit. Key capabilities include:

a. The solution shall have built in functions, including advanced tools for parsing and serializing data, to transform data formats before data is delivered to downstream destinations. The functions must support converting events among JSON, Comma Separated Values/Tab Separated Values (CSV/TSV), Key-Value, and Common Event Format (CEF).

b. The solution shall be able to mask and redact data at ingest.

c. The solution shall reduce noise and duplicative information by suppressing similar or identical events and removing null or empty fields and unnecessary fields.

d. The solution shall be able to flatten data. Fields must be able to be pulled out of a nested structure and moved to a higher level in an object. Customization must include the fields to flatten, the depth of the flattening, and the format of the resulting field names.

e. The solution shall support a built-in schema validation that checks events against stored JSON schemas.

f. The solution shall convert logs into compact metrics by calculating sums, counts, and averages on streaming data.

g. The solution shall decrease the amount of high-volume, low-value data being processed by filtering out events based on specified criteria and rates. Rates must be automatically adjusted based on the volume of incoming data.

h. The solution shall enrich or tag data via Lookup, Domain Name System (DNS), or GeoIP functions. Tags must include, but are not limited to, threat information, asset names, physical locations, and data owners. RBAC must be able to be applied to the destination based upon the tags/enrichment.

i. The solution shall include Redis as a key-value store for caching and enriching data, setting and retrieving key-values, and performing lookups against external Redis deployments.

j. The solution shall accept JavaScript expressions and provide flexibility to add or modify events using the native JavaScript language.

k. The solution shall segment large streams of events into discrete events within a data processing pipeline.

2.4 Monitoring

State requires the solution to include a monitoring console for tracking the health of deployments and the amount of data being processed by each source, destination, route, pipeline, and pack. Key metrics include:

a. Central Processing Unit (CPU) usage and load across Worker Nodes

b. Resource utilization

c. Data Throughput

d. Events Processed

e. Byte Counts

2.5 Metadata

State requires the solution to provide real-time access to detailed host metadata and insights into the processes, containers, files, and overall system state of the edge node(s) deployed on the host. Key host information includes:

a. Operating system version

b. Hardware specifications

c. Network configurations

2.6 Scalability, Licensing, and Customization

State requires a COTS solution that is flexible and scalable to enable expansion of data observability and processing features and broader analytical functionality to keep pace with the rapid evolution of data streaming technologies and the Department’s continuously evolving mission requirements. Key capabilities include:

a. The solution shall be a highly customizable COTS platform that enables applications, analyses, and workflows to be tailored to meet specific mission needs and user requirements.

b. The solution shall provide the ability to scale with increasing users, data, demand, and pipeline complexity, while providing fine-grained ways to adjust resource consumption. In addition, the solution shall have a proven ability to scale up to thousands of users (including thousands of potential outside collaborators globally), process hundreds of terabytes per day of both raw and processed data, handle daily changes in the number of terabytes processed, and manage thousands of complex pipelines requiring processing components developed in a variety of languages and environments. Deployments must be able to scale horizontally by adding more worker nodes and scale vertically by increasing the compute power for existing worker nodes.

c. Please describe the licensing/pricing model for the proposed solution including all relevant costs and fees, including costs/fees related to scaling the solution. This is not a request for quote (RFQ). The Department is seeking to understand the proposed solution’s licensing/pricing model, its complexity/simplicity, and its approach to effectively managing costs for the consumer.

2.7 Accessibility, Deployment and User Experience

State requires the solution to be versatile and user-centric, ensuring accessibility from various devices and browsers, while maintaining a consistent and intuitive user experience. The solution should cater to both technical and non-technical users and provide tools for easy application development and automation without the need for extensive coding. Key capabilities include:

a. The solution shall comply with all Section 508 requirements.

b. The solution shall enable access from various device types (e.g., mobile, tablet, desktop) and browsers (e.g., Chrome, Firefox, Safari).

c. The solution shall produce a consistent layout and an accessible user experience across specified device types and browsers.

d. The solution shall provide a user-friendly interface for both technical and non-technical users that supports seamless interaction with Artificial Intelligence (AI) tools and applications.

a. The solution shall provide clear error messages and offer effective feedback during system timeouts or other operational failures.

b. The solution shall offer a Low Code/No Code interface that leverages an intuitive and graphical interface to give users the power and freedom to automate business processes without having to write code line by line.

c. The solution shall be able to deploy worker and edge nodes on-premises (virtual machine or bare-metal), within a cloud environment, within container and Kubernetes environments, and at a remote post.

2.8 Training and Support

State will need access to a comprehensive training and support model able to support all personnel, from new users to the Department’s “power users” and system administrators. This training shall be provided as requested and required by the Department as part of the integrated solution. Key capabilities include:

a. The solution shall provide comprehensive training and education support. Please describe the approach to providing end user training and education, including training options, delivery methods, and any training that is included at no additional cost as part of the solution.

b. The solution shall provide and deploy support teams that can scope, prototype, and troubleshoot in real-time in accordance with government security standards.

c. The solution shall provide sandboxes for users to test functionalities and gain hands-on experience.

2.9 Other Considerations

Please provide information on the solution’s capabilities related to the following:

a. Given the continuously evolving nature of data streaming and AI capability, please provide information on any additional unique capabilities or practical experience your solution can bring to support the Department’s IT and security data management requirements.

b. Does your solution currently have an active or provisional Authority to Operate (ATO) for the Department? If your solution has an active or provisional, please provide an overview of the ATO, including the scope and what it covers.

SECTION 3: PAST PERFORMANCE

State is seeking a solution that has a proven track record in the USG with demonstrated experience supporting agencies of similar size, global scope, and mission complexity as State. Such experience is critical to ensuring the solution has been proven to be secure and reliable under the required IT security, data management, privacy, and other federal regulations that IT implementations within the USG environment.

Please provide the following information regarding your experience deploying your solution for USG customers.

a. Please list at least one (1), but no more than three (3), USG agencies where you have successfully deployed the proposed solution in the last three (3) years. For each experience noted please provide the information requested below to further demonstrate your relevant experience:

i. U.S. agency and organization name.

ii. Contract and task order number.

iii. Period of performance.

iv. Dollar value (across the full period of performance).

v. State whether your company was a prime or subcontractor.

vi. Customer point of contact (Name, Title, Email, Phone) that the Department may reach out to for more information.

vii. Description of the solutions provided to the Agency and relevance and applicability to demonstrating your proposed solution’s ability to deliver the full set of capabilities outlined in Section 2.

Disclaimer: This RFI is for information gathering purposes only as a means to identify interested and capable sources that can provide a solution that fully meets all requirements outlined in Section 2 of this notice.

The information provided in this Notice is subject to change and is not binding on the USG. State has not made a commitment to procure any of the items/services discussed, and release of this RFI shall not be construed as such a commitment or as authorization to incur cost for which reimbursement would be required or sought.

State will not publicly disclose proprietary information obtained as a result of this RFI. To the full extent that it is protected by law and regulations, information identified by a respondent as “Proprietary” or “Confidential” will be kept confidential. All submissions become USG property and will not be returned.

SENSITIVE BUT UNCLASSIFIED