(2) Appendix A Building IC Data Consortium Interface at www.icdata.gov.pdf

UNCLASSIFIED

Appendix A

Building the IC’s Data Consortium Interface at www.icdata.gov

Many of the IC Data Consortium’s (ICDC) design features are novel, transformative concepts that will create a new operating model. To mitigate legacy operational assumptions and foster innovative thinking, flexibility around critical principles, rather than overly rigid and lengthy

“requirements” will be crucial for a successful prototype. The Government’s intent and technical specifications will primarily be conveyed through the design principles listed below (See also

Figure 1). The Respondents will animate these principles with specialized labor and creativity to create a holistic system. The “Technical Narrative” section will expand on some of the design principles in greater detail but listing the short-hand version up front sets the tone for this project.

Respondents are encouraged to print or post these design principles in their work environments as a reminder of the Government’s intent and the challenges to be solved.

ICDC Design Principles

• Data is accessed at the unclassified computing level.

• The aspirational architectural goal is “zero copy.”

o Data will be queried in place – to the maximum extent possible on vendor IT systems (i.e., data provider source query mechanisms). www.icdata.gov is largely a marketplace to query and interact with vendor holdings.

o To the maximum extent possible, data will not be deliberately copied or ported to classified government computing systems. The intent is on-demand access at vendor origin, not making copies or indexing substantial local data copies unless necessary on an unclassified system.

o Data will only be transferred to government-controlled computing systems for performance increases such as caching or building a minimally sufficient graph, data store, or index. Majority of changes and updates to the data will come from the vendors’ infrastructure, not replicated data on government infrastructure.

• All work will remain unclassified. The Government may not sponsor Top Secret clearances by default and may use other background check options to keep Respondents focused on unclassified-level work.

• www.icdata.gov will have a minimum of four data interaction options: indexed GUI/WUI search, API calls, bulk data access, and external web portal login options.

• Open source code will live in government-hosted version control and Continuous

Integration/ Continuous Development (CI/CD) pipelines. Respondents will focus on innovative techniques and design elegance, not source code hosting.

• Software created should be modified open source software to the maximum extent possible. Proprietary software may be used as sub-components within the overall architecture, but the Government prefers “mixed source” (closed source and open source) to maintain some flexibility within the core or extensions. Because the design goals are unique, the use of pure “off-the-shelf” or pure proprietary software to cover all functions is not desirable: substantial source code-level flexibility is required to avoid “vendor lock.”

• www.icdata.gov will be automated and rebuilt with updates and upgrades every 120 days

– DevSecOps best practices. Version numbers will follow 1.0 for major updates (v 2.0),

1.1 for minor updates (v 2.1), and for 1.1.1 patches and bugs (v 2.1.1). See the Semantic

Versioning 2.0.0 (https://semver.org) specification for additional guidance.

• All developed API’s will follow standard industry documentation formats. See OpenAPI

Specification (OAS) v3.1.1 (https://spec.openapis.org/oas/v3.1.1.html) for additional guidance.

• Respondent designs should include intentional thought into ensuring the application produces relevant system and service observability metrics that makes application performance, usage and issue detection straight forward when later integrated with modern time-series databases and dashboards.

• Data vendors will receive continuous data usage feedback (token usage, product citations, and qualitative notes) to continually improve data quality and service delivery to www.icdata.gov.

o Respondents will need to build a feedback service within www.icdata.gov where data vendors can receive the feedback noted above, privately and isolated from the broader catalog, within a tab or widget located near their data holdings within the catalog of www.icdata.gov.

Figure 1

Technical Narrative

Attribution

The ICDC is an overt program as ODNI, the host, is an overt organization. Government infrastructure will be attributable to the US Government, ODNI, or a vendor operating in an attributable manner on the IC’s behalf. All data relationships will be held in strict confidence at the contractual level, but any proxy or other non-attributable IT arrangements will be judged on a case-by-case basis but will not be the default arrangement. These types of IT arrangements can increase operating costs and complexity.

When usernames or keys are issued for data access, they will follow a standard convention such as user001odni, user002odni, key003odni, etc. These are just samples.

No agency-specific handles will be used. Portions of security and risk reduction will be achieved by abstracting agency and non-Title 50 entity interactions within the broad overarching equities of ODNI, thus obscuring any specific participation from individual participants.

API catalog

A robust and well-documented catalog describing vendor APIs, how to interact, and how to apply for API keys must be built inside of www.icdata.gov. If a vendor already describes their APIs robustly, yet hosted in other places, links to that documentation will be sufficient if it is easily accessible. The Respondent must build a talkback mechanism within www.icdata.gov for IC developers to request and be assigned API keys associated with ODNI for clear tracking and metrics. If vendors develop or modify APIs for listing with www.icdata.gov, OpenAPI

Specification (OAS) standards are required.

Simple search, parsing and indexing

As noted in Figure 1, www.icdata.gov’s holdings will be parsed and indexed so regular users can search and interact with the marketplace and its connected holdings. Internal government power users and software developers may build local services with API keys and object storage accesses listed within the well-documented API catalog, but www.icdata.gov must be layered for both regular users (non-developers) and power users (developers). The Respondent must also integrate a vendor’s API offerings into www.icdata.gov for regular users to discover via auto-completing search. More simply, the interface must strike a balance between catalog and research platform, especially if APIs offer search as a service.

If a vendor utilizes object storage (buckets), parsing and indexing the object storage files will be required for regular users to interact with the data within the object storage systems via auto-completing search. The search index should query object storage data in-place to minimize data replication to government-controlled unclassified cloud systems. Some naming convention standards within object storage systems and other open standards may be required to maximize parsing and search. The Government will engage with Respondents frequently and iteratively regarding minimal standards to improve search and discovery from object storage systems.

Bulk data icons explained

As noted in Figure 1, object storage systems will likely contain CSV, JSON, Excel, KML, XML files, etc. The auto-completing search suggestions and the more detailed search returns derived from object buckets should display simple file type icons near the search return results to alert users that downloads are available (bulk access). Respondents will need to coordinate with data suppliers and possibly create access-based controls to efficiently manage large downloads that can incur substantial transfer and egress charges, especially multi-cloud transfers. Respondents will also need to work with data suppliers to set thresholds, and possibly re-enforced with access controls, if API consumption and billing is based on the volume of data returned or other metered arrangements.

Bulk data access options may also be displayed within a simple catalog portion of www.icdata.gov and described in the API catalog.

Metered APIs and Data Tokens

Metered APIs can help the Government move more toward consumption-based pricing rather than sweeping “enterprise” arrangements. The Consortium Manager will be tasked with potentially linking “token-based consumption” billing arrangements to existing metered APIs to gain further efficiencies. A token may be a time-based unit of accounting for accessing data and/or a volume-based unit of accounting for accessing data. For example, the ICDC may purchase 20,000 data tokens for six months from a data vendor to govern and monetize access to their data. Respondents should think about coding arrangements to assist with the execution of the token concept, but do not need to expand on it significantly within initial white paper submissions.

Metadata Harmonization

The Respondents need to propose ideas to generate an index that finds commonalities in diverse data types and descriptions as an initial function then work with data suppliers to refine, if needed, for effectively querying across diverse datasets remotely. Metadata harmonization proposals may be based on scripts or Machine Learning (ML). However, if ML is introduced to help with metadata harmonization, it must not overwhelm the initial primary goals of effective federated search from remote sources without copying data and ordering and documenting APIs for the creation of flexible services. Once API accesses are ordered and well documented, next steps using Large Language Models (LLMs) querying remote data without copying it locally or other federated approaches may be pursued in the future.

External web portal logins and IC-wide login service

Programmatic data access, where vendor data is decoupled from their web portal, described in

Figure 1 will be the ICDC’s primary interaction mechanism. However, the Respondent must also build a service that tracks the number of users logged into vendor web portals when using the vendor’s web portal to navigate the data is far more impactful than the decoupled interactions within www.icdata.gov. Vendors will issue logins based on the ODNI naming conventions such as User001odni. If 20 external web portal logins are purchased by the ICDC, a service running inside of www.icdata.gov must display how many logins are checked out, how many logins are available, a notification function when 80% of the portals are in use, and a wait time estimate if all logins are checked out. The Respondent will work with each web portal supplier to assemble the data and event or time-based functions for the “login” service to function within www.icdata.gov.

Government-hosted version control

Respondents will be provided with onboarding instructions needed to receive accounts for version control and other CI/CD tools sponsored by the Government where all open source code for this project will live and be the primary location for development and deployment of built services. If multiple Respondents are selected, all parties will work together in the same repositories.

Top-level domain name

The proposed URL, www.icdata.gov, may change later.