Questions and Answers - RFI for AI - ML Image Adjudication.pdf

RFI for Artificial Intelligence for Image Adjudication

1. Have similar services been performed in the past, or is this being considered a new requirement? May I have the incumbent information, if applicable?

Answer: CBP leveraged DHS’s Commercial Solutions Opening Process (CSOP) in 2023 to request similar services in the past. The original publication can be found here:

SAM.gov

2. What are the current LS-NII image formats, resolutions, and volumes per port of entry (images/day)?

Answer: The current LS-NII x-ray image data is formatted according to UFF2 or UFF3 standards. On average, CBP scans approximately 50,000 vehicles per day across all ports of entry.

3. Does CBP prefer cloud-hosted inference, on-premises GPU nodes, or hybrid deployment?

Answer: Algorithms will be deployed in the CBP cloud hosted by an internally developed execution environment, CBP is currently using a multi-cloud architecture. CBP may consider edge or hybrid models in the future.

4. What existing CBP data platforms must the AI component integrate with?

Answer: The AI component must integrate with CBP’s cloud computing environment, currently hosted in a multi-cloud architecture Algorithms will be deployed in an internally developed execution environment within these platforms.

5. What is the anticipated procurement timeline from RFI to RFP award?

Answer: This current RFI is not scheduled to move forward with an RFP award but rather, to investigate whether AI capabilities in this landscape have advanced enough for CBP to consider another RFP award.

6. The RFI states that many large-scale NII systems support the World Customs Organization Unified File Format and legacy data may not. To best target our algorithmic development and metadata-handling design:

a. Will sample UFF image files and the corresponding metadata schema definition be made available to RFI respondents (or RFP respondents in a subsequent solicitation) for development and pre-integration testing?

Answer: Yes, CBP data will be shared with a vendor should CBP decide to move forward with the RFP process.

b. If sample UFF data cannot be shared, can CBP point to a publicly available WCO reference set or simulator that vendors should target?

Answer: No data is publicly available.

https://sam.gov/opp/69b6a4d7b229418ab8f2d607083e58e2/view

c. What is the approximate ratio of UFF-conformant vs. legacy-format imagery in production today, and is the migration trajectory toward 100% UFF a stated program target?

Answer: The scope of this RFI is constrained to UFF-conformant image data.

7. The RFI distinguishes vendor-supplied test data from government-supplied data (Q9a):

a. Does CBP envision providing labeled training and validation data to selected vendors (e.g., via a secure government enclave, GovCloud workspace, or on-site CBP facility), or are vendors expected to bring all training data from their own internal sources?

Answer: CBP has labeled training data available for model development and evaluation.

b. For respondents without an existing CBP Background Investigation footprint, what is the expected pathway to access government data (on-site work at a CBP facility, sponsored BIs for a named-team subset, or a sandboxed enclave with reduced-fidelity data)?

Answer: CBP is currently exploring multiple pathways to enable access to government data for respondents without an existing CBP Background Investigation footprint. This includes providing onsite access to CBP image data at CBP facilities. Additional approaches, such as sponsored background investigations for select team members or the use of sandboxed enclaves with reduced-fidelity data, may also be considered as the program evolves.

c. Is there a CBP-managed simulated/synthetic threat-imagery dataset that vendors could use for early development, separate from sensitive operational data?

Answer: CBP does not currently have data available outside sensitive operational data.

8. Detection and risk-score interchange schema (cascading algorithms): Question 10d asks whether vendor models can interoperate in parallel or cascading operation via standard data structures:

a. Has CBP defined (or is CBP planning to define) a canonical schema for exchanging detections, risk scores, segmentation outputs, and provenance metadata between algorithms, analogous to DICOM Structured Reports or COCO/OGC standards in adjacent domains?

Answer: Yes, CBP will support - but has not defined - multiple models running in parallel and/or cascading operation. A given vendor's group of algorithms will be able to share data internally.

CBP would welcome vendor proposals for draft schemas for consideration.

b. If no such schema is yet defined, would CBP welcome vendor proposals for a draft schema as part of this RFI, with the expectation that a follow-on government-industry working group would converge on a final standard?

Answer: CBP welcomes vendor proposals for draft schemas for consideration.

9. Common Viewer integration interface: Questions 10b and 10c reference the CBP Common Viewer / Common Integration Platform:

a. Is there a published API specification, plug-in framework, or SDK for the Common Viewer that vendors should target for displaying areas of interest, risk scores, and officer-feedback capture (Q11)?

Answer: CBP currently hasn't published an API for sharing areas of interest or risk-scores. At this time, vendors can propose how their results would be formatted. Results would then be posted to the Kafka Results Integration Platform and further processed by NMT/CV software to be shared with end-user.

b. If the Common Viewer interface is not yet publicly documented, can CBP indicate the expected interaction model (REST overlay APIs, embedded web component, server-side rendered overlay, etc.) so that vendor architectures align?

Answer: CBP currently hasn't published an API for sharing areas of interest or risk-scores. At this time, vendors can propose how their results would be formatted. Results would then be posted to the Kafka Results Integration Platform and further processed by NMT/CV software to be shared with end-user.

10. The RFI emphasizes incremental implementation and references "explore and test AI/ML use cases.”:

a. What pilot pathway does CBP anticipate following an RFI response of interest? For example, a single-site shadow-mode evaluation against historical imagery, a multi-site live pilot with officer-in-the-loop scoring, or a benchmark-only evaluation prior to any field deployment?

Answer: CBP anticipates a shadow-mode evaluation using real NII data from all integrated CBP sites, followed by testing at a single location before any large-scale deployment.

b. What is the expected timeline from RFI submission to a possible pilot opportunity?

Answer: This current RFI is not scheduled to move forward with an RFP award but rather, to investigate whether AI capabilities in this landscape have advanced enough for CBP to consider another RFP award or technology demonstration.

c. What procurement mechanisms (OTA, SBIR, IDIQ task order, etc.) are under consideration?

Answer: CBP will determine procurement mechanisms if CBP determines to move forward with a procurement action.

11. Will CBP provide labeled NII imagery (with ground truth) for algorithm development and evaluation, or are vendors expected to supply their own training data?

Answer: CBP will provide labeled training and validation data to selected vendors, should CBP move forward with an RFP award.

12. What classification level will NII image data reside at, and will vendors require facility clearances beyond individual background investigations?

Answer: CBP is working to allow this at the lowest level, facility clearances will not be required at this time.

13. What latency requirements exist for image adjudication (real-time at the point of scan versus near-real-time batch processing)?

Answer: For the purposes of NII, CBP defines real time as less than five seconds, but operationally a smaller standard may be necessary. An AI model would have to execute against an NII image set in five seconds or less.

14. Does CBP anticipate a single integrator with algorithm sub-providers, multiple parallel algorithm contracts, or a hybrid approach?

Answer: CBP is considering both single integrator and hybrid approaches, with evaluations to determine the most effective model for operational deployment. Multiple parallel algorithm contracts may also be considered based on program needs.

15. Is CBP considering a phased pilot or evaluation prior to full operational deployment?

Answer: Yes, to both. CBP is conducting evaluations in a shadow mode environment using real NII data but without impacting operations, while determining if model performance is sufficient for field deployment. If a model is determined to be operationally suitable for field deployment, it would be deployed at a single location for evaluation prior to a large-scale deployment.

16. What contract vehicle(s) is CBP considering for follow-on procurement (e.g., existing BPAs, new full-and-open competition, OTA)?

Answer: CBP will determine procurement mechanisms if CBP determines to move forward with a procurement action.

17. Will CBP establish minimum detection and false alarm rate thresholds, or will vendors propose their own metrics?

Answer: A combination of both. CBP will ensure that through minimum detection and false alarm rates, the field personnel will not be negatively impacted however, CBP will consider vendor metrics as a suggestion.

18. How does CBP intend to measure "success" during an initial pilot—by throughput increase, detection rate, officer time savings, or a combination thereof?

Answer: A combination thereof and the answer will depend on the capability of the model. The operational goal of using AI/ML for NII adjudication is efficiency of operations; however, this must be balanced appropriately with ensuring sufficient detection rate.

19. Who is the incumbent on this contract? Please provide the contract number.

Answer: There is no incumbent and there is no current contract.

20. In the absence of an incumbent, how is similar AI/ML-assisted image adjudication currently performed within CBP operational environments?

Answer: CBP does not currently have similar AI/ML-assisted anomaly detection being performed in a live CBP operational environment.

21. What estimated volume of LS-NII images and associated metadata does CBP anticipate making available, and at what frequency will such datasets be refreshed or updated?

Answer: CBP curates a weekly dataset of several hundred annotated images. The applicable updated images are shared with the vendor weekly.

22. What are the anticipated data delivery formats for LS-NII imagery, metadata, manifest information, and operational case data?

Answer: Furnished image data will be UFF-compliant. CBP will supply metadata in standardized format of its choosing.

23. Please provide the expected turnaround time for CBP personnel to review and adjudicate contractor-generated detections, anomalies, or referral outputs.

Answer: CBP currently has a team of subject matter experts that meets with contracted vendors weekly to review annotated datasets and discuss contractor generated detections.

24. What is the expected turnaround time for CBP to evaluate contractor-generated detections, anomaly referrals, or model-generated areas of interest?

Answer: CBP currently has a team of subject matter experts that meets with contracted vendors weekly to review annotated datasets and discuss contractor-generated detections.

25. Will CBP provide feedback data associated with adjudicated results to support model refinement and continuous improvement efforts? If so, in what format and at what cadence will such feedback be provided?

Answer: Yes, CBP will provide feedback during model demonstrations, development and refinement project phases.

26. Please clarify the expected operational deployment approach for AI/ML models, including any necessary requirements for deployment within CBP enterprise environments. Additionally, specify if containerized, externally managed, or hybrid deployment models are acceptable, provided they comply with CBP security and integration standards.

Answer: The operational deployment approach is to first ensure that the model is training and will provide operational efficiencies for CBP, prior to a true field deployment, ensuring that using AI in the field will benefit operations. Once applicable, CBP will test a model at a single Port of Entry to validate operational efficiency, prior to a large-scale deployment.

Algorithms will be deployed in the CBP cloud hosted by an internally developed execution environment. CBP may consider edge or hybrid models in the future.

27. Can CBP clarify whether it envisions AI/ML algorithms being deployed primarily at the edge (port-of-entry sites), in centralized CBP data centers, or in a hybrid model, and whether any preferred cloud environments (e.g., AWS GovCloud, Azure Government) or baseline performance/throughput expectations will be specified in a future solicitation?

Answer: Algorithms will be deployed in the CBP cloud hosted by an internally developed execution environment. CBP prefers platform agnostic models, but may consider edge or hybrid models in the future.

28. Will the Government please provide the percentage of the historical LS-NII data is currently stored in WCO UFF versus proprietary OEM formats?

Answer: The scope of this RFI is constrained to UFF-conformant image data

29. Pertinent to non-UFF data, will the Government please provide the proprietary SDKs/APIs necessary for image normalization, or is the vendor expected to provide their own parsers for each OEM?

Answer: The scope of this RFI is constrained to UFF-conformant image data.

30. Pertinent to Latency Requirements, for "pre-primary" drive-through concepts, will the Government please clarify the maximum allowable latency thresholds for an AI inference to be returned to the remote command center officer?

Answer: In real time is defined by CBP as less than five seconds, but for operational efficiency this is typically shorter.

31. Will the Government please clarify if the "existing architecture" prioritizes inference at the Tactical Edge (at the Port of Entry) to handle high-throughput vehicle lanes, or alternatively, it is the Government intent to the focus on centralized Cloud-based adjudication?

Answer: Algorithms will be deployed in the CBP cloud hosted by an internally developed execution environment. CBP may consider edge or hybrid models in the future.

32. Will the Government please clarify if the Data collection and data holding methodology includes the requirement for data management and storage?

Answer: CBP is managing data holding, management, and storage internally in a government owned system. Data collection is being done using both internal government solutions and through the contracted common integration platform.

33. Pertinent to Data Diversity, will the Government please provide a diverse "training set" that includes images across different Original Equipment Manufacturers (OEMs) to ensure the model generalizes across various X-ray sensor technologies?

Answer: A successful model will generalize across the scanner data provided by CBP, which will be as diverse as is feasible.

34. Will the Government please clarify if it intends to utilize FAR 8.4 procedures to issue a subsequent, targeted Request for Quotation (RFQ) exclusively to the most qualified vendors identified through these RFI responses to streamline the acquisition of these AI/ML capabilities?

Answer: CBP will determine next procurement steps if CBP determines to move forward with a procurement action.