BENGAL QA Round 4 FINAL12-20-2023C.pdf

BENGAL

IARPA-BAA-24-03

QUESTIONS & ANSWERS (Round 4)

# Question Answer

1 Must references fit into the 3-page limit? Or is there extra space for references?

White paper requirements are stated in Section 4.1 of the BAA. The page limit for the white papers is 3 pages. References must fit within the three-page limit. Additional pages will not be considered.

2 IARPA often has a requirement that the PI and co-PIS must have 25% effort. I see no statement about this in the BAA. Does this requirement apply for this super seedling?

The BAA provides all the requirements. The Principal Investigator (PI)/Key Personnel role is critical to the success of the overall project and the degree of involvement will be considered when evaluating overall scientific and technical merits. The BAA is hereby amended to clarify. In BAA Section 4.2.1, paragraph A., Proposal Overview, add the following to the project contributor paragraph: “Demonstrate that key personnel/Principal Investigator are sufficiently dedicated to the project.”

In BAA Section 5.1. add the following under paragraph A. Overall Scientific and Technical Merit: "The proposed technical team has the expertise and experience to accomplish the proposed tasks and is sufficiently dedicated to the project”.

3 If we submit the abstract in the early part of the submission window (e.g., Nov 27th), will we get feedback 2 weeks from the submission date (submitted 11/27, feedback provided 12/11) or will all feedback be provided 2 weeks from the final due date for submission (due 12/5, feedback provided 12/19)?

Section 5 of the BAA states: "All efforts will be made to provide a response within 15 days of the due date.” An earlier submission does not mean an earlier response. IARPA's intent is to provide all offerors the same amount of time from white paper notification to proposal due date. The BAA will be clarified.

4 Topic 3 subtopic 1 refers to "unlearning" -- is there a difference between "unlearning" and "correcting" the model?

Yes. The goal of unlearning is to remove inappropriate or sensitive information from the model. Model correction might leave inappropriate or sensitive information in place.

5 The wording also says "IARPA is not interested in filtering outputs". By "filtering outputs" does IARPA mean that it is not interested in approaches for post-generation correction? If so, is this the case for this subtopic or for all relevant subtopics e.g. methods for reducing hallucinations? Is IARPA interested in pre-generation corrections only?

IARPA is not interested in approaches that focus on filtering outputs (e.g. Regex, keyword filtering, hard-coded filters) in Topic 3.

However, IARPA is interested in other novel approaches for targeted “unlearning” in pre-trained or fine-tuned LLMs that are relevant to this Topic 3 sub-topic.

6 Do Academic Institutions Acknowledgment Letters need to be submitted with White Papers?

If applicable, Academic Institution Acknowledgment Letter(s) will only be required as attachment(s) to the Technical/Management Proposal (BAA Section 4.2.1).

7 Is it desirable to include references of the proposing team's work, and if so, are they excluded from the 3-page count of the White Paper?

See Answer to Question #1.

8 I am unable to access the IDEAS webpage to begin my whitepaper submission. The "Select BAA" field does not list the BENGAL BAA. What should I do?

A posting was issued on SAM.gov on 11/22/23 which explained the delay in IDEAS access. We are working to fix this. It should be ready to accept registration before the submission period for white papers begins on 12/1/23. If there are any further delays, a notification will be issued.

9 Per the solicitation’s instructions it does not appear that this organization will have to include a Transmittal Letter as part of the white paper package, but the IDEAS system requires that one be attached alongside a white paper and completed coversheet form before the whole package can be submitted.

As the solicitation doesn’t mention a Transmittal Letter during the pre-proposal phase (only the full proposal) or provide a template or instructions for one, we are assuming that this a system error. We are happy to put one together or use a placeholding document to satisfy the system’s requirement but wanted to run this by you first.

A transmittal letter is not required for the white paper. The IDEAS system lists the transmittal letter as “optional” because it is required for the proposal (we were limited to this choice by system constraints in the IDEAS set up). The transmittal letter link for the white paper, however, will take the Offeror to a page where they can object to Non- Government personnel seeing their submittal, if that is the case. The white paper is limited to 3 pages. Please follow the BAA instructions for the white paper requirements. Complete the IDEAS coversheet for the white paper (not included in white paper page count). For the proposal, you will also include the technical and cost proposal coversheets that are included in BAA Appendix A.

10 Does the three-page limit for white papers include references, or can those be included as an attachment (outside the three pages)

See answer to question #1

11 Is the white paper deadline still December 5th?

Yes

12 Also, in BAA, it says "The prime contractor must be a U.S. company."

Can a university be the leading PI of a proposal with other universities and a company as co-PIs?

Yes. Either a U.S. university or a U.S. company can be prime. BAA-Section 3.1.2, Foreign Entities will be clarified

13 if needed, can one site submit different white papers with varying teams for distinctive topics?

Yes

14 Is IARPA expecting that the PI will necessarily hold a PhD?

No, the BAA does not require the Principal Investigator to hold a PhD. However, in accordance with BAA Section 5.1, Overall Scientific and Technical Merit, proposals will be assessed to the extent to which “proposed technical team has the expertise and experience to accomplish the proposed tasks and is sufficiently dedicated to the project.”

15 Under the Topic 3 “unleanring” subtopic, are Retrieval-Augmented Generation (RAG) approaches of interest? The sensitive information can be compartmentalized into a separate reference datastore, utilized for generation for a variety of tasks, and not necessarily be encoded into the pre-trained LLM parameters.

IARPA is interested in novel solutions that advance state of the art. IARPA makes pivotal early technology investments that create or prevent technological surprise.

16 Related to Topic 3 test domain/environment (“offerors must propose unclassified testing and evaluation schemes that credibly simulate environments”), is the desire to have surrogate real-world use cases (e.g. copyrighted materials) or is it reasonable to have an ad-hoc unclassified simulation of multi-level authorized access to specific users?

Any unclassified testing and evaluation scheme that credibly simulates environments in which sensitive information must be protected is acceptable. This includes real-world use cases as well as ad hoc simulations.

17 Many classified documents contain images, charts or other non-text sensitive information. Does BENGAL envision removing the sensitive information from this non-text sensitive information while preserving its meaning? For example, in a graph one might remove the performance curve for a sensitive system while keeping the performance curves for other systems. Would removal of sensitive information from images be in scope, e.g. blurring or replacing a picture of a sensitive human source’s face?

Projects focusing on multimodal LLMs are within scope for all topic areas. Projects focusing on non-text models (e.g., computer vision models) are out of scope.

[REST OF PAGE LEFT INTENTIONALLY BLANK]

18 Can multiple non-duplicative proposals from the same organization be funded at $4M award value each, or should they be combined into a single proposal that covers multiple topics/TAs?

Yes, multiple non-duplicative proposals from the same organization may be funded at $4M award value each. It is anticipated that one proposal will be submitted for each White Paper research project that is “encouraged”. It is understood that White Papers and any corresponding proposals submitted, may include more than one topic/subtopic. Any proposal submitted shall be less than $4,000,000.00 for the combined Phase A (Base Period) and Phase B (Option Period) as set forth in the BAA. There may be cases where more than one White Paper from the same organization is “encouraged”. If the research projects are interrelated, the proposal stays within the BAA page limit (15 pages) and the cost does not exceed, $4,000,000.00, they could conceivably be combined into one proposal. However, it should be identified which White Paper research project(s) are being addressed in the proposal. The BAA will be amended to clarify. See Answer to Question #21.

Please note: If multiple proposals are submitted from the same organization and include the same technical team, the offeror must demonstrate, if more than one is selected, that the proposed technical team would be sufficiently dedicated to each research project contract. Further, If multiple proposals from the same organization are determined to be selectable for negotiations, the proposed costs/prices of the Offeror will be reviewed and adjusted, as necessary, to remove duplicative costs (See BAA Section 2.2, Multiple Submissions to the BAA).

19 Our team submitted 3 white papers.

If more than one white paper gets invited, can we squeeze all the topics into one proposal?

See answer to Question #18.

20 Do we get extra pages if we include multiple topics in a single proposal?

No. Also, see answer to Question #18.

21 On reviewing priority: if we include three topics in the proposal – 2 topics whose white papers are invited but one is not, would our proposal still get prioritized for review?

If your proposal submission contains any of the White Paper research projects encouraged during the white paper stage, it will be prioritized with the proposals that were encouraged. Additionally, the BAA is hereby amended to add White Paper Title(s) and IARPA Reference Number(s) to Transmittal Letter in BAA Section 4.2.1, Volume 1 – Technical & Management Proposal, as follows:

The Transmittal Letter shall include the following (not to exceed one page):

Introduction of Offeror and team (subcontractors and consultants), the BAA number, Offeror’s Program name, the White Paper Title(s), the IARPA Reference Number(s), the proposal validity period, the type of contract vehicle being requested (Cost/Cost- Plus-Fixed-Fee/FFP/Cost-Sharing procurement contract) with a short rationale, any non-negotiable conditions on which the offer is based (such as contract type, Intellectual Property (IP ) restrictions, etc.), any restrictions from review (e.g., Government Eyes Only or restricted from certain entities, see paragraph

4) and the Offeror’s points of contact information including: name, email and phone number for both technical and administrative issues.

22 What is the expected start date of the project?

Assume a period of performance start date of July 1, 2024.

23 Is there a preferred estimated start date for the BENGAL cost volume?

See answer to Question #22.

24 We appreciate any clarification you can provide and your consideration of our request.

In the BAA, the program schedule includes kickoff/PI meetings. Page 19 of the BAA states: “… Limited travel is anticipated. Offerors may require travel to meet with team members.

Offerors will not be required to travel to meet with IARPA.”

Will the kickoff and PI meetings be virtual? Or should budget travel for those limited meetings with IARPA?

The BAA is hereby amended to clarify. In BAA Section 4.2.2, Cost/Price Proposal, Paragraph B., Narrative Supporting Information, replace the entire paragraph with the following for Travel:

“The proposed travel supporting detail shall include destination and purpose of the trip, number of trips, number of travelers, days per trip and cost/price per traveler in sufficient detail to verify the BOE and determine reasonableness. Limited travel is anticipated; however, the proposed travel costs should align with the key activities reflected in the Figure 1 of the BAA. Kickoff/PI meetings will require in-person attendance of relevant personnel in the Washington DC metropolitan area. Additionally, Offerors may propose travel costs to/from team member sites, as needed to facilitate government site visits and/or as necessary in the performance of the research project. Proposed travel costs shall comply with the limitations set forth in FAR Part 31.

Conference travel will require prior approval from the Contracting Officer under any contract resulting from this BAA.”

25 What is the estimated start date of the program that offerors should use for budgeting purposes?

See answer to Question #22.

26 Will IARPA provide a template spreadsheet for the cost proposal?

Or should the applicant use its own spreadsheet?

Offerors can use the Cost Proposal Excel spreadsheet template provided with this Amendment 004 or submit contractor-provided spreadsheet files that satisfy the format provided in Appendix A.7.

Additionally, Amendment 004 corrects BAA # on Appendix A.1 and provides Word document templates for BENGAL BAA appendices.

27 Travel - the BAA mentions that “Offerors will not be required to travel to meet with IARPA” - does that mean that offerors do not need to attend any of project meetings in person (i.e. PI meetings, Kickoff, Demonstrations, etc)? Will those meetings be held virtually?

See answer to Question #24.

28 BAA states, “applications for the rapid contextualization of multilingual information.” Does it mean that evaluating our proposed approach(s) with such applications is mandatory?

No. Offerors must provide testing and evaluation procedures suitable to their proposed project. Successful submissions will clearly articulate a testing and evaluation protocol that can be either run on the performer site and validated by the T&E team or replicated by T&E using T&E’s own infrastructure. See Section 1.2 of the BAA.

29 (If [the answer to Question #28 above is Yes] does IARPA have any priority for languages other than English?

See answer to Question #28

30 Does IAPRA plan to support sole-PI proposals or encourage forming a team with other researchers?

The Offeror’s proposed technical team should have the expertise and experience to accomplish the proposed tasks and must be sufficiently dedicated to the project. See BAA Section 5.1.

31 Can we include PhD or MS students as key personnel?

See answer to Question #30

32 Phases A and B include “Gov. visits performer site.” Does it mean an actual visit to my research lab at a university?

Yes.

33 Do teams need to prove through human factors methods such as user testing that the proof of concept software is usable by non-computer programming government person[n]el?

Other reasonable methods to demonstrate usability are permitted, subject to the Government approval.

34 Can we expect usability feedback during the option phase from the T&E team?

Yes.

35 As part of the proof of concept software, for usability, is there an expection for the contractor to provide training?

The Performer must provide thorough documentation of the software.

36 If there is data not used in published work, is there still an expectation to publically release that data?

No.

37 Is there an expectation for deliverables to be sent to IARPA or the T&E sites through physical means?

Where appropriate, the Government may permit electronic transfer of deliverables.

38 May we include a table of contents for volume 1? If we do, please confirm it will not be counted in the 15 page limit.

A Table of Contents is not required but may be included if the offeror chooses. If so, it should not contain any additional information. It will not be included in the 15 page limit.

39 What is the definition of "perspective" and "perspective space" in the context of this program?

The term “perspective” is to be interpreted in its common usage (i.e., as a synonym for “point of view”). “Perspective space” is to be understood as a spatial representation (e.g., an embedding space) of perspectives.

40 Are approaches to debias LLMs within scope of topic 1?

No.