Kallisti QA_v2.pdf

(Unclassified)

(U) DARPA STO Kallisti Industry Day Submitted Questions and Answers

(U) Kallisti Environment

(U) E2. Who is developing the Kallisti M&A and Adjudicator Environment?

(U) Answer: DARPA has selected a government team from AFRL and other partners to develop the environment.

(U) E3. In phase 1 will there be real-world data provided? What data will be our first results being based upon?

(U) Answer: All data is simulated as real-world data is insuƯicient in volume to test generality of performer models. Initial data will be hosted on the Gov-Cloud instance.

Please refer to the CUI version of the program solicitation for details.

(U) E4. Will performers be able to expand the information available to performer solution from the simulator?

(U) Answer: DARPA had designed and specified the simulator environment to support performance goals of Kallisti.

(U) E5. How many actions & eƯects to actions would exist in the simulation environment?

Are actions and reactions any way related to dots moving on a map? How bad would it be to apply brute force as a solution?

(U) Answer: DARPA anticipates that brute force approaches may not scale. Proposers are encouraged to propose any approaches they believe will best achieve program objectives.

(U) E6. Can any details be shared on the specific compute resources being provided as

GFE?

(U) Answer: GFE compute resources for Kallisti will be Gov-Cloud.

(U) E7. What will be the fidelity of the simulation? E.g., will the model include geography, geopolitical situation and tensions, actor military capabilities, history of past actor behaviors and reactions, etc

(U) Answer: See E1, E3, T13 and T20 answers.

(U) E8. Can they share with us more about their Sim tools? Interface? Capabilities?

Fidelity? Compute? What data points will be provided for red team? How much apriori knowledge can we assume about red team? ISR, capabilities, etc. Should we supply our own sim and scenarios? In parallel?

(U) Answer: See E1, E3, E6, T13 and T20 answers. Performers are expected to use the government provided environment for milestone evaluations. Performers may propose the approach they believe will achieve Kallisti program objectives.

(U) Kallisti Technical

(U) T1. Would numerical solutions as opposed to closed form (mostly) solutions be acceptable? Is explainability of results important? Would use of pre-trained large models be, ok? Some of the moves you described say for the Spratly islands are expressed in high level language. Will the simulation environment accommodate such high-level expressions?

(U) Answer: DARPA believes this is an open research question on the program. DARPA is open to a diversity of potential approaches the proposers feel can accomplish program objectives. For simulation environment questions see E1.

(U) T3: At the perception & comprehension stages is the forward model (black box) informed by human decision-making patters? Or are they simulated projection algorithms on arbitrary graphs? Is irrational behavior considered/modelled stochastically?

(U) Answer: DARPA will use simulated algorithms that are not fully representative of, but attempt to capture, human behavior. DARPA will explicitly model irrational behavior and performers are encouraged to propose their best approaches to solve this challenge.

(U) T4. Does Kallisti have an interest in deciphering human perception/cognition/action patterns of apparently irrational behavior?

(U) Answer: Yes, addressing irrational behavior is in scope and DARPA will introduce these constraints.

(U) T6. Is resilience to data poisoning value for establishing a Theory of Mind (TOM)?

(U) Answer: Data poisoning attacks are not a part of the Kallisti program objectives.

(U) T7. Is having an extensive human brain processing model based on high-resolution human subjects’ data a valuable asset of a proposal?

(U) Answer: Prior work in this area is potentially useful. Proposals should include scientific evidence and prior work for any proposed approaches. DARPA is looking for surprising results and novel solutions. Note that Human Subjects Testing (HST) is not authorized the Kallisti program.

(U) T9. How important is explaining power vs. predictive power of the submitted models?

How important is it for solutions to be explainable/interpretable vs. black box? How much of a priority will interpretability be?

(U) Answer: Explainability, interpretability are not necessary for Kallisti goals but are of interest to the program.

(U) T11. Will the testing and evaluation models produced by IDA capture and represent cultural properties that influence decisions? E.g. Will a model be clearly from [country A] and [country B]?

(U) Answer: Cultural properties are in scope for test and evaluation models but will be generalized in terms of “Greenlandia”, “Redlandia”, etc. Please refer to E1 and the CUI version of the program solicitation for details.

(U) T12 Q1. In FA1, are performers sending as input only the belief state that will result in the desired action? Or are they also providing a blue team plan to induce those beliefs? Q2.

Will platforms always be identified with regards to their country, designation, and type, or can these details be obscured? How do we ensure that our representations for belief and risk/reward match those being used for evaluation as needed for accuracy measures? Can you provide any examples of what a “belief” would be that we would match?

(U) Answer T12 Q1: DARPA will initially be providing an Oracle function which returns next action from an adversary from a belief state during the sprint associated with FA1. In later Kallisti sprints DARPA will remove or reduce functionality of the black box Oracle.

(U) Answer T12 Q2: Some of these details may be obscured as a result of scenario specifics. One of the goals of Kallisti is to develop the capability to develop the model of behavior based upon information available (limited, incorrect or missing information) that is representative of real-world scenarios. Please see the CUI version of the program solicitation section 1.5 and 2.0 for an example.

(U) T13 Q1. Should the potential COAs in national security scenarios of Kallisti account for all aspects of DIME or DIMEFIL (Diplomacy, Information, Military, Economic, Financial, Law enforcement)? Q2. Will non-physical considerations and communications (policies, treaties, agreements, etc.) be supported with natural language text or will they entirely be represented within the Kallisti API and its ontology?

(U) Answer T13 Q1: DIMEFIL is in scope for Kallisti, however DARPA anticipates use of a subset of DIMEFIL. DARPA is leveraging other DoD work in this space and will be incorporating best in class results for Kallisti’s test and evaluation environment.

(U) Answer T13 Q2: A limited set of DIMEFIL considerations will be represented within the Kallisiti API.

(U) T14. What is the scale (number of platforms, area covered, timescale) of the scenarios?

(U) Answer: DARPA doesn’t expect the scale to exceed 10,000 units unless needed for testing. The objective of Kallisti is develop the Theory of Mind as the focus to solve the problem and to test the limits of that hypothesis. Please refer to the CUI version of the program solicitation for additional details.

(U) T15. What is the richness of geopolitical relationships, action goals, intentions, etc.

expected in the scenarios that will inform platform behavior?

(U) Answer: A: Performer proposed approaches should not be sensitive to or dependent upon the backend. Geopolitical aspects will be part of the API and surrogate metrics for Kallisti. Please refer to the CUI version of the program solicitation for details.

(U) T16: Action state space: Should we model “continuous, high- dimensional” actions or would representing discrete actions (states) be suƯicient? If continuous – can you talk about how “beliefs” should be provided for evaluations (e.g. As parameterized probability densities or value functions?).

(U) Answer: One objective of Kallisti will be to evaluate the sensitivity of COAs to continuous and discrete states and action spaces.

(U) T17. Times and dynamics: Are actions taken in turns synchronously by blue and red? Do all actions have same durations? Or should we model “continuous” times (non-discrete, non-synchronous). Can you also talk about important of learning state “dynamics” vs rewards/interests).

(U) Answer: DARPA intends to provide a hybrid between continuous and discrete time.

(U) T18. Multi agent interactions: Is it important to represent how red agents interact/communicate to achieve their objectives, synchronize their beliefs, understand “each other”, decompose their group value / reward into individual values/polities?

(U) Answer: DARPA considers this one of the research questions to solve.

(U) T19. Training data: Can you tell us if training data will contain labels/rewards/red intent?

(U) Answer: DARPA has not decided on the level of labeling to provide in training data yet.

The goal is to include a realistic training data set for the performers. See T24 Q3.

(U) T20. Can you describe your vision for the state space and actions? Is modeling EW, asset detection, or ISR too fine of resolution?

(U) Answer: DARPA’s vision includes assets and non-physical components that will pull results from miliary and DIMEFIL sources (results of upcoming election, overflight rights, etc.). The state space will vary for each scenario and will be described in the API and scenario document provided to each performer.

(U) T21. To what extent should cognitive factors be reflected and incorporated in the adversarial theory of mind model?

(U) Answer: Performers are open to include all factors they deem necessary, including cognitive factors, when developing models to achieve overall program goals. Evaluation will be based on program metrics. Please refer to the CUI version of the program solicitation for details.

(U) T22. If an adversary is not purely rational but has a cohesive strategy, what constrains the adversary’s behavior?

(U) Answer: Irrational actors are in scope for Kallisti with the overall goal of developing the most eƯective models for realistic adversaries that meet program metrics. Adversaries will be driven by commander’s intent defined by the test and evaluation team.

(U) T23. What is the expected form of the government provided models of comprehension and projection in Sprint 1? For example, would they be in the form of a Markov Decision Process, or perhaps a black-box simulation model?

(U) Answer: The government will provide black-box simulation models for comprehension and projection.

(U) T24 Q1. Are scenarios across sprints going to be common? Or will diƯerent scenarios be provided for each sprint? Q2. For the perception component in Sprint 1, what types of input/ information can we assume will be provided to performers?

Q3. Will a time series of data be available for each agent? What about the granularity of such data? Will data provided have (partial) labels? What data or information beyond that provided can we use- e.g., publicly available GIS/ satellite data corresponding to a given map of a region of the world? Q4. Will information about the performance of SOTA frameworks be provided to performers as a reference?

(U) Answer T24 Q1: No, DARPA anticipates diƯerent scenarios across the sprints. Please see T2 Q3 answer.

(U) Answer T24 Q2: DARPA will provide various types of state information for performers;

this will include but not limited to time, Time-Space Position Information (TSPI) and other data elements related to the scenario. Please refer to the CUI program solicitation details related to “...common, partially observable, environments.”

(U) Answer T24 Q3: DARPA intends the Kallisti program to cover partially observable environments where data is incomplete, when outside of sensing range. Granularity of data will vary on the basis of simulated sensors and capabilities. Kallisti scenarios, while based in realistic scenarios, will not directly model real world entities, commanders, or crisis scenarios. Real world historically information may be freely used for transfer learning purposes. DARPA anticipates providing reference benchmarks to performers.

(U) Answer T24 Q4: Performers are encouraged to research current performance of SOTA frameworks; DARPA will not provide this information to performers.

(U) T25 Q1. What are the planned types or classes of observables and actions that will be available through the simulation API? Some approaches may need specific types of actions to probe or influence the simulation to cause observable side eƯects or may need specific types of observables that are normally available in the real world. Will IV&V work with performers to refine the supported observables and actions needed to support proposed technical approaches?

(U) Answer T25 Q1: Please see T2 answer.

(U) T25 Q3. How will performer technologies determine if a simulated actor's behavior violates laws, treaties, or other agreements? Will the simulation provide this information or will performers be given the applicable laws, treaties, and agreements to use in analyzing the simulated actor's behaviors?

(U) Answer T25 Q3: DARPA will provide relevant information for each scenario suitable for analyzing the actors behavior. DARPA will provide a limited set of DIMFEL information that will be necessary and suƯicient for program goals. See T24 Q3.

(U) T25 Q4. At industry day, it was indicated that the simulation API will return observables with defined semantics so that NLP and image recognition will not be needed. Will observables include textual or other communications between and public statements by simulated actors?

(U) Answer T25 Q4: Observables will be scenario dependent and may include text based communications and/or other meta data.

(U) T25 Q5. What will be the time window for observables? E.g., Watching a ship turn and move out of a zone occurs over some period of time.

(U) T25 Q5 Answer: See answers to E1, T2, T5, T14, T17, T24 and T25.

(U) T26 Q2 Answer: Can you confirm that there is no expectation of a human-machine interface for this program? Although it may be within scope to build engineering interfaces for development, testing, and evaluation?

(U) Answer T26 Q2: While human-machine interfaces are desirable, they are not necessary for program metrics.

(U) T27. Within context of the government provided simulation environment, is it possible for Blue agents to take actions that change environment states that aƯect observations by Grey and Red agents?

(U) Answer: Yes.

(U) T28. Is there an enemy order of battle input to the system that specifies red platforms and actions they are capable of or does that need to be inferred?

(U) Answer: That needs to be inferred.

(U) T29. What are the inputs and outputs of the government simulation environment? Are beliefs and intent part of the inputs to the simulation and/or are they available for training purposes?

(U) Answer: See T2 Q1 and T24 Q3.

(U) T30. What are red sensors used to perceive blue and does blue observe red sensor actions or have knowledge of them?

(U) Answer: See T24 and T25 answers.

(U) T31. What is the blue action space? Does it include posturing/ positioning of blue assets? Does it include deception/ decoys? Does it include messaging?

(U) Answer: Blue action space will depend upon specific scenarios which are under development. It may include positioning, messaging, deception, decoys. See T13 and T20.

(U) T32. Is there a baseline red and blue agent in the simulation?

(U) Answer: No, agents will depend upon specific scenarios.

(U) T33. What are the inputs and outputs for models for each of perception, comprehension, and projection? What prior intel is available on red – is it just sensor observations of red platform positions or do we have additional information such as partial information on red acceptable risk or intent?

(U) Answer: See T24 Q3.

(U) T34. Do we have a priori information for archetypes (either of individual platforms or of the nation state)?

(U) Answer: No.

(U) T35. What is a desired set of actions by blue in the fishing example or a range of such actions?

(U) Answer: Desired actions will depend upon specific scenarios

(U) T36. For projection, is the goal to predict a set of actions for all non-blue platforms or a specified platform or set of platforms?

(U) Answer: The goals will be scenario specific see CUI program solicitation section 2 for details.

(U) T37. Are we assuming red has perfect comms or are we also inferring the comms structure of red? Are we assuming red is homogenous or that diƯerent red platforms or groups of platforms may have diƯerent intent?

(U) Answer: Specific assumptions, if any, will be provided in each scenario.

(U) T38. Does blue have a specified acceptable level of risk for its actions? Is blue using assets that may be hard to perceive (e.g. stealth) or using techniques like decoys? In general, is it hard for red to observe blue or is it easy or could it be either?

(U) Answer: Specified risk levels and asset details will depend upon each scenario.

Observation of blue assets could be either easy or hard.

(U) T39. Will the gov be supplying Red agent tactics/intent? Will we have inter-adversarial play? I.e. will red agent be too easy to beat?

(U) Answer: The government anticipates supplying one or two Red agents for training and withholding further Red agents for testing.

(U) T40. Could they provide more information on “government provided model of comprehension and projection”

(U) Answer: Information about the government provided model will be provided with each scenario. See T24 Q3.

(U) T43. Can you provide more information on how the metrics will be evaluated with respect to the output of the model? What data will the government team be evaluating for program metrics across all three sprints.

(U) Answer: Please see T15, T21 and T26.

(U) T44. Will performers have access to the simulator in order to interact with Red and learn from these interactions ahead of the evaluations? If so, are there limitations on the amount of interaction? Will performers be able to interact with the simulator by specifying Blue actions and observing Red reactions? Will performers be able to interact with the simulator by specifying specific scenarios to play in?

(U) Answer: Please refer E4, T2, T16, T17, T18, T20, T41 and the CUI program solicitation.

(U) T45. Will the government or T&E team provide any "practice tests" that performers could use ahead of the midterm and final evaluation events to gauge the performance of their modules?

(U) Answer: Performers will be able to coordinate with the government team ahead of time to test their solutions ahead of time in the government environment. Please refer to the CUI program solicitation.

(U) Kallisti Programmatics

(U) P1. How will funding sequencing within each sprint be managed. E.g.: Upfront, incremental vs success on metrics? What are the time limits for each sprint? With milestone payment schedule, are performers operating “at risk” in direct competition with another? This seems to discourage risk taking.

(U) Answer: Kallisti will use standard milestone payments used by the Other Transaction agreement. Proposed approaches should reflect what each team believes is feasible to achieve Kallisti goals. Please refer to the CUI version of the program solicitation for details related to sprints, metrics, milestones and payment schedule details.

(U) P2. What will the size of each proposal be roughly. Or in other words, what is the size of the program? Are you planning to release program budget and/or number of performers? If so, what is planned?

(U) Answer: DARPA is anticipating multiple awards but does not release program budget information.

(U) P3: How does the program diƯer from the DARPA COMPASS Program?

(U) Answer: Kallisti will research the ability of AI technology to infer a theory of behavior model from observations. Please refer to the CUI version of the program solicitation for details.

(U) P4. Is there more than one presenter permitted for Abstract/Proposal Oral presentations?

(U) Answer: Yes, performers are permitted to have more than one presenter provided the performer’s presentation follows the guidelines of the solicitation.

(U) P5. Do you expect this to be multi-institutional collaborations, or can also one organization apply?

(U) Answer: DARPA encourages multi-disciplinary teams that may or may not include multi-institutional collaborations.

(U) P6. We’re quite a small non-traditional performer bringing a valuable solution from outside typical approaches – how do we best ‘project’ our alignment with the program despite not having established relationship in the existing community?

(U) Answer: DARPA encourages novel, multi-disciplinary approaches from non-traditional groups to address the challenges and goals of the Kallisti program. DARPA is interested in building a community around this research and development area with a diƯerent set of people with a wide range of ideas and performers. Existing relationships are not required for successful proposals.

(U) P7. For university performers, is there an option to partition work at (U) and/or fundamental research?

(U) Answer: This program will not be fundamental research and will not be initially authorized for public release. However, performers can submit through the DISTAR process and request public release of specific results.

(U) P8. Who will the end user of the models?

(U) Answer: Use of Kallisti models depends upon the success of the program and demonstrated capabilities by the performers.

(U) P9. Will there be any phases/sprints within Kallisti eƯorts that involve direct engagement with end user customers other than DARPA or other research partners? In other words, is the intent that Kallisti vendors remain attached and become part of the implementation process for successful projects?

(U) Answer: DARPA’s intent is to introduce performers to interface/interact with end users when appropriate. Please refer to the CUI version of the program solicitation for information on security. Please refer to DARPA Connect to understand transition opportunities.

(U) P10. What is the tech transition strategy to ensure this work finds applications?

Pentagon + Treasury were mentioned. What is AFRL’s Role? Other DOD oƯramps?

(U) Answer: DARPA anticipates a significant potential for DoD, no-DoD transitions and other oƯramps. World events may change transition priorities.

(U) P11. The CUI Addendum for the Program Solicitation currently indicates CTI, EXPT, and OPSEC CUI Category with Distribution Control of Distribution Statement D (Department of Defense and U.S. DoD contractors only). There is no NOFORN category listed. Q1: Does Distribution D limit the releasability to a foreign entity? Q2: Would extraction and sharing of only material with (U) Unclassified paragraph-level markings allow for distribution to foreign entities?

(U) Answer P11 Q1: Yes, Distribution D does limit the releasability to foreign entities.

(U) Answer P11 Q2: The performer could work with program security to put together an unclassified version of the document (DARPA will need to see it and concur before they send it). We want to emphasize, that it is the performer’s responsibility to follow export law and not release anything that would require a license to a foreign party.

(U) P12. Given the CUI status of much of the BAA, do you anticipate any issues with structuring a proposal so that university team members work on non-CUI parts of the project and a corporate member works on all the CUI restricted parts?

(U) Answer: DARPA does not have any issues with this as long as the Universities understand that this will not be Distro A, and they will have to go through our DISTAR process. DISTAR is DARPA's public release process.

(U) P13. Is the intention is still to allow open academic publication? (i.e., should I interpret the DISTAR requirement as a procedural delay in publication to ensure that no CUI information is released, or as a stronger signal that open academic publications are discouraged).

(U) Answer: DARPA instructions emphasize the desire to release all results at the broadest level of dissemination possible while maintaining concerns related to security. While we can never promise an individual result will be releasable to the public, the DISTAR process is a normal part of our procedures and is frequently and successfully used by performers.

Non-US persons generally cannot be key members of an eƯort on a CUI program, but there is a process to submit requests for individual participation on defined portions of a project which will need to be reviewed on a case-by-case basis.

(U) P14. Q1. Will US Persons (permanent residents/green card holders) be eligible to work on Kallisti? Q2 Can a US person (green card holder, but not yet a citizen) be in a PI role in this program?

(U) Q1 Answer: Yes. Please see the DARPA CUI guidelines.

(U) Q2 Answer: Yes. They can only work on Unclassified and CUI.

(U) P15. Reading through the CUI solicitation please provide if the exit criteria is dictated by the Gov per sprint (discussed in section 2.1, page 10) OR by the proposer (discussed in section 3.4, page 20) which then presumably the Gov would adopt for evaluating the success of the performer in the sprint as exit criteria for payment? (U) Answer: Per section

3.4 the performer should propose milestones, payments and exit criteria for their proposed approach in accordance with the Kallisti program metrics and schedule of sections 2.1 and

2.2. If selected, upon satisfactory contract negotiations, the government will approve the proposed exit criteria. Please refer to sections 1.4 and 2.5 for additional details.

(U) P16. Just confirming that both the written and oral (slides) components of the abstract are due on February 28? May we make modifications to the slides for the oral presentation?

Are references included in the 3-page limit?

(U) Answer: Yes. both written and oral (slides) for abstract are due Feb 28. Please have a final copy of oral presentation submitted by Feb 28th. Yes, any references must be included in the 3-page limit.

(U) P17 Q1. Is it the Government’s intent to preclude UARC participation in the solicitation?

If so, the word “all” surely seems out of place. Q2. Given the DARPA outreach to UARCS, how do connect these words to DARPA’s sentiment expressed on Oct 1?

(U) P17 Q1 and Q2 Answers: UARC should reference Section 2.9 for eligibility.

(U) P18 Q1. We are preparing an abstract that will not include any CUI-specific information.

Considering this, would it be acceptable to submit an abstract that is not designated CUI?

Q2. Is there an expectation that all proposal responses must be marked as CUI since the program and solicitation documents include CUI information? If so, what are the required CUI markings, and who is the point of contact for guidance?

(U) Answer P18 Q1: Yes, unclassified proposals may be submitted. See the updated program solicitation for details.

(U) Answer P18 Q2: No, there is not an expectation that proposal submissions must be marked CUI 5200.48 Controlled Unclassified Information. Please email Kallisti program security representative michael.perry.ctr@darpa.mil for assistance.

(U) P19. At the proposer’s day, we were informed that the abstract should be sent via encrypted email due to its CUI designation. However, the Program Solicitation (page 15) states that abstracts and proposals may be submitted through DARPA’s BAA website.

Could you clarify the preferred submission method?

(U) Answer: All abstract and proposal submissions must be submitted through DARPAs BAAT system. See updated program solicitation for details.

(U) P20. Can you clarify the level of detail required for the ROM in the abstract? The Program Solicitation (page 16) states, “Provide a ROM for the total cost of the proposed solution with minimal, high-level instantiations of said cost.” What specific breakdown or information is expected?

(U) Answer: Proposers may submit high level abstract ROM costing in the format and breakdown of their choice.

(U) P21 Q1. Are there any concerns with proposing an OTA-CPFF type contract?

(U) Q2. Given that DoD SAFE should not be used for orals proposal submission, will submission via https://baa.darpa.mil be allowed for large files?

(U) Q3. Can subcontractors separately submit their fully disclosed cost volumes?

(U) Q4. Can the program CUI guide be shared with teams invited to the orals proposal stage?

(U) Answer P21 Q1: An OTA is not a FAR-based contract, so comparisons to FAR contract types, such as FFP or CPFF, is not entirely accurate. If a proposal is deemed selectable, and a performer is invited to negotiate an OT with DARPA, a variety of payment schemes may be negotiated leveraging the inherent flexibility in OTAs.

(U) Answer P22 Q2: Please refer to sections 4.1 of the CUI version of the program solicitation.

(U) Answer P22 Q3: Yes.

(U) Answer P22 Q4: No, the program CUI guide will be provided to teams post award.

(U) P22. I was wondering if foreign nationals working for a US company can participate in the program.

(U) Answer: Foreign nationals working for a US company will have to be submitted for approval. Please see P23 Q2 answer below related to approvals of foreign nationals to work on Kallisti.

(U) P23 Q1. Typically, agreements (32 CRF 2002.4) flow down. We have not submitted or obtained for a SPRS score for CMMC 2.0 / NIST SP 800-171 compliance related to the Protection of Controlled Unclassified Information for the proposed Microsoft Government Cloud enclave. We have a cloud service provider, TACC, with a perfect score who can store-only CUI data. Q2. [In the case of a foreign national proposed to work Kallisti} “At what point would the exception process begin? Who initiates it? What if the paper is accepted but later the exception for the foreign national is denied? Q3. Even if a prime contractor attempted to “shield” a performer from CUI, please confirm whether the performer would be held responsible for the NIST SP 800-171 and other CUI compliance requirements. Q4.

Could the prime guarantee our researchers would not publish, store, transmit CUI and that prime would not transmit or cause the performer to store CUI? In other words, is there assurance performer researchers would only work on non-CUI portions of the project? Q5.

What is the DISTAR process for publishing?

(U) Answer P23 Q1: It is the responsibility of the performer to conduct a self-assessment in compliance with NIST Special Publication 800-171 and NIST Special Publication 800-171a.

(U) Answer P23 Q2: DARPA will initiate the process, but DARPA cannot guarantee an outcome.

(U) Answer P23 Q3: It is the responsibility of the prime performer to ensure all CUI is removed.

(U) Answer P23 Q4: This program is not Distro A, all potential publications will have to go through the DISTAR process. The performer will need to submit a DISTAR. It is the prime performers responsibility to ensure all CUI is removed.

(U) Answer P23 Q5: DISTAR is our process that DARPA use for public release. During this process, the case will be reviewed by security and policy reviewers. The case may either be approved as submitted, tentatively approved pending the additional amendments, or rejected due to content restrictions.

(U) P24. If a bibliography is included in the abstract containing references, will it be excluded from the page count?

(U) Answer: Yes, the bibliography is included in the abstract page count of 3 pages. Please refer to answer from P16 and sections 2.8 and 2.11 of the CUI program solicitation.

(U) P25 Q1. Milestone 10 indicates monthly report; do you anticipate performers submitting monthly reports and receiving monthly payments based on the payment amount of Appendix 4? Q2. Traditionally monthly reports may be CDRLs but not necessary align with payment milestones. Would there be a total of 40 payments on this program (30 monthly reports and 10 additional program milestones)? Q3. Milestone 8 indicates the ability to hold/access secret information. What artifacts are included in this milestone?

Would there be a corresponding payment associated with the completion of this milestone? Q4. Milestone 11 indicates submittal of a final report. This is listed as month 31, would this be considered outside the period of performance but have a corresponding payment?

(U) Answer P25 Q1: No. Please see appendix 1 of the CUI program solicitation for details of the Kallisti OT. See answer to P15.

(U) Answer P25 Q2: No. Please see appendix 1 of the CUI program solicitation for details of the Kallisti OT. See answer to P15 and P32.

(U) Answer P25 Q3: Performers will be required to provide appropriate artifacts for DARPA security to look up the status of a performer’s ability to hold/access classified information.

DARPA may sponsor performers for an FCL and work with DCSA to obtain an FCL in order to proceed to later stages of the Kallisti program.

(U) Answer P25 Q4: No. Please refer to section 2.7 of the CUI program solicitation.

(U) P26. BAA Appendix 4 defines the milestones against which payments are made.

Milestones are scheduled anywhere from 3-6 months apart, so payments will be irregular.

Some potential primes and subcontractors may not be able to carry project costs for such a long period of time, which could discourage bidding. May bidders propose additional monthly-based milestones so that payments will be more timely?

(U) Answer. Proposers are encouraged to submit proposals compliant with the program milestones and goals as provided in the CUI program solicitation. Proposers may propose additional milestones they feel are appropriate and in accordance with the overall goals of the Kallisti program. See answer to P15 and P32.

(U) P27. DARPA-PS-25-06 Kallisti require a 12-point Times New Roman font, however there is no mention of figures, tables, and charts. Are smaller fonts allowed if they are legible?

(U) Answer: No, all fonts must be no smaller than 12-point per section 3.2 of the CUI program solicitation.

(U) P28. Are Foreign Entities allowable to participate within this program if all procedures are met (e.g., export control, Safeguarding of CUI)? Section 4.2 seems to indicate yes, “foreign proposers are encouraged to understand U.S. export law and have a plan in place to obtain export licenses when necessary. Possible methods include teaming with a U.S.

prime and/or having a U.S. subsidiary/parent company.” And Section 2.9 (Eligibility) makes no mention of foreign entities.

(U) Answer: Please see Q22 and Q23 Q2.

(U) P29. Per the program solicitation (DARPA-PS-25-06), section 3.4 Oral Proposal Content, item d, Budget Estimation, the price breakdown should be loaded across major milestone events as a resource loaded IMS. Also required is Attachment 3, Cost Volume which indicates the contractor is responsible for providing a breakdown of cost for each task identified in the Task Description Document. Should proposers prepare to present cost details by milestone or by task?

(U) Answer: Proposers should present cost details by task that are clearly aligned to specific milestones.

(U) P30 Q1. The solicitation diƯers in how it describes the expected timing of some events/milestones. For example, the nominal 30-month program execution in Figure 1 shows Sprint 1 Midterm at t0 + 6 months while Table 2 indicates t0 + 8 months for Milestone

3. Is the expectation that the milestone dates per Table 2 / Schedule of Milestones are “no later than” deadlines? And Figure 1 is a potential schedule for a performer?

(U) Answer: Table 2 milestone #3 and Appendix 4 milestone #3 should indicate t0+6 months. Please see updated program solicitation and Appendix 4 for correction.

Performers are encouraged to deliver milestones as soon as they achieve them; dates in Table 2 are “no later than” deadlines. Figure 1 is provided as the nominal program execution schedule; proposers should propose the schedule they can achieve to meet program goals.

(U) P31. If a performer completes a sprint earlier than expected duration (e.g., meets Sprint 1 final criteria in t0 + 9), does sprint 2 duration eƯective extend until t0 + 18 months (9 months duration) or is sprint 2 expected to be completed by t0 + 15 months (6 months duration)?

(U) Answer: Sprint milestones are required to be met as “no-later-than" the dates in Figure

1. If a performer completes a sprint milestone early, that will not change the “no-later-than" date for the next sprint milestone.

(U) P32. Appendix 4 Schedule of Milestones and Payments provides DARPA Payment, Payment Criteria, and Exit Criteria-Eligibility for Next Milestone columns. Is the intent that Payment will be made for the Milestone based on the Payment Criteria? That eƯectively, the performer will be allocated funds to spend against the upcoming Milestone, paid upon the Payment Criteria for the Milestone, but not be able to be allocated funds for the next milestone until the Exit Criteria is met? For example, DARPA Payment for Milestone 2 (TE-0 System Integration) could/should eƯectively cover costs to meet Milestone 2 (TE-0 System Integration), would be allocated after Milestone 1 Exit Criteria is met, and paid upon Milestone 2 Payment Criteria? Put another way, performers will be allocated funds to achieve the next milestone, paid when the Payment Criteria for that milestone is reached, not upon meeting the exit criteria. The language in 1.4 seems to make this less clear, “Once the metrics and goals of the sprint are successfully met, the Government Team will evaluate the deliverables for success, pay the sprint midterm milestone, or pay the sprint completion milestone and authorize the performer to initiate the next sprint.”

(U) Answer: Final exit criteria will be clearly defined during award negotiations for selected proposals. These criteria will focus on verifying observable technical achievements that demonstrate achievement of the project's overarching goals. While specific metrics may be used as indicators, the ultimate determination of successful completion will be determined through a comprehensive assessment of the deliverable content and the Performer's contributions to the project’s critical path, as assessed by the DARPA Program Manager and/or Agreements OƯicer’s Representative (AOR). Please see P15 and P16. See updated Appendix 4.

(U) P33. Is it allowable or expected for performers to add additional milestones to the Schedule of Milestones and Payments in Appendix 4? The goal here would be to add more timely payments to avoid prohibitive durations between expenses incurred and paid (e.g., greater than 3 months).

(U) Answer: Proposers are encouraged to propose milestones per the Kallisti program per Table #2. See answer to P15, P32 and P26.

(U) P34. Is Figure 3 itself considered Unclassified or CUI? Figure 3 title is noted as (U) but there is no block label for the figure itself.

(U) Answer: Figure 3 is unclassified.

(U) P35. It is noted in the instructions: “(U) All Abstracts and Oral Proposals are anticipated to be unclassified.” Is it permissible to include CUI in either the written abstract and/or the materials for the oral presentation? If CUI content/markings are allowed and document(s) are encrypted, would DARPA prefer to have the access code(s) sent via email to this address?

(U) Answer: Proposers are encouraged to submit unclassified abstracts per the program solicitation but may submit CUI material as part of their written abstract and/or oral presentation materials in accordance with CUI handling instructions. All abstracts and proposals should be submitted to the DARPA BAAT system. See updated program solicitation.

(U) P36. Reading through the CUI solicitation please provide if the exit criteria is dictated by the Gov per sprint (discussed in section 2.1, page 10) OR by the proposer (discussed in section 3.4, page 20) which then presumably the Gov would adopt for evaluating the success of the performer in the sprint as exit criteria for payment?

(U) Answer: Please see P15.

(U) P37: Are service labs allowed to propose to Kallisti as a subcontractor?

(U) Answer: Please refer to eligibility requirements of the CUI program solicitation.

(U) P38. What deliverables will be considered the “algorithmic theory of mind deliverable” for the program? (end-to-end code, demo, final report?)

(U) Answer: The focus of the program is on generating an algorithmic theory of mind.

Deliverables should be software, reports, and other artifacts as required.

(U) P39. Are expenditure-base milestones acceptable on this OT?

(U) Answer: Please see P15, P32 and the CUI program solicitation on program milestones

(U) Proposers should submit a request for the CUI Kallisti Questions and Answers and CUI addendum amendment 1 at kallisti@darpa.mil.