Kallisti QA.pdf

(Unclassified)

(Unclassified)

(U) DARPA STO Kallisti Industry Day Submitted Questions and Answers

(U) Kallisti Environment

(U) E2. Who is developing the Kallisti M&A and Adjudicator Environment?

(U) Answer: DARPA has selected a government team from AFRL and other partners to develop the environment.

(U) E3. In phase 1 will there be real-world data provided? What data will be our first results being based upon?

(U) Answer: All data is simulated as real-world data is insu icient in volume to test generality of performer models. Initial data will be hosted on the Gov-Cloud instance.

Please refer to the CUI version of the program solicitation for details.

(U) E4. Will performers be able to expand the information available to performer solution from the simulator?

(U) Answer: DARPA had designed and specified the simulator environment to support performance goals of Kallisti.

(U) E5. How many actions & e ects to actions would exist in the simulation environment?

Are actions and reactions any way related to dots moving on a map? How bad would it be to apply brute force as a solution?

(U) Answer: DARPA anticipates that brute force approaches may not scale. Proposers are encouraged to propose any approaches they believe will best achieve program objectives.

(U) Kallisti Technical

(U) T1. Would numerical solutions as opposed to closed form (mostly) solutions be acceptable? Is explainability of results important? Would use of pre-trained large models be, ok? Some of the moves you described say for the Spratly islands are expressed in high level language. Will the simulation environment accommodate such high-level expressions?

(U) Answer: DARPA believes this is an open research question on the program. DARPA is open to a diversity of potential approaches the proposers feel can accomplish program objectives. For simulation environment questions see E1.

(U) T3: At the perception & comprehension stages is the forward model (black box) informed by human decision-making patters? Or are they simulated projection algorithms on arbitrary graphs? Is irrational behavior considered/modelled stochastically?

(U) Answer: DARPA will use simulated algorithms that are not fully representative of, but attempt to capture, human behavior. DARPA will explicitly model irrational behavior and performers are encouraged to propose their best approaches to solve this challenge.

(U) T4. Does Kallisti have an interest in deciphering human perception/cognition/action patterns of apparently irrational behavior?

(U) Answer: Yes, addressing irrational behavior is in scope and DARPA will introduce these constraints.

(U) T6. Is resilience to data poisoning value for establishing a Theory of Mind (TOM)?

(U) Answer: Data poisoning attacks are not a part of the Kallisti program objectives.

(U) T7. Is having an extensive human brain processing model based on high-resolution human subjects’ data a valuable asset of a proposal?

(U) Answer: Prior work in this area is potentially useful. Proposals should include scientific evidence and prior work for any proposed approaches. DARPA is looking for surprising results and novel solutions. Note that Human Subjects Testing (HST) is not authorized the Kallisti program.

(U) T9. How important is explaining power vs. predictive power of the submitted models?

How important is it for solutions to be explainable/interpretable vs. black box? How much of a priority will interpretability be?

(U) Answer: Explainability and interpretability are not necessary for Kallisti goals but are of interest to the program.

(U) T11. Will the testing and evaluation models produced by IDA capture and represent cultural properties that influence decisions? E.g. Will a model be clearly from [country A] and [country B]?

(U) Answer: Cultural properties are in scope for test and evaluation models but will be generalized in terms of “Greenlandia”, “Redlandia”, etc. Please refer to E1 and the CUI version of the program solicitation for details.

(U) T12. In FA1, are performers sending as input only the belief state that will result in the desired action? Or are they also providing a blue team plan to induce those beliefs?

(U) Answer: DARPA will initially be providing an Oracle function which returns next action from an adversary from a belief state during the sprint associated with FA1. In later Kallisti sprints DARPA will remove or reduce functionality of the black box Oracle.

(U) T13. Should the potential COAs in national security scenarios of Kallisti account for all aspects of DIME or DIMEFIL (Diplomacy, Information, Military, Economic, Financial, Law enforcement)?

(U) Answer: DIMEFIL is in scope for Kallisti, however DARPA anticipates use of a subset of DIMEFIL. DARPA is leveraging other DoD work in this space and will be incorporating best in class results for Kalllisti’s test and evaluation environment.

(U) T14. What is the scale (number of platforms, area covered, timescale) of the scenarios?

(U) Answer: DARPA doesn’t expect the scale to exceed 10,000 units unless needed for testing. The objective of Kallisti is develop the Theory of Mind as the focus to solve the problem and to test the limits of that hypothesis. Please refer to the CUI version of the program solicitation for additional details.

(U) T15. What is the richness of geopolitical relationships, action goals, intentions, etc.

expected in the scenarios that will inform platform behavior?

(U) Answer: A: Performer proposed approaches should not be sensitive to or dependent upon the backend. Geopolitical aspects will be part of the API and surrogate metrics for Kallisti. Please refer to the CUI version of the program solicitation for details.

(U) T16: Action state space: Should we model “continuous, high- dimensional” actions or would representing discrete actions (states) be su icient? If continuous – can you talk about how “beliefs” should be provided for evaluations (e.g. As parameterized probability densities or value functions?).

(U) Answer: One objective of Kallisti will be to evaluate the sensitivity of COAs to continuous and discrete states and action spaces. DARPA is interested in developing models that can solve continuous, discrete, and hybrid actions at the level of fidelity necessary to achieve compellence and deterrence.

(U) T17. Times and dynamics: Are actions taken in turns synchronously by blue and red? Do all actions have same durations? Or should we model “continuous” times (non-discrete, non-synchronous). Can you also talk about important of learning state “dynamics” vs rewards/interests).

(U) Answer: DARPA intends to provide a hybrid between continuous and discrete time.

(U) T18: Multi agent interactions: Is it important to represent how red agents interact/communicate to achieve their objectives, synchronize their beliefs, understand “each other”, decompose their group value / reward into individual values/polities?

(U) Answer: DARPA considers this one of the research questions to solve.

(U) T19. Training data: Can you tell us if training data will contain labels/rewards/red intent?

(U) Answer: DARPA has not decided on the level of labeling to provide in training data yet.

The goal is to include a realistic training data set for the performers.

(U) T20. Can you describe your vision for the state space and actions? Is modeling EW, asset detection, or ISR too fine of resolution?

(U) Answer: DARPA’s vision includes assets and non-physical components that will pull results from miliary and DIMEFIL sources (results of upcoming election, overflight rights, etc.). The state space will vary for each scenario and will be described in the API and scenario document provided to each performer.

(U) T21. To what extent should cognitive factors be reflected and incorporated in the adversarial theory of mind model?

(U) Answer: Performers are open to include all factors they deem necessary, including cognitive factors, when developing models to achieve overall program goals. Evaluation will be based on program metrics. Please refer to the CUI version of the program solicitation for details.

(U) T22. If an adversary is not purely rational but has a cohesive strategy, what constrains the adversary’s behavior?

(U) Answer: Irrational actors are in scope for Kallisti with the overall goal of developing the most e ective models for realistic adversaries that meet program metrics. Adversaries will be driven by commander’s intent defined by the test and evaluation team.

(U) Kallisti Programmatics

(U) P1. How will funding sequencing within each sprint be managed. E.g.: Upfront, incremental vs success on metrics? What are the time limits for each sprint? With milestone payment schedule, are performers operating “at risk” in direct competition with another? This seems to discourage risk taking.

(U) Answer: Kallisti will use standard milestone payments used by the Other Transaction agreement. Proposed approaches should reflect what each team believes is feasible to achieve Kallisti goals. Please refer to the CUI version of the program solicitation for details related to sprints, metrics, milestones and payment schedule details.

(U) P2. What will the size of each proposal be roughly. Or in other words, what is the size of the program? Are you planning to release program budget and/or number of performers? If so, what is planned?

(U) Answer: DARPA is anticipating multiple awards but does not release program budget information.

(U) P3: How does the program di er from the DARPA COMPASS Program?

(U) Answer: Kallisti will research the ability of AI technology to infer a theory of behavior model from observations. Please refer to the CUI version of the program solicitation for details.

(U) P4. Is there more than one presenter permitted for Abstract/Proposal Oral presentations?

(U) Answer: Yes, performers are permitted to have more than one presenter provided the performer’s presentation follows the guidelines of the solicitation.

(U) P5. Do you expect this to be multi-institutional collaborations, or can also one organization apply?

(U) Answer: DARPA encourages multi-disciplinary teams that may or may not include multi-institutional collaborations.

(U) P6. We’re quite a small non-traditional performer bringing a valuable solution from outside typical approaches – how do we best ‘project’ our alignment with the program despite not having established relationship in the existing community?

(U) Answer: DARPA encourages novel, multi-disciplinary approaches from non-traditional groups to address the challenges and goals of the Kallisti program. DARPA is interested in building a community around this research and development area with a di erent set of people with a wide range of ideas and performers. Existing relationships are not required for successful proposals.

(U) P7. For university performers, is there an option to partition work at (U) and/or fundamental research?

(U) Answer: This program will not be fundamental research and will not be initially authorized for public release. However, performers can submit through the DISTAR process and request public release of specific results.

(U) P8. Who will the end user of the models?

(U) Answer: Use of Kallisti models depends upon the success of the program and demonstrated capabilities by the performers.

(U) P9. Will there be any phases/sprints within Kallisti e orts that involve direct engagement with end user customers other than DARPA or other research partners? In other words, is the intent that Kallisti vendors remain attached and become part of the implementation process for successful projects?

(U) Answer: DARPA’s intent is to introduce performers to interface/interact with end users when appropriate. Please refer to the CUI version of the program solicitation for information on security. Please refer to DARPA Connect to understand transition opportunities.

(U) P10. What is the tech transition strategy to ensure this work finds applications?

Pentagon + Treasury were mentioned. What is AFRL’s Role? Other DOD o ramps?

(U) Answer: DARPA anticipates a significant potential for DoD, no-DoD transitions and other o ramps. World events may change transition priorities.

(U) P11. The CUI Addendum for the Program Solicitation currently indicates CTI, EXPT, and OPSEC CUI Category with Distribution Control of Distribution Statement D (Department of Defense and U.S. DoD contractors only). There is no NOFORN category listed. Q1: Does Distribution D limit the releasability to a foreign entity? Q2: Would extraction and sharing of only material with (U) Unclassified paragraph-level markings allow for distribution to foreign entities?

(U) Answer: P11 Q1: Yes, Distribution D does limit the releasability to foreign entities.

(U) Answer: P11 Q2: The performer could work with program security to put together an unclassified version of the document (DARPA will need to see it and concur before they send it). We want to emphasize, that it is the performer’s responsibility to follow export law and not release anything that would require a license to a foreign party.

(U) P12. Given the CUI status of much of the BAA, do you anticipate any issues with structuring a proposal so that university team members work on non-CUI parts of the project and a corporate member works on all the CUI restricted parts?

(U) Answer: DARPA does not have any issues with this as long as the Universities understand that this will not be Distro A, and they will have to go through our DISTAR process. DISTAR is DARPA's public release process.

(U) P13. Is the intention is still to allow open academic publication? (i.e., should I interpret the DISTAR requirement as a procedural delay in publication to ensure that no CUI information is released, or as a stronger signal that open academic publications are discouraged).

(U) Answer: DARPA instructions emphasize the desire to release all results at the broadest level of dissemination possible while maintaining concerns related to security. While we can never promise an individual result will be releasable to the public, the DISTAR process is a normal part of our procedures and is frequently and successfully used by performers.

Non-US persons generally cannot be key members of an e ort on a CUI program, but there is a process to submit requests for individual participation on defined portions of a project which will need to be reviewed on a case-by-case basis.

(U) P14. Q1. Will US Persons (permanent residents/green card holders) be eligible to work on Kallisti? Q2 Can a US person (green card holder, but not yet a citizen) be in a PI role in this program?

(U) Q1 Answer: Yes. Please see the DARPA CUI guidelines

(U) Q2 Answer: Yes. They can only work on Unclassified and CUI.

(U) P15. Reading through the CUI solicitation please provide if the exit criteria is dictated by the Gov per sprint (discussed in section 2.1, page 10) OR by the proposer (discussed in section 3.4, page 20) which then presumably the Gov would adopt for evaluating the success of the performer in the sprint as exit criteria for payment?

(U) Answer: Per section 3.4 the performer should propose milestones, payments and exit criteria for their proposed approach in accordance with the Kallisti program metrics and schedule of sections 2.1 and 2.2. If selected, upon satisfactory contract negotiations, the government will approve the proposed exit criteria. Please refer to sections 1.4 and 2.5 for additional details.

(U) P16: Just confirming that both the written and oral (slides) components of the abstract are due on February 28? May we make modifications to the slides for the oral presentation?

Are references included in the 3-page limit?

(U) Answer: Answer: Yes. both written and oral (slides) for abstract are due Feb 28. Please have a final copy of oral presentation submitted by Feb 28th. Yes, any references must be included in the 3-page limit.

(U) Proposers should submit a request for the Kallisti Questions and Answers CUI addendum at kallisti@darpa.mil.