HR001122S0039.pdf

Broad Agency Announcement

Assured Neuro Symbolic Learning and Reasoning (ANSR)

INFORMATION INNOVATION OFFICE

HR001122S0039

June 1, 2022

TABLE OF CONTENTS

PART I: OVERVIEW INFORMATION

PART II: FULL TEXT OF ANNOUNCEMENT

I. Funding Opportunity Description

A. Introduction and Background B. Program Description C. Technical Areas D. Program Phase and Schedule E. Metrics F. Deliverables G. Intellectual Property

II. Award Information A. General Award Information B. Fundamental Research

III. Eligibility Information A. Eligible Applicants B. Organizational Conflicts of Interest C. Cost Sharing/Matching D. Other Eligibility Criteria

IV. Application and Submission Information A. Address to Request Application Package B. Content and Form of Application Submission

V. Application Review Information A. Evaluation Criteria B. Review of Proposals

VI. Award Administration Information A. Selection Notices and Notifications B. Administrative and National Policy Requirements C. Reporting D. Electronic Systems E. DARPA Embedded Entrepreneur Initiative (EEI)

VII. Agency Contacts VIII. Other Information

IX. APPENDIX 1 – PROPOSAL SUMMARY SLIDE

PART I: OVERVIEW INFORMATION

Federal Agency Name – Defense Advanced Research Projects Agency (DARPA), Information Innovation Office (I2O) Funding Opportunity Title – Assured Neuro Symbolic Learning and Reasoning

(ANSR)

Announcement Type – Initial Announcement Funding Opportunity Number – HR001122S0039 Catalog of Federal Domestic Assistance Numbers (CFDA) – 12.910 Research and

Technology Development Dates o Posting Date: June1, 2022 o Proposers Day: June 1, 2022 o Abstract Due Date and Time: June 13, 2022, 12:00 noon (ET) o Questions Due: June 24, 2022, 12:00 noon (ET) o Proposal Due Date and Time: July 26, 2022, 12:00 noon (ET)

Anticipated individual awards – DARPA anticipates multiple awards in technical area (TA) 1. No more than two awards in each of TA2 and TA3, and one award in TA4

Types of instruments that may be awarded – Procurement Contract, Cooperative Agreement or Other Transaction

Agency contact o Points of Contact

The BAA Coordinator for this effort can be reached at:

Email: ANSR@darpa.mil

DARPA/I2O

ATTN: HR001122S0039

675 North Randolph Street Arlington, VA 22203-2114

PART II: FULL TEXT OF ANNOUNCEMENT

I. Funding Opportunity Description

This publication constitutes a Broad Agency Announcement (BAA) as contemplated in Federal Acquisition Regulation (FAR) 6.102(d)(2) and 35.016 and 2 C.F.R. § 200.203. Any resultant award negotiations will follow all pertinent laws and regulations, and any negotiations and/or awards for procurement contracts will use procedures under FAR 15.4, Contract Pricing, as specified in the BAA.

The Defense Advanced Research Projects Agency (DARPA) is soliciting innovative proposals in the following areas of interest: Artificial Intelligence (AI) algorithms and architectures that deeply integrate symbolic reasoning with data-driven machine learning to create robust, assured, and, therefore, trustworthy AI-based systems. Proposed research should investigate innovative approaches that enable revolutionary advances in science, devices, or systems. Specifically excluded is research that primarily results in evolutionary improvements to the existing state of practice.

A. Introduction and Background

Autonomy and highly autonomous systems are a desired capability for many Department of Defense (DoD) missions – Intelligence, Surveillance and Reconnaissance (ISR), Logistics, Planning, Command and Control among others. The purported benefits are many, including – (1) improved operational tempo and mission speeds; (2) reduced cognitive demands on warfighter in operation and supervision of autonomous systems; and (3) increased standoff for improved warfighter safety. A crucial desideratum associated with autonomy is the need for trustworthiness and trust, as emphasized by the 2016 Defense Science Board (DSB) Report on Autonomy1. Informally, trust is an expression of confidence in an autonomous system’s ability to perform an underspecified task. Assuring that autonomous systems will operate safely and perform as intended is integral to trust, which is key to DoD’s success in adoption of autonomy.

In the six years since the publication of the DSB report on Autonomy, significant improvements have been made in machine learning (ML) algorithms that are central to achieving autonomy.

Simultaneously, innovations in assurance technologies have delivered mechanisms to assess the correctness and safety trustworthiness of systems at design time and be resilient at operation time. In spite of this progress, high levels of autonomy remain elusive, which we attribute to fundamental limitations of data-driven ML (discussed below), motivating new thinking and approaches that will take ML beyond data-driven pattern recognition and augment it with knowledge-driven reasoning that includes context, physics, and other background information.

The last decade witnessed tremendous progress in applications of data-driven ML, fueled by growth in compute power and data, in areas that span a wide spectrum ranging from board games to protein folding, language translation to medical image analysis. In several of these

1 Report of the Defense Science Board Summer Study on Autonomy 2016:

https://irp.fas.org/agency/dod/dsb/autonomy-ss.pdf.

applications, ML and related techniques have demonstrated performance that rivals, and occasionally surpasses, human capability with respect to a set of narrowly curated metrics.

However, despite these apparent successes, there are a number of concerns associated with state-of-the-art (SOTA) ML algorithms. It is well known, for example, that SOTA ML algorithms do not generalize well2, lack transparency and interpretability, and are not robust to environmental3 and adversarial perturbations. Some of the limitations, such as a lack of robustness to adversarial examples, have been theoretically determined to be fundamental4 in nature.

The prevailing trend in industrial ML research is towards scaling up to Giga- and Tera- scale models (100’s of billions of parameters) as a means to improve accuracies and performance.

These trends are not sustainable because of the extremely high computational5 and data needs for training such models, as well as scaling laws6. These trends are also not responsive to the needs of DoD applications, which are typically data- and compute-starved with limited access to cloud-scale compute resources. Furthermore, DoD applications are safety and mission-critical, need to operate in unseen environments, need to be auditable, and need to be trustable by human operators. In sum, the prevailing trends in ML research are not conducive to the assurability and trustworthiness needs of DoD applications.

The traditional approaches to building intelligent applications and autonomous systems7,8 rely heavily on knowledge representations and symbolic reasoning. For example, complex decision-making in these approaches is often implemented with programmed condition-based rules, stateful logic encoded in finite state machines, and physics-based dynamics of environments and objects represented using ordinary differential equations. There are numerous advantages of these classical techniques:

they use rich abstractions that are grounded in domain theories and associated formalisms and that are supported by advanced tools and methods (Statecharts, Stateflow, Simulink, etc.);

they can be modular and composable in ways supported by software engineering practices that promote reuse, precision, and automated analyses; and they can be analyzable and assurable in ways supported by formal specification and verification technologies that have been demonstrated in hardening mission and safety critical systems against cyber attacks9.

2 Recht et al, “Do ImageNet Classifiers Generalize to ImageNet?,” arXiv, 2019, https://arxiv.org/abs/1902.10811.

3 Zhao et al, “Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey,” arXiv 2021, https://arxiv.org/abs/2009.13303.

4 Shafahi et al, “Are Adversarial Examples Inevitable?”, arXiv, 2020, https://arxiv.org/abs/1809.02104.

5 https://cset.georgetown.edu/wp-content/uploads/AI-and-Compute-How-Much-Longer-Can-Computing-Power- Drive-Artificial-Intelligence-Progress.pdf.

6 Thompson et al, “Deep Learning’s Diminishing Returns”, IEEE Spectrum, September 2021, Part of special report on The Great AI Reckoning - Deep learning has built a brave new world—but now the cracks are showing https://spectrum.ieee.org/deep-learning-computational-cost.

7 Schrage et al, “Software-enabled control for intelligent UAVs,” Proceedings of the 1999 IEEE International Symposium on Computer Aided Control System Design (Cat. No.99TH8404), 1999, pp. 528-532, doi:

10.1109/CACSD.1999.808703.

8 Venugopalan et al, “Autonomous landing of an Unmanned Aerial Vehicle on an autonomous marine vehicle,” 2012 Oceans, 2012, pp. 1-9, doi: 10.1109/OCEANS.2012.6404893.

9 Cofer et. Al, “Steal This Drone: DEF CON 29 Aerospace Village Activity,” http://loonwerks.com/publications/cofer2021defcon.html.

However, these approaches also have limitations when used in real-world autonomy applications. They fare poorly when dealing with real-world uncertainty and high dimensional sensory data, which is integral to perception and situation-understanding applications10, The rule-set and stateful logic in these decision-making applications are often incomplete and insufficient when exposed to unanticipated situations. Further, it is well understood that common-sense knowledge is intractable to codify. For example, the Cyc11 knowledge base includes millions of concepts and tens of millions of rules and yet is inadequate for many real-world tasks.

The challenge of assuring cyber physical systems (CPS) with ML components has been an active area of research supported by DARPA’s ongoing Assured Autonomy program as well as other research initiatives. Specifically, in Assured Autonomy, the assurance approach developed by the program has resulted in: (1) formal and simulation-based verification tools that can comprehensively explore the behavior of a CPS; (2) monitoring tools that can detect deviations of ML components from expected inputs and behavior; resilience and recovery strategies to avoid worst-case safety consequences; and (3) an assurance case framework that enables structured argumentation backed by evidence in support of the claim that major safety hazards have been identified and their root causes have been adequately mitigated.

The advances in assurance technologies, including formal and simulation-based approaches, have helped in accelerating identification of failure modes and defects of the ML algorithms.

Unfortunately, the ability to repair defects in SOTA ML remains limited to retraining12, which is not guaranteed to eliminate defects or to improve the generalizability of ML algorithms. Further, while the runtime assurance architecture – including monitoring and recovery – ensures operational safety, frequent invocations of fallback recovery – triggered by brittleness and generalizability of ML – compromises the ability to accomplish the mission.

The ANSR program seeks breakthrough innovations in the form of new hybrid AI algorithms that deeply integrate symbolic reasoning with data-driven learning to create robust, assured, and therefore trustworthy systems. We define a system as trustworthy, if it is: (a) robust to domain-informed and adversarial perturbations; (b) supported by an assurance framework that creates and analyzes heterogenous evidence towards safety and risk assessments; and (c) predictable with respect to some specification and models of "fitness."

We hypothesize that several of the limitations in ML today are a consequence of (1) the inability to incorporate contextual and background knowledge; and (2) treating each data set as an independent uncorrelated input. In the real-world, observations are often correlated and a product of an underlying causal mechanism, which can be modeled and understood. We posit that hybrid AI algorithms capable of acquiring and integrating symbolic knowledge and performing symbolic reasoning at scale, will deliver robust inference, generalize to new situations, and

10 Davies, Alex, “An Oral History of the DARPA Grand Challenge, the Grueling Robot Race That Launched the Self-Driving Car,” Wired Magazine, https://www.wired.com/story/darpa-grand-challenge-2004-oral-history/ 11 Lenat, D., “CYC: a large-scale investment in knowledge infrastructure,” Commun. ACM 38, 11 (Nov. 1995), 33–

38. https://doi.org/10.1145/219717.219745 12 Dreossi et al, “Counterexample-Guided Data Augmentation,” in 27th International Joint Conference on Artificial Intelligence (IJCAI), 2018.

provide evidence for assurance and trust.

We envision modifying both the training and inference procedures to interleave symbolic and neural representations for iterative inference and mutual adaptation of the representations to exploit the benefits and reduce the limitations of each representation. The modified training procedure will result in representations that are grounded in domain-specific symbols, essentially a symbolic equivalent of the Neural Network’s (NN) implicit data representation. The modified inference procedure iteratively converges to a response that is conformant to both the symbolic and neural representations. The symbolic representation can explicitly include prior knowledge and domain-specific rules and constraints and enables verification against specification and construction of assurance arguments.

Some recent results for specific applications provide the basis for confidence. For example, a recent study13 prototyped a hybrid reinforcement learning (RL) architecture that acquires a set of symbolic policies through data-driven learning. The symbolic policies are in the form of a small program that is interpretable and verifiable. The approach demonstrably inherits the best of both worlds: it learns policies that are highly performant in a known environment, and it generalizes well by remaining safe (crash-free) in an unknown environment. Another recent approach14 uses symbolic reasoning to fix errors in a NN in estimating the object-poses in a scene, and it achieves substantially higher (30-40% above baseline) accuracy in several cases.

The hybrid AI techniques developed by the program will enable new mission capabilities. The program intends to demonstrate assured execution of an unaided ISR mission to develop a Common Operating Picture (COP) of a highly dynamic dense urban environment. The autonomous system performing the ISR mission will carry an effects payload to reduce sensor-to-effects delivery time. While the delivery of effects is gated by human on-the-loop, an effects-carrying system is quintessentially a safety and mission-critical system and, therefore, requires strong guarantees of collision avoidance and mission performance. The capabilities required of the autonomous system in terms of deep situational understanding and decision-making are not achievable by SOTA machine learning or standalone symbolic reasoning systems. The training data is sparse, further motivating the use of hybrid AI methods.

B. Program Description

The overarching goal of the ANSR program is to advance hybrid AI algorithms and develop evidence-based techniques that support confident assurance judgments for these algorithms. The program intends to explore diverse hybrid architectures that can be seeded with prior knowledge, acquire both statistical and symbolic knowledge through learning, and adapt learned representations. The program intends to demonstrate and evaluate hybrid AI techniques through DoD mission-relevant use-cases where assurance and autonomy are mission-critical.

The program envisions a new take on representation learning and inference to lead towards hybrid AI. The SOTA ML, specifically NNs, can be viewed as learning a low-dimensional

13 “Neurosymbolic Reinforcement Learning with Formally Verified Exploration”, Anderson G. et al., NeurIPS, 2020.

14 “3DP3: 3D Scene Perception via Probabilistic Programming”, Gothoskar et al., arXiv, 2021, https://arxiv.org/abs/2111.00312 representation of a high-dimensional data set. Figure 1 provides a grossly oversimplified rendering of the training process. The gray hill is a depiction of an objective function (loss function) with respect to the parameters (or weights) of the NN. Each dot on the gray hill is a point in this parametric space and represents the value of the objective function with respect to the current parametric configuration of the NN.

Standalone neural machine learning entails a climb up the gradient to optimize the objective function. The optimal configuration represents a best fit for the training data. The representation, while being a good fit to the underlying training data, remains agnostic about the causality or the underlying mechanisms that produced the data. In the absence of any knowledge about the underlying mechanisms, inference tasks remain bound by the distribution of the training data and are unable to generalize beyond the training data distribution.

Figure 1: Neuro Symbolic Representation Learning

A variant on this training procedure may take domain knowledge into consideration and attempt to learn the representation of the data in terms of the domain primitives. The light blue cloud in the figure depicts the space of such domain primitives, which could be mathematical equations, symbolic programs composed from domain-specific primitives (components or functions), or other representations. The revised training procedure entails interleaving data-driven updates with a synthesis problem of finding a composition of domain-specific primitives that can reproduce the data. The learned symbolic representation can be combined with the neural representation and used to reassess the objective function. The iterative updates continue until reaching a fixpoint in the symbolic representation that, when combined with the neural representation, maximizes the objective function. The outcome of this training procedure is a hybrid neural and symbolic representation of the data that can be used for inference. Arguably, such a hybrid representation could combine the best of both worlds, i.e., data-driven learning and symbolic reasoning, with potential benefits of better performance in uncertain real-world situations, generalizability, interpretability, and assurability.

It is worth noting that the approach articulated above is only a notional approach towards learning hybrid neural and symbolic representations, and it is not intended as a prescription for a solution. The program anticipates many architectures – best suited for the specific application task – that combine neural and symbolic representations with different approaches to acquire, optimize, and use in inference the closely coupled neural and symbolic representations.

The development in the program will be orchestrated in four technical areas (TAs) summarized below:

TA1. Algorithms and Architecture – The goal of TA1 is to develop and model new AI algorithms and architectures that deeply integrate symbolic reasoning with data-driven machine learning. TA1 will explore and evaluate a range of possible algorithms and architecture patterns that are suitable for different tasks.

TA2. Specification and Assurance – The goal of TA2 is to develop an assurance framework and methods for deriving and integrating evidence of correctness and quantifying mission-specific risks. TA2 will establish a pipeline that abstracts the hybrid neuro-symbolic representations into formally analyzable representations and analyzes them with respect to a set of mission-dependent specifications. TA2 will also explore techniques to estimate and quantify mission-specific risks.

TA3. Platforms and Capability Demonstration – The goal of TA3 is to develop use-cases and an architecture for engineering mission-relevant applications of hybrid AI algorithms suitable for the demonstration and evaluation of robust and assured performance. Specifically, the program intends to pursue demonstration through assured execution of an unaided ISR mission to develop a Common Operating Picture (COP) of a highly dynamic dense urban environment.

TA4. Assurance Assays and Evaluation – The goal of TA4 is to 1) develop an assurance test harness with adversarial AI; and 2) evaluate the technologies in individual technical areas and their compositions in systems. TA4 will act as a red team that probes the validity of assurance claims through adversarial evaluations. TA4 will also refine the proposed program metrics and define measures to characterize the trustworthiness of the system. TA4 will need to assess robustness, generalizability, and assurance claims through adversarial evaluation that employs confounding perturbations and quantify the loss of system performance.

The guiding challenge for the program will be the assured execution of an unaided ISR mission in a highly dynamic dense urban environment. At present, ISR missions are conducted by warfighters either through forward presence, or through teleoperated ISR assets such as drones.

The ISR asset in these cases simply provides a video feed to the warfighter, who then has to process and analyze video feeds. The warfighter needs to distinguish adversaries from non-combatants, understand adversarial activities, analyze the scene to identify additional scan paths and focus areas, and maneuver the ISR asset to maximize stealth and safety. These are challenging activities that currently impose high cognitive burden on the warfighter and require them to continuously be in-the-loop. These activities cannot be realized with direct applications of SOTA ML, and they will require substantial use of context and background knowledge, interleaved reasoning with learning, and rigorous analyses and evidence to derive assurance claims about safety and mission success.

The program will seek to demonstrate a fully autonomous ISR mission with the warfighter only identifying the area of interest (AOI). The autonomous system provisioned with ANSR technologies will need to develop a comprehensive situation understanding that encompasses navigability and characterizes potential threat actors. The autonomous system will need to make maneuver decisions that maximize the situation understanding while maintaining safety. The system will autonomously and incrementally generate a comprehensive, timely, and accurate COP and deliver insights that help characterize friendly, adversary, and neutral entities, the operating environment, and threat and safe corridors. The program anticipates multiple demonstrations beginning with simulation-based experiments (SIMexp) conducted in multi-player role-playing gaming environments and concluding with live experiments at DoD facilities.

Phase 1 of the program will develop and prove out high-risk technology components – situation understanding, activity recognition, and safe maneuver decision making. The Phase 1 experiments will be multiple partial threads conducted in gaming environments and through SIMexp: (thread 1) demonstrate safe and assured decision making for maneuvers, while assuming perfect perception; (thread 2) demonstrate both activity recognition and situation understanding, while assuming human-guided safe maneuvers; and (thread 3) demonstrate COP development, insights, and analytics, while assuming perfect perception and human guided safe maneuvers. The evaluation will be performed against mission capability metrics as well as against SOTA baselines.

Phase 2 of the program will integrate these individual threads and demonstrate close-loop situation understanding, assured and safe maneuvers decision-making, COP building and analytics for end-to-end demonstration of unaided ISR missions. The evaluation will be done with respect to mission capability metrics (e.g., COP completeness, accuracy, and timeliness, scan efficiency, human effort required, sensor-to-effect timeline), and technology metrics (e.g., assurance validity, robustness, generalizability, and accuracy).

Phase 3 of the program will demonstrate an end-to-end ISR mission with live exercises in DoD facilities. The evaluation will include after-action reviews and soldier feedback in addition to the mission capability and technology metrics.

C. Technical Areas

The program has four technical areas (TAs):

TA1: Algorithms and Architectures TA2: Specification and Assurance TA3: Platforms and Capability Demonstration TA4: Assurance Assays and Evaluation

There are multiple points of essential collaboration among TAs, and the Government expects all performers to collaborate effectively. Proposers should read the descriptions of all TAs and the Program Assessments/Schedule section to ensure a full understanding of the program context, structure, and anticipated relationships required among performers. To facilitate the open exchange of information, all program performers will have Associate Contractor Agreement (ACA) language included in their award. See Section VIII for further information. Proposers may submit a single TA proposal or an integrated TA1 and TA2 proposal. No additional combination of proposals will be accepted.

TA1: Algorithms and Architectures The goal of TA1 is to develop and model new AI algorithms and architectures that deeply integrate symbolic reasoning with data-driven ML. The TA will explore and evaluate a range of possible architectures suitable for different components of the autonomy pipeline in an unaided ISR mission.

The applications for hybrid AI algorithms may include deep situational understanding, activity recognition, and developing safe and optimal maneuvers. Situational understanding in these applications may entail not just a labeling of the entities in a scene, but more complex attributions such as trajectories, capabilities, and even intent. The tasks are expected to require the fusion of diverse knowledge sources and modalities in addition to perceptual data alone.

The deep integration of interleaved symbolic and NN elements at multiple scales should enable vastly superior performance in terms of accuracy, robustness, and generalizability over the SOTA of either type of methods - symbolic alone or data-driven alone – in applications of interest in this BAA such as perception, planning, and control. The feedback between symbolic and neural components should enable performance improvements at all scales and quantities of interest, for example, in perception, from entity identification and resolution to more complex situational understanding, as well as in timeliness for decision making in closed-loop systems such as real-time control.

It is desirable that the hybrid AI algorithms be generalizable to diverse symbolic domains both for incorporating available prior knowledge as well as for being suitable for inter-working with the domain tools for assurance reasoning (see TA2). The underlying knowledge representation for the application domains of interest may include logic systems (e.g., first-order logic, linear temporal logic), dynamical systems (e.g., differential equations), finite state machines, graphs, visual concepts, etc.

The program seeks hybrid AI algorithms that are application-specific (algorithms for perception, algorithms for planning, algorithms for control, etc.), as well as architectures in which many such algorithms could be composed in a federated or hierarchical configuration. TA1 proposers should present approaches for monitoring and regulating individual hybrid AI algorithms in such a federated architecture.

Research challenges in TA1 include, but are not limited to: (a) symbolic knowledge extraction from latent representation of NN so that such extracted knowledge could be pooled with other available symbolic knowledge to support reasoning to derive context and for adaptation; (b) cueing the NN for continual adaptation (parameters, architectures) based on contextual knowledge for hypothesis resolution and better performance; and (c) integrated and iterative reasoning over symbolic and neural representations.

These challenges are important and illustrative, but are not exhaustive, so proposals should clearly articulate other challenges considered in the proposal and the methods to address them.

The BAA considers the deep integration of symbolic reasoning with data-driven ML sought here to be a fertile ground for new ideas. Proposals may bring to bear multiple methods (for example, imitation learning and mirror descent, hierarchical predictive processing, generative models, neural network adaptation, etc.) or propose entirely new approaches. In all cases, the proposals should explain, presenting any preliminary results as evidence, how the proposed approaches are likely to be successful in terms of the desired algorithmic capabilities and the ability to meet the program metrics.

TA2: Specification and Assurance The goal of TA2 is to develop an assurance framework as well as methods for deriving and integrating evidence of correctness and methods for quantifying mission-specific risks. Given a deeply integrated neuro-symbolic algorithm trained for a certain set of specifications, the goal of this TA is to ‘assure’ its workings, i.e., formally establish that its function and behavior satisfies the specifications, some of which are dynamic and evolving based on mission needs.

Additionally, when the entire behavior space could not be analyzed a priori, which is likely to be the case for these systems, the goal of the TA is to provide a characterization of risk to inform system operators.

The assurance challenges for the ANSR program include dealing with (a) composed hybrid neural and heterogeneous symbolic representations; (b) dynamic mission-dependent specifications; and (c) the integration of heterogeneous evidence to satisfy assurance claims and quantify residual risk. The methods should be applicable to different symbolic representations such as state machines or differential equations and across applications such as perception, planning, and control.

The algorithmic technical challenges include: (a) mapping the composed hybrid neural and symbolic representations into representations suitable for formal analysis, and (b) developing methods for analysis to find proofs through evidence and claims and to characterize ‘assured regions’ in the behavior space as well as ‘regions of no-assurance’ for given specifications. With respect to assurance, a fundamental difference between ANSR and assurance in traditional systems is the inherent uncertainty involved in learning systems, which has to be considered by the algorithms, along with possible effects of feedback loops in deep neuro-symbolic integration.

The proposed methods should ensure that the ‘assured region’ so derived is maximal relative to the ‘expected mission region’ implied by the specifications. Proposals should also address methods for analyzing the fundamental tradeoffs involved between the specifications (introducing relaxation/restrictions) and the ‘assured region’ (expansion/contraction), and they should develop methods to change the specification and/or the neuro-symbolic parameters to maximize the assurance envelop. Further, proposals should propose methods to quantify the risk involved in regions of no-assurance and methods to minimize the risk.

The BAA considers the assurance methods for deeply-integrated symbolic reasoning and data-driven machine learning sought here to be a fertile ground for new ideas. Proposals may bring to bear a combination of multiple methods (for example, probabilistic and hybrid model checking, dynamic assurance case technology, runtime risk assessment, etc.) or propose entirely new approaches. In all cases, the proposals should describe, presenting any preliminary results, how the proposed approaches are likely to be successful in terms of the assurance capabilities desired here and the ability to meet the program metrics.

TA3: Platforms and Capability Demonstration The goal of TA3 is to develop use-cases and an architecture for engineering mission-relevant applications of hybrid AI algorithms for the demonstration and evaluation of robust and assured performance. Specifically, the program intends to pursue demonstration through assured execution of an unaided ISR mission to develop a COP of a highly dynamic dense urban environment. The demonstration will entail fully autonomous operation, with the warfighter identifying an AOI to an autonomous system. The autonomous system will be provisioned with ANSR technologies. It will autonomously and incrementally generate a comprehensive, timely, and accurate COP and deliver insights that help characterize friendly, adversarial and neutral entities, the operating environment, and threat and safety corridors.

Multiple demonstrations are anticipated, beginning with simulation-based experiments (e.g., SIMexp) using multi-player role-playing gaming environments, while program capabilities are being developed and concluding with live experiments at a DoD facility.

TA3 proposers should propose an autonomy platform on which ANSR technologies could be deployed. The platform should be provisioned with a baseline autonomy stack that enables autonomous navigation using a suite of perceptual sensors. TA3 proposers will need to develop challenge problems and tasks for TA1 performers. TA3 proposers will also need to provide the background knowledge and datasets necessary for developing the hybrid AI algorithms for these challenge problems – which may include, but are not limited to, deep situational understanding, activity recognition, maneuver planning, and COP construction and analytics – capabilities integral to executing an unaided ISR mission. Further, each TA3 proposed platform should be able to support simulation-based experiments and as well as be used in eventual live experiments. Simulation-based experiments would require an embodiment of the platform including the necessary sensors and actuators, high-fidelity dynamics, and an encapsulation of the baseline autonomy stack in a simulation environment such as Gazebo or Airsim.

The mission and technology challenges for such an integrated platform include, but are not limited to the abilities to: (a) generate comprehensive, timely, and accurate COP; (b) demonstrate generalizability and adaptability to different urban environments and scenarios; (c) provide activity and scene understanding with different sensory data; (d) ingest and reason with symbolic knowledge, for example, maps, entity dynamics, mission specification, and rules of engagement;

and (e) provide assurance and quantify risk to mission performance to establish trustworthiness.

The TA3 performers should work closely and support TA1 and TA2 performers to address these mission and technology challenges.

TA4: Assurance Assays and Evaluation The goal of TA4 is to (a) develop an assurance test harness that includes adversarial AI, and (b) evaluate ANSR technologies both in individual technical areas and in their compositions into complete systems, especially for the use case described in TA3. TA4 proposals should address both of these goals. TA4 proposals should clearly describe an approach to achieving these goals and articulate the unique benefits of the proposed approach in comparison to other possible approaches.

Assurance test harness with adversarial AI: TA4 will act as a red team that probes the validity of assurance claims and assurance case through adversarial evaluations. Proposals must present the approaches to generate adversarial methods for anticipated hybrid-AI algorithms (to be developed by TA1) and methods for spoofing the ML, also considering the full range of the ‘expected mission region’ (see TA2 description). Proposal must present a technical rationale and any preliminary evidence they have for the potential success of their approach.

Evaluate individual technologies and composed systems: The program seeks to characterize the performance of individual hybrid AI components with respect to accuracy metrics using benchmarks and the performance of composed systems using mission-specific metrics. TA4 proposals must present approaches to (a) develop benchmarks for individual technologies in the domains of perception, planning, and control; (b) develop and refine mission-specific metrics for the use case of unaided dense-urban ISR described in the TA3 section; and (c) develop a test harness to characterize the performance of said systems in all program phases including field experiments.

TA4 will also refine the proposed program metrics (listed in Section E), identify baselines, and evaluate the program developed technologies (encompassing TA1, TA2, and TA3) with respect to the proposed metrics. Trustworthiness is challenging to quantify, and measurement will be a topic of research for TA4 performers. The program seeks to characterize trustworthiness along two dimensions: (1) ability to operate with acceptable performance even when diverging from the planned scenario; and (2) transparency and predictability in the behavior of the system obtained by assurance argumentation and evidence. TA4 proposals should present in detail the Test and Evaluation (T&E) approaches for these two dimensions. They are further encouraged to present other compelling ways to perform T&E of the use case described in the TA3 section.

D. Program Phase and Schedule The ANSR program is organized in three phases. Phase 1 will be eighteen (18) months and will consist of Prototype Hybrid AI Algorithms and Mission Threads. Phase 2 will be fifteen (15) months and will focus on System Composition and Mission Assurance. Phase 3 will also be fifteen (15) months and will focus on Live Demonstration and Transition. Each phase will end with demonstrations on the selected challenge problems, which will provide an opportunity to evaluate the progress made against the program objectives (See Figure 2). Proposers should submit a detailed schedule of logically sequenced tasks and subtasks that in sum constitute a constructive plan for achieving the proposed technical objectives while appropriately managing risk. Schedules will be synchronized across performers, as required, and monitored and reviewed throughout the ANSR program’s period of performance. For budgeting purposes, use January 5, 2023, as a start date for all TAs.

Figure 2: Schedule

The Government will specify the locations for Principal Investigator (PI) meetings during program performance. There will be two PI meetings in Phase 1, held approximately 6 months and 12 months after the kick-off meeting. There will be one PI meeting in both Phase 2 and Phase 3, held roughly 7.5 months after the beginning of each phase. Each phase will end with a demonstration workshop. PI meeting locations are likely to be spread across performer locations, and the proposers should plan to host at least one PI meeting over the duration of the program.

The goals of the PI meetings will be to present new research findings and accomplishments, review plans for the next period, discuss implementation milestones, and resolve any programmatic, budgeting, or logistics issues.

In addition to these program-wide events, the Government team will conduct site visits and will hold monthly teleconference meetings with each PI to enhance communications with the Government team.

For travel planning and costing, assume eight (8) trips during the entire three (3) phases (2022- 2026) per the program schedule shown above, alternating between Washington, DC and San Diego, CA, with each trip requiring 3-days and 2-nights.

E. Metrics For the Government to evaluate the effectiveness of a proposed solution in achieving the stated program objectives, proposers should note that the Government hereby promulgates the following program metrics (Table 1 below) that may serve as the basis for determining whether satisfactory progress is being made to warrant continued funding of the program. Although the following program metrics are specified, proposers should also note that the Government has identified these goals with the intention of bounding the scope of effort, while affording the maximum flexibility, creativity, and innovation in proposing solutions to the stated problem.

Trustworthiness is challenging to quantify, and measurement remains a topic of research on its own. Consistent with the definition offered earlier, the program will characterize trustworthiness along two dimensions: (1) ability to operate with acceptable performance even when diverging from the planned scenario; and (2) transparency and predictability in the behavior of the system obtained by assurance argumentation and evidence. The program will characterize performance of individual hybrid AI components with respect to accuracy metrics using benchmarks and performance of composed systems using mission-specific metrics (e.g., completeness and accuracy of COP), and target an order of magnitude improvement over current baselines. The program will assess robustness, generalizability, and assurance claims by an adversarial evaluation (akin to red teaming for cyber assurance) that employs semantic variations (for example changing movement and types of entities in a scene) from baseline (training) scenarios and quantify the loss of system performance (target degradation below 5%).

Table 1: Program Metrics

1Includes object classification, geolocation, and activity recognition (from a defined set).

2SOTA COPter (name given to the ISR platform for developing COP) will be teleoperation with SOTA ML. Training data will be from exercises and synthetic. The projected SOTA accuracy for object recognition from video/images is 60% at moderate size training data; SOTA activity recognition accuracy is only 30-40% or less in 2020 Activity-Net benchmarks (https://aiindex.stanford.edu/wp-content/uploads/2021/03/2021-AI- Index-Report-_Chapter-2.pdf.).

F. Deliverables All performers will be required to provide the following deliverables:

Technical papers covering work funded by the ANSR program;

Source code, build scripts, and any toolchains required to compile code, algorithms and interface description documents, user guides, other necessary data, and documentation, assumptions, and limitations for all software developed under this program;

Slide Presentations. Annotated slide presentations shall be submitted within one month after the program kick-off meeting and after each program event (program reviews, PI meetings, and technical interchange meetings);

Quarterly Progress Reports. A quarterly progress report describing technical progress made, resources expended, major risks, planned activities, trip summaries, changes to key personnel, and any potential issues or problem areas that require the attention of the Government team shall be provided within 15 days after the end of each quarter;

Monthly Progress Reports. A monthly progress report in the form of a PowerPoint document describing technical progress, planned activities for the next month, and any https://aiindex.stanford.edu/wp-content/uploads/2021/03/2021-AI-Index-Report-_Chapter-2.pdf https://aiindex.stanford.edu/wp-content/uploads/2021/03/2021-AI-Index-Report-_Chapter-2.pdf technical, financial, and programmatic issues shall be provided and presented in a teleconference with DARPA;

Monthly financial status reports;

A final phase report after each program phase. The final phase report shall concisely summarize the effort conducted within that phase; and Final Technical Report.

G. Intellectual Property The program will emphasize creating and leveraging open source technology and architectures.

Intellectual property rights asserted by proposers are strongly encouraged to be aligned with open source regimes.

A key goal of the program is to establish an open architecture and algorithms that allow for integrating symbolic knowledge and domain-specific representations with data-driven machine learning. This goal includes the ability to easily add, remove, substitute, and modify the type of symbolic representation as well the type of data. This capability will facilitate rapid innovation by providing a base for future users or developers of program technologies and deliverables to build upon and rapidly customize and reuse for different AI applications in both DoD and non- DoD use-cases. Therefore, it is desired that all noncommercial software (including source code), software documentation, hardware designs and documentation, and technical data generated under the program be provided as deliverables to the Government, with a minimum of Government Purpose Rights (GPR), as lesser rights may adversely impact the lifecycle costs of affected items, components, or processes.

II. Award Information

A. General Award Information

Multiple awards are anticipated. The amount of resources made available under this BAA will depend on the quality of the proposals received and the availability of funds.

The Government reserves the right to select for negotiation all, some, one, or none of the proposals received in response to this solicitation and to make awards without discussions with proposers. The Government also reserves the right to conduct discussions if it is later determined to be necessary. Additionally, DARPA reserves the right to accept proposals in their entirety or to select only portions of proposals for award. In the event that DARPA desires to award only portions of a proposal, negotiations may be opened with that proposer. The Government reserves the right to fund proposals in phases with options for continued work, as applicable.

The Government reserves the right to request any additional, necessary documentation once it makes the award instrument determination. Such additional information may include but is not limited to Representations and Certifications (see Section VI.B.2., “Representations and Certifications”). The Government reserves the right to remove proposers from award consideration should the parties fail to reach agreement on award terms, conditions, and/or cost/price within a reasonable time, and the proposer fails to timely provide requested additional information. Proposals identified for negotiation may result in a procurement contract, grant, cooperative agreement, or other transaction, depending upon the nature of the work proposed, the required degree of interaction between parties, whether or not the research is classified as Fundamental Research, and other factors.

Proposers looking for innovative, commercial-like contractual arrangements are encouraged to consider requesting Other Transactions. To understand the flexibility and options associated with Other Transactions, consult http://www.darpa.mil/work-with-us/contract-management#OtherTransactions.

In accordance with 10 U.S.C. § 4003(f), the Government may award a follow-on production contract or Other Transaction (OT) for any OT awarded under this solicitation if: (1) that participant in the OT, or a recognized successor in interest to the OT, successfully completed the entire prototype project provided for in the OT, as modified; and (2) the OT provides for the award of a follow-on production contract or OT to the participant, or a recognized successor in interest to the OT.

In all cases, the Government contracting officer shall have sole discretion to select award instrument type, regardless of instrument type proposed, and to negotiate all instrument terms and conditions with selectees. DARPA will apply publication or other restrictions, as necessary, if it determines that the research resulting from the proposed effort will present a high likelihood of disclosing performance characteristics of military systems or manufacturing technologies that are unique and critical to defense. Any award resulting from such a determination will include a requirement for DARPA permission before publishing any information or results on the program. For more information on publication restrictions, see the section below on Fundamental Research

B. Fundamental Research

It is DoD policy that the publication of products of fundamental research will remain unrestricted to the maximum extent possible. National Security Decision Directive (NSDD) 189 defines fundamental research as follows:

‘Fundamental research’ means basic and applied research in science and engineering, the results of which ordinarily are published and shared broadly within the scientific community, as distinguished from proprietary research and from industrial development, design, production, and product utilization, the results of which ordinarily are restricted for proprietary or national security reasons.

As of the date of publication of this solicitation, the Government expects that program goals as described herein may be met by proposed efforts for fundamental research and non-fundamental research. Some proposed research may present a high likelihood of disclosing performance characteristics of military systems or manufacturing technologies that are unique and critical to defense. Based on the anticipated type of proposer (e.g., university or industry) and the nature of the solicited work, the Government expects that some awards will include restrictions on the resultant research that will require the awardee to seek DARPA permission before publishing any information or results relative to the program.

http://www.darpa.mil/work-with-us/contract-management#OtherTransactions

University or non-profit research institution performance under this solicitation may include effort categorized as fundamental research. In addition to Government support for free and open scientific exchanges and dissemination of research results in a broad and unrestricted manner, the academic or non-profit research performer or recipient, regardless of tier, acknowledges that such research may have implications that are important to U.S.