DARPA-PA-18-02-02.pdf

Artificial Intelligence Exploration (AIE) Opportunity

DARPA-PA-18-02-02

Artificial Intelligence Research Associate (AIRA)

I. Opportunity Description

The Defense Advanced Research Projects Agency (DARPA) Defense Sciences Office (DSO) is announcing a new Artificial Intelligence Exploration (AIE) Opportunity that is focused on elevating Artificial Intelligence (AI) to the role of an insightful and trusted collaborator in scientific discovery and the scientific process. This AIE Opportunity announces the Artificial Intelligence Research Associate (AIRA) program, and invites submissions of innovative basic research proposals to address two main objectives:

1) explore and develop novel new AI algorithms and approaches for discovery of scientific laws and governing equations for complex physical phenomena; 2) explore new approaches to assess where data are too sparse, noisy, or are otherwise inadequate to build predictive models; to generate testable hypotheses; to identify high value experiments that could alleviate the problems of data shortfalls; and to quantify the confidence of predictions outside of the training space.

This AIE Opportunity is part of the DARPA AIE thrust that aims to advance the state-of-the-art in AI through both standard programs and relatively short-term technology development projects that quickly test and validate new AI concepts. It is issued under the Program Announcement DARPA-PA-18-

02. All proposals in response to the technical area(s) described herein will be submitted to DARPA-PA- 18-02 and, if selected, will result in an award of an Other Transaction (OT) for prototype project agreement for a proof-of-concept project not to exceed $1,000,000.

A. Introduction

For more than five decades, DARPA has been a leader in spurring groundbreaking research and development (R&D) that facilitated the advancement and application of “First Wave” (rule based) and “Second Wave” (statistical learning based) AI technologies. With its AIE program, DARPA seeks to develop “third wave” AI theory and applications that address current limitations by enabling machines to contextually incorporate available data, facts, models, heuristics, and additional information to achieve greater robustness, adaptability and generalizability.

While AI and deep learning approaches in particular have recently proven very effective in certain application domains, AI still falls short of being a trusted and valued collaborator in scientific discovery and technology development. A key challenge is how well statistical AI methodologies can generalize beyond the narrow set of questions they are initially trained on. As AI and deep learning are increasingly being applied to predict the behavior of complex nonlinear dynamic systems where the system state is not fully observable and the data lack coverage, this question of generalizability becomes even more important. Generalizability in today’s state-of-the-art approaches remains poorly understood and quantified. In fact, current AI systems are in general incapable of “understanding” whether the questions they have been trained on can be usefully answered, or whether they even make sense given the provided set of input data.

The objective of the AIRA program is to explore new approaches for enabling AI to be a more valued partner in scientific discovery and the scientific process. Here, the scientific process refers to an iterative loop incorporating the components of observation, analyses and questioning, development of testable hypotheses and predictions, experimental design, and theory development.

Today’s AI, if incorporated at all in the process, is applied in yes-no and regression types of analyses of big data. Human scientists handle most of the knowledge-centric aspects of the process, often relying on human experience, heuristics, ad hoc and domain-specific methodologies. Assessing the adequacy of the data, formulating questions, generating hypotheses and testable predictions, identifying additional high value experiments that could alleviate data shortfalls, and quantifying confidence of predictions outside of the training space all lie outside the capabilities of most state-of-the-art AI approaches and fall into the domain of human experts. As science attempts to understand ever more complex systems, and as datasets become larger and more difficult to process, this process is challenged on multiple fronts.

Overcoming these limitations will require new AI architectures and methods to generate “deep insights” based on physics-, math-, and context-aware representations that succinctly capture relevant dynamics and behaviors of the complex systems under study and can generalize across domains. It will also require new capabilities to assess both the power and the limitations of the provided data to answer both the training questions and a broader set of questions pertaining to the system behaviors, stability, controllability, robustness, and parametric dependencies.

B. Objective/Scope

Under the AIRA program, DARPA seeks to address the challenge of embedding AI into physics, and more broadly into science and the scientific process. In this context, “embedding AI into physics” refers to the use of AI technologies, in collaboration with scientists, to facilitate the discovery of physical laws with parsimonious mathematical representation, governing complex physical phenomena (which may involve multiscale spatiotemporal dynamic processes, phase transitions, and chaotic behavior). It also refers to the use of AI to assist scientists in the scientific process as described above. It is anticipated that multidisciplinary teams incorporating domain expertise, together with some combination of AI, mathematics, statistics, information theory, physics, control theory and/or other disciplines, will be required to achieve the transformational goals of the program.

As motivation for the AIRA vision, it is useful to consider an illustrative example. Suppose one has been given Tycho Brahe’s data (along with some supplemental data) mapping the trajectories of planets in the solar system as seen from an observatory on earth. The trajectories, although complicated, clearly correspond to reduced dimensional manifolds. Here we pose some rhetorical questions to challenge the community to think about the potential value-add and capabilities of an AI Research Associate. Could, for example, an AI Research Associate accomplish or assist in the following:

Recognize that the coordinate system of the observed data is non-ideal and that a better coordinate system might lie two or three unitary transformations removed?

Identify areas where data are inadequate, e.g. due to ambiguities, gaps, inadequate coverage, inadequate resolution, inadequate signal-to-noise, insufficient modalities, confounding variables, measurement errors, or outlier results, and provide guidance for obtaining the highest-value augmenting data?

Arrive at a parsimonious, predictive model (e.g. uncover Kepler’s laws)?

Arrive at a parsimonious, predictive model with the potential to generalize across multiple domains (e.g. uncover Newton’s laws and a 1/r gravitational potential model for gravity)?

Deduce conservation of energy, momentum and angular momentum?

Propose testable experiments to validate hypotheses and new models (e.g. predict the next return of Halley’s comet)?

Uncover a problem with Mercury’s orbit and deduce a better abstraction such as Einstein’s general relativity?

Test for consistency across multiple datasets and observations?

As a second, contemporary real world example, one may consider multiscale structured materials.

Dense Functional Theory and other ab initio modeling methods at the primitive cell level are computationally intensive and don’t scale well beyond a small number of atoms. Modeling materials properties across scales generally requires a hand-tweaked stitching together of a patchwork of different models at different scales. What happens if one changes constituents or adds different dopants? How do defects and microstructures affect properties? Could AI derive deep insights from existing experimental and model data obtained for similar material structures? Could it generate useful multiscale representations and models that generalize gracefully across different constituents? Could it guide the discovery of new materials with desired properties? And importantly, could it provide feedback regarding the adequacy of the data, where additional data would help, and how confident the predictions are in regimes not covered in the training database?

Humans, while capable of generating deep insights, lack the bandwidth to process the high throughput data of complex, high dimensional systems. For AI to collaboratively support humans, AI technologies must be able to process and distill big data into physically-relevant representations comprehensible and suitable for collaboration. AIRA proposers should bear this in mind in crafting responsive proposals that address the AIRA research project objectives described below for each of the AIRA Technical Areas.

Proposers should note the focus of this AIE Opportunity on developing innovative and principled math-, physics-, and information-driven approaches for deep discovery. Literature mining, interactive learning, ontology frameworks, human-machine interfaces, learning automation and integration of such tools, while all potentially useful additions to broadening the role of AI in the scientific process, will be considered non-responsive.

Technical Area 1 (TA1): Explore and develop prototype, proof-of-concept AI architectures, algorithms and approaches that can uncover natural laws and succinct, generalizable, coarse-grained governing equations from high dimensional data in a principled way. AIRA is especially interested in generative math- and physics-driven AI approaches that can learn, model, and predict the behavior of complex nonlinear dynamic systems in one the following domains: solid state physics, condensed matter or material science; turbulence, plasma, atmospheric or ocean fluid dynamics; circadian- or cell-cycle-driven cellular biology and neuroscience; complex collectives of autonomous and semi-autonomous agents. Models should capture enough of the underlying physics in parsimonious, physically-grounded representations that they can extend predictions beyond the original training datasets and even generalize across domains. Initially, performers may make use of “toy” or exemplar models from simulations or simple experimental systems where the behaviors and answers are generally already known as a useful starting point to test architectures and methodologies, but proposers must begin to apply their methods to real-world systems by the second phase of the program. Proposers should discuss how their models might deal with phase transitions and chaotic behavior, and if their models can handle non-stationary and non-ergodic processes. Proposals should also discuss the types of questions their generative models could potentially answer beyond the original training questions.

Technical Area 2 (TA2): Explore and test novel approaches to identify regions of input space or parameter space where datasets are sparse, noisy, lack coverage, lack consistency, or are otherwise inadequate to build predictive models. Methods should be informative in closing the loop of scientific discovery by identifying high value experiments, generating testable predictions, and assessing the impact of new data. New approaches using AI to augment sparse or noisy data, or fill in gaps, using prior knowledge, physical or mathematical constraints (e.g. symmetries and conservation laws), or orthogonal data modalities, are encouraged. Approaches to identify approaching system instabilities or imminent phase transitions in a given parameter space are also encouraged.

Technical Area 3 (TA3): Based upon TA1 and TA2 results, refine prototype AI system, algorithms and approaches and evaluate performance against one or more complex and dynamic DoD-relevant system(s) as described in TA1, using real-world data not part of the training data. Proposed testing should address the following performance criteria in comparison with current state-of-the-art machine learning systems:

1. How will you assess both accuracy and generalizability of your models? How well does the system predict evolutionary trajectories, dependencies on input or control parameters, and system response to perturbations?

2. Can your models find and resolve inconsistencies and ambiguities, either in the data or in model predictions? Can they answer new questions of the system for which they haven’t been trained?

Can you quantify or provide bounds for model performance for both in-training and out-of-training input space?

3. Can you quantify the extent to which additional data, higher resolution data or new data modalities can help, and can you identify high value experiments to better inform and constrain your models?

C. Structure

AIRA is seeking innovative approaches that address TA1, TA2, and TA3 objectives (as described in Section B) to substantially improve upon current data-driven and AI approaches in bringing deep insight into the scientific process.

AIRA is structured as an aggressive eighteen (18) month program. During the initial nine (9) month Phase I base effort, performers must address both TA1 and TA2 objectives for exploring new prototype AI architectures, algorithms and approaches that have the potential to meet the broad goals of the program. These systems will be tested in well understood “toy” or “model” systems using either simulated or real data. Initial methods will be developed to assess “accuracy,” “robustness,” and “generalizability,” and novel approaches will be validated against current baseline machine learning methods.

In the following nine (9) month Phase II option effort, performers must address TA3 objectives for extending and refining their prototype approach, quantifying performance in DoD-relevant, real-world application domains, and comparing with existing state-of-the-art machine learning methods.

Proposals submitted to DARPA-PA-18-02 in response to the technical areas of this AIE Opportunity must be UNCLASSIFIED and must address all three Technical Areas in two independent and sequential project phases, Phase I base (Technical Areas 1 and 2) and Phase II option (Technical Area 3). Failure to do so may cause the proposal to be deemed non-conforming.

The combined Phase I Base and Phase II Option efforts for this DO shall not exceed 18 months. Base Phase I award value shall not exceed $500,000, and the total combined award is limited to $1,000,000.

D. Schedule/Milestones

Proposers must address the following research project milestones and deliverables, along with fixed payable milestones in their proposals. The task structure must be consistent across the proposed schedule, Task Description Document (TDD), and the Vol. 2 Price Volume. Proposers must complete the “Schedule of Milestones and Payments” excel Attachment provided with this AIE Opportunity as part of submitting a complete proposal and fulfilling the requirements under Vol. 2 Price Volume. Performers can supplement the required milestones with additional milestones if needed, and should propose estimated funding for each. If selected for award negotiation, the fixed payable milestones provided will be directly incorporated into Attachment 2 of the OT agreement (“Schedule of Milestones and Payments”). Proposers are encouraged to use the TDD template provided with this AIE Opportunity, which will be Attachment 1 of the OT agreement.

Phase I base effort fixed payable milestones for this program shall include, at a minimum, the following (but proposers should provide additional detail, specific to their proposed project, as necessary):

Month 1: Report on initial architectures, algorithms, and approaches Month 3: Report on acquisition of initial training and test data sets (experimental, simulated or modeled), proposed evaluation metrics and initial results Month 5: Interim report describing performance of prototype system Month 7: Demonstration of predictive capability of generative models for both in-training and out-of-training input space; validation that approach can re-discover known physical laws from data without human guidance

Month 9: Final Phase I Report summarizing approach; architectures; algorithms; data sets;

results; comparison with alternative state-of-the-art methodology; quantification of accuracy;

quantification of robustness to errors, noise, dropouts, distortions; and quantification of generalizability (ability to model and predict behaviors not explicitly trained on)

Phase II option fixed milestones for this program shall include, at a minimum, the following (but proposers should provide additional detail, specific to their proposed project, as necessary):

Month 10: Based on lessons learned from Phase I, report on proposed updates to AI architectures, algorithms and approaches, and their application to the Phase II real world problem domain

Month 12: Report on acquisition of Phase II real-world data sets, proposed evaluation metrics and initial analyses results

Month 14: Interim report describing performance of real-world system and documenting ability of system to assess the power and limitations of the provided data in describing the system

Month 16: Interim report quantifying system performance; comparing with alternative state-of-the art approaches using machine learning, control theory or other conventional methods;

documenting lessons learned; and documenting the discovery of governing equations in representative real-world complex dynamic system and discovery of deep insights.

Month 18: Final Phase II report documenting final architectures; algorithms; methods; results;

comparisons with alternative methods; contribution of system in the scientific process and design of experiment; and quantification of accuracy, robustness and generalizability

For planning and budgetary purposes, proposers should assume a program start date of December 7, 2018.

All proposals must include the following meetings and travel in the proposed schedule and costs:

For budgeting purposes, plan for two two-day meetings or hackathons during Phase 1, one on the east coast and one on the west coast. Similarly, plan for two two-day meetings during Phase 2 option, one on the east coast and one on the west coast.

Regular teleconference meetings will be scheduled among program participants and the Government team for progress reporting, problem identification and mitigation. Plan for one teleconference meeting every two months.

E. Deliverables

Performers will be expected to provide at a minimum the following deliverables:

Negotiated deliverables specific to the proposed effort and TDD. These may include reports;

experimental and simulated data sets; architectures; algorithms; protocols; software codes;

publications; model data; metrics; validation data; and other associated documentation and results.

II. Award Information

Selected proposals that are successfully negotiated will result in award of an OT for prototype project agreement. See Section 3 of DARPA-PA-18-02 for information on awards that may result from proposals submitted in response to this notice.

Proposers must review the model OT for Prototype agreement provided as an attachment to DARPA-PA- 18-02 prior to submitting a proposal. DARPA has provided the model OT in order to expedite the negotiation and award process and ensure DARPA achieves the AIE goal of rapid program starts enabling DARPA to initiate a new investment in less than 90 days from idea inception. The task description document, schedule of milestones and payments, and data rights assertions requested under Volumes 1, 2, and 3 will be included as attachments to the OT agreement upon negotiation and award.

Proposers may suggest edits to the model OT for consideration by DARPA and provide a copy of the model OT with track changes as part of their proposal package. Suggested edits may not be accepted by DARPA. The Government reserves the right to remove a proposal from award consideration should the parties fail to reach agreement on OT award terms and conditions. If edits to the model OT are not provided as part of the proposal package, DARPA assumes that the proposer has reviewed and accepted the award terms and conditions to which they may have to adhere and the sample OT agreement provided as an attachment, indicating agreement (in principle) with the listed terms and conditions applicable to the specific award instrument.

III. Eligibility

See Section 4 of DARPA-PA-18-02 for information on who may be eligible to respond to this notice.

AIRA is a fundamental research effort and does not impose citizenship requirements for program participation.

IV. Opportunity Responses

Responses to this AIE Opportunity must be submitted as full proposals to DARPA-PA-18-02 as described therein. All proposals must be unclassified.

A. Proposal Content and Format

All proposals submitted in response to this opportunity notice must comply with the content and format instructions in Section 5 of DARPA-PA-18-02. All proposals must use the templates provided as Attachments to the PA and the “Schedule of Milestones and Payments” Excel Attachment provided with this AIE Opportunity and the and follow the instructions therein.

Information not explicitly requested in DARPA-PA-18-02, it’s Attachments, or this notice may not be evaluated.

B. Proposal Submission Instructions

See Section 5 of DARPA-PA-18-02 for proposal submission instructions.

C. Proposal Due Date and Time

Proposals in response to this notice are due no later than 4:00 PM on October 29, 2018. Full proposal packages as described in Section 5 of DARPA-PA-18-02 must be submitted per the instructions outlined therein and received by DARPA no later than the above time and date. Proposals received after this time and date may not be reviewed.

Proposers are warned that the proposal deadline outlined herein is in Eastern Time and will be strictly enforced. When planning a response to this notice, proposers should take into account that some parts of the submission process may take from one business day to one month to complete.

V. Proposal Evaluation and Selection

Proposals will be evaluated and selected in accordance with Section 6 of DARPA-PA-18-02. Proposers will be notified of the results of this process as described in Section 7.1 of DARPA-PA-18-02.

VI. Administrative and National Policy Requirements

Section 7.2 of DARPA-PA-18-02 provides information on Administrative and National Policy Requirements that may be applicable for proposal submission as well as performance under an award.

VII. Point of Contact Information

Jim Gimlett, Program Manager, DARPA/DSO, AIRA@darpa.mil mailto:AIRA@darpa.mil

VIII. Frequently Asked Questions (FAQs)

All technical, contractual, and administrative questions regarding this notice must be emailed to AIRA@darpa.mil. Emails sent directly to the Program Manager or any other address may result in delayed or no response.

All questions must be in English and must include name, email address, and the telephone number of a point of contact. DARPA will attempt to answer questions publically in a timely manner; however, questions submitted within 7 days of the proposal due date listed herein may not be answered.

DARPA will post an FAQ list under the AIE Opportunity on the DARPA/DSO Opportunities page at http://www.darpa.mil/work-with-us/opportunities?tFilter=&oFilter=2&sort=date. The list will be updated on an ongoing basis until one week prior to the proposal due date. In addition to the FAQ specific to this notice, proposers should also review the AIE General FAQ list on the DARPA/DSO Opportunities page under the AIE Program Announcement (DARPA-PA-18-02).

mailto:AIRA@darpa.mil http://www.darpa.mil/work-with-us/opportunities?tFilter=&oFilter=2&sort=date