HR001119S0026.pdf

Broad Agency Announcement Guaranteeing AI Robustness against Deception (GARD)

HR001119S0026

February 12, 2019

Defense Advanced Research Projects Agency Information Innovation Office 675 North Randolph Street Arlington, VA 22203-2114

HR001119S0026 GUARANTEEING AI ROBUSTNESS AGAINST DECEPTION (GARD) 2

Table of Contents

I. Funding Opportunity Description

A. Introduction

B. Program Description

C. Foundations

D. Multi-Modality

E. Program Structure

F. Technical Areas

G. Metrics

H. Evaluation Scenarios

I. Out of Scope

J. Schedule, Milestones, and Evaluation

K. Deliverables

II. Award Information

A. Awards

B. Fundamental Research

C. Disclosure of Information and Compliance with Safeguarding Covered Defense Information Controls

III. Eligibility Information

A. Eligible Applicants

B. Organizational Conflicts of Interest

C. Cost Sharing/Matching

D. Other Eligibility Requirements

IV. Application and Submission Information

A. Address to Request Application Package

B. Content and Form of Application Submission

C. Submission Dates and Times

D. Funding Restrictions

E. Other Submission Requirements

V. Application Review Information

A. Evaluation Criteria

B. Review and Selection Process

VI. Award Administration Information

A. Selection Notices

HR001119S0026 GUARANTEEING AI ROBUSTNESS AGAINST DECEPTION (GARD) 3

B. Administrative and National Policy Requirements

C. Reporting

VII. Agency Contacts

VIII. Other Information

A. Frequently Asked Questions (FAQs)

B. Proposers Day

C. Submission Checklist

HR001119S0026 GUARANTEEING AI ROBUSTNESS AGAINST DECEPTION (GARD) 4

PART I: OVERVIEW INFORMATION

Federal Agency Name: Defense Advanced Research Projects Agency (DARPA), Information Innovation Office (I2O)

Funding Opportunity Title: Guaranteeing AI Robustness against Deception (GARD)

Announcement Type: Initial Announcement

Funding Opportunity Number: HR001119S0026

Catalog of Federal Domestic Assistance Numbers (CFDA): 12.910 Research and Technology Development

Dates o Posting Date: February 12, 2019 o Proposers Day: February 6, 2019 o Abstract Due Date: February 26, 2019, 12:00 noon (ET) o Proposal Due Date: April 11, 2019, 12:00 noon (ET) o BAA Closing Date: April 11, 2019, 12:00 noon (ET)

Anticipated Individual Awards: DARPA anticipates multiple awards for Technical Area 1 and a single award for Technical Area 2.

Types of Instruments that May be Awarded: Procurement contracts, cooperative agreements, grants or other transactions (OTs)

Agency Contacts o Technical POC: Dr. Hava Siegelmann, Program Manager, DARPA/I2O o BAA Email: GARD@darpa.mil o BAA Mailing Address:

DARPA/I2O

ATTN: HR001119S0026

675 North Randolph Street Arlington, VA 22203-2114 o I2O Solicitation Website: http://www.darpa.mil/work-with-us/opportunities mailto:GARD@darpa.mil http://www.darpa.mil/work-with-us/opportunities

HR001119S0026 GUARANTEEING AI ROBUSTNESS AGAINST DECEPTION (GARD) 5

PART II: FULL TEXT OF ANNOUNCEMENT

I. Funding Opportunity Description

DARPA is soliciting innovative research proposals in the area of theoretical foundations, principled algorithms, and evaluation frameworks that significantly improve the robustness of machine learning systems to adversarial attacks. Proposed research should investigate innovative approaches that enable revolutionary advances in science, devices, or systems.

Specifically excluded is research that primarily results in evolutionary improvements to current practices.

This Broad Agency Announcement (BAA) is being issued, and any resultant selection will be made, using procedures under Federal Acquisition Regulation (FAR) 6.102(d)(2) and 35.016.

Any negotiations and/or awards will use procedures under FAR 15.4 (or 32 CFR § 200.203 for grants and cooperative agreements). Proposals received as a result of this BAA shall be evaluated in accordance with evaluation criteria specified herein through a scientific review process.

DARPA BAAs are posted on the Federal Business Opportunities (FBO) website (https://www.fbo.gov/) and the Grants.gov website (https://www.grants.gov/).

The following information is for those wishing to respond to this BAA. Proposers are strongly encouraged to read the entirety of this document, as information on interactions among technical areas (TAs) and information on evaluation, schedule, and deliverables is provided in sections other than those describing the particular TAs.

A. Introduction

The GARD program will develop a new generation of defenses against deception attacks on machine learning (ML). The program is soliciting game-changing research proposals to develop theory, create defenses, and implement appropriate testbeds leading to robust, deception-resistant ML/AI algorithms. Proposed research should investigate defenses that address entire threat scenario classes. Specifically excluded is research solely focused on developing defenses to specific attacks rather than addressing broad issues of defensibility.

The growing sophistication and ubiquity of ML components in advanced systems dramatically increase capabilities, but as a byproduct, also increases the potential for new vulnerabilities. The current era of adversarial AI focuses on approaches where imperceptible perturbations to ML inputs could deceive an ML classifier, significantly altering its response. Such results have initiated a rapidly proliferating field of research characterized by ever more complex attacks that require progressively less knowledge about the ML system being attacked, while proving increasingly strong against defensive countermeasures.

The acceleration in ML attack capabilities has promoted an arms race. As defenses are developed to address new attack strategies and vulnerabilities, improved attack methodologies capable of bypassing the defense algorithms are created. The field now appears increasingly pessimistic, sensing that developing effective ML defenses may prove significantly more difficult than designing new attacks, leaving advanced systems vulnerable and exposed. Recent https://www.fbo.gov/

HR001119S0026 GUARANTEEING AI ROBUSTNESS AGAINST DECEPTION (GARD) 6

theoretical research also suggests the inevitability of adversarial attacks targeting the deep neural networks commonly used in advanced systems. In addition to developing strongly defensible AI algorithms, GARD seeks to ensure that advanced systems are robust to the perturbations encountered in real-world operation.

Although the field of Adversarial AI is relatively young, dozens of attacks and defenses have already been proposed. For instance, recent optimization-based adversarial training presents a promising defense based on a principled definition of robust ML that maintains classification accuracy in the presence of small, size-bounded perturbations added to the inputs. This approach is powerful since it is designed to work against any attack algorithm leading to those particular input perturbations. The resultant defense, however, has proved slow to train. More significantly, it is only capable of protecting against perturbing the inputs in a predetermined metric; deception attacks working in other metrics are still able to penetrate the defense.

At present, the Adversarial AI field lacks a comprehensive theoretical understanding of ML vulnerabilities. Failing to fully take into account ML’s theoretical underpinnings leaves significant blind spots that can be exploited, and limits efforts to develop effective defenses.

GARD seeks to establish theoretical ML system foundations to identify system vulnerabilities, characterize properties that will enhance system robustness and encourage the creation of effective defenses. Novel adversarial AI defenses may also gain insight and inspiration from biological systems (e.g., the immune system, interactions between bacteria and viruses, sensory perception), where multiple mechanisms work synergistically to increase robustness. Currently, ML defenses tend to be highly specific, and are effective only against particular attacks. GARD seeks to develop defenses with more generalized applicability capable of defending against broad categories of attack. Furthermore, current evaluation paradigms of AI robustness often focus on simplistic measures that are less relevant to security. To verify robustness and wide applicability, defenses generated under GARD will be measured in a novel testbed employing scenario-based evaluations.

In summary, GARD’s purpose is to encourage both development of underlying theory and to functionally and substantially improve ML defensibility, leading to a new generation of defense approaches beyond current mathematical and algorithmic thinking.

B. Program Description

GARD has three objectives:

1. Create a sound theoretical foundation for defensible AI.

2. Develop principled, general defense algorithms.

3. Produce and apply a scenario-based evaluation framework to characterize which defense is most effective in a particular situation, given available resources. GARD defenses will be evaluated using realistic scenarios and large datasets.

C. Program Focus

GARD will focus on deception attacks that induce incorrect behavior in ML systems by manipulating their inputs. The program will develop defenses against attacks that have appeared

HR001119S0026 GUARANTEEING AI ROBUSTNESS AGAINST DECEPTION (GARD) 7

to date in literature, as well as any attacks published during the program’s four-year duration.

GARD’s defenses will consider many possible scenario factors, including:

1. Attacks at various learning phases: dataset poisoning in addition to inference time attacks that exploit extant system weaknesses.

2. Attacks with varying information about the attacked system including:

White box capability (i.e., knowledge of the network and all its weights or in general, the full detail of the ML employed);

Black box access to the system (attacker can query the system by providing input and observing the output); and

Blind attacks (aka, “transfer attacks”), where little is known about the system and no direct query access is possible. Though rarely considered to date, this method holds important implications for future scenarios, since it negates target defenses, which only hide the ML model used.

3. Attacks with varying information about the training data and the role of the ML system.

4. Attacks perturbing the input to the target ML system at different acquiring stages, including:

Perturbing the digital image already input into the target ML system (most of the current adversarial AI field); and

Alterations to the physical environment, e.g., placing stickers on a street sign.

Physical attacks, while not extensively studied to date, are of prime importance to real systems, since they nullify target defenses that apply only to inputs already in the system.

5. Attacks on different sensory modalities such as images, videos, and audio. To date, adversarial AI has primarily considered still images, concentrating on visual inputs (cameras); increasingly, ML systems process other data modalities such as acoustic, LIDAR, and radio signals; considering only image-based inputs limits understanding of attack modalities and system vulnerabilities, leaving ML unprotected.

6. Attacks and defenses with explicit consideration of computational resources and time constraints. Current attack approaches often unrealistically assume unbounded resources.

In addressing realistic conditions, some defenses will have to provide real-time protection, while others will use additional time and computational resources. Hence, GARD requires all algorithms to be analyzed and measured in terms of resource and timing constraints.

7. Detection as a form of defense. Early identification of attacks can be an effective real-time defense, allowing the system to implement countermeasures, such as closing itself from further communication and input.

8. Adaptive forms of defense. Current defense methods are fixed; utilizing a specific defense against a particular attack is of limited utility when the system is faced with multiple attacks featuring varying characteristics. A defense that can adapt itself to attack variations is considerably more robust.

9. AI deception attacks, to date, have focused on ML classifiers; GARD will consider other applications of ML, such as detection, localization, prediction, and decisions.

HR001119S0026 GUARANTEEING AI ROBUSTNESS AGAINST DECEPTION (GARD) 8

10. Deep networks, while popularly employed in many systems, are only one of many possible machine learning tools providing advanced capabilities; other advanced ML algorithms and techniques are within GARD’s scope. Similarly, system defense solutions to protect embedded ML such as smart, active data collection will be explored in later phases.

11. Multiple deception attacks can, in principle, take place in parallel. Analogously, attacks may occur against systems that utilize multiple sensory inputs rather than only one input modality. While such attacks have yet to be introduced, they constitute a realistic scenario possibility that will be explored in later program phases.

Program evaluations will include scenarios that combine the above factors in different ways, so technical approaches should be sufficiently general to address such combinations. Biologically inspired methods may prove invaluable, yet biological fidelity is not by itself evidence for adversarial robustness. All approaches will be evaluated in terms of their success in defending AI systems, rather than contribution to computational understanding of biological systems, which is not a goal of GARD.

C. Foundations

Adversarial AI is a new field arising from very recent advances in ML. To date, defenses against AI attacks have been reactive and highly specific to particular attacks, so they lack a comprehensive theoretical foundation. GARD will lay a broad foundation for defense of advanced ML systems, including:

1. Establishing new metrics for robust generalization and for measuring different levels of vulnerabilities;

2. Identifying the primary factors underlying ML vulnerabilities. This information will be critical in fitting future AI methods to particular roles and scenarios; and

3. Enhancing completeness in algorithmic analyses by evolving methods to more accurately calculate economy of defense based on perceived cost of resources and constraints in plausible scenarios.

D. Multi-Modality

GARD will address at least three sensor modalities: 2D multichannel still images, audio, and video. All performers will be tested in the still image category, and a second modality of their choice: either audio or video categories, or, at the performer’s discretion, both.

1. 2D multichannel still images. To date, most work in adversarial AI has focused on 2D multichannel still images, but much of that work involves datasets too simple to functionally evaluate largescale ML-based defenses. GARD scenarios that involve 2D multichannel still images will use datasets on the scale of ImageNet and its successors.

While RGB images are appropriate in Phase 1 of the GARD program, Phase 2 and Phase 3 will introduce alternative multichannel image modalities (e.g., RGB-depth or IR).

2. Audio. While some work on audio ML deception attacks exists, the literature is only a fraction of that covering image ML. At present, no physical audio attacks or defenses exist in the literature. Performers who choose audio (in addition to still images) will be

HR001119S0026 GUARANTEEING AI ROBUSTNESS AGAINST DECEPTION (GARD) 9

expected to propose techniques that defend against attacks on audio sound classification and speaker identification challenges. Phase 2 and Phase 3 will include defense against attacks on sound prediction and limited-dictionary speech to text.

3. Video. Like audio, video ML tools are currently limited. Most involve applying 2D still image approaches to video frame sequences. Performers who choose video (in addition to still images) are encouraged to utilize both spatial and temporal information in their approach. Performers choosing this modality will be expected to propose techniques that defend against attacks on object identification, classification, and localization in 3D space. Later phases will include defense against attacks on prediction as well.

GARD will rely exclusively on published attacks to enable development and evaluation of defensive technologies. It is out of scope for performers to develop novel attacks for use in advancing their research. Should there be insufficient published attacks to support proposed research at any time, the Government will negotiate appropriate changes to the statement of work.

E. Program Structure

GARD is a 48-month program, which has been divided into three phases. Phase 1 will be 12 months in duration; Phases 2 and 3 will each be 18 months long. Technical work under GARD has been organized into two technical areas (TAs):

TA1: Defense Theories and Algorithms o TA1.1: Theoretical Foundations for Defensible AI o TA1.2: Principled Defenses

TA2: Evaluation Framework

Each abstract and proposal submitted against this solicitation shall address only one TA.

Organizations may submit multiple abstracts/proposals to any one TA, or they may propose to both TAs. TA1 proposals may address either TA1.1, TA1.2, or both.

While a proposer may submit proposals for both technical areas, a particular proposer (as identified by Commercial and Government Entity (CAGE) Code), if selected for TA1 (including either TA1.1 or TA1.2), will be unable to be selected as a performer for any portion of TA2.

This selection process is intended to avoid organizational conflicts of interest (OCI) situations between the research TA and the integration and evaluation activities, as well as to ensure objective test and evaluation results.

F. Technical Areas

TA1: Defense Theories and Algorithms

Each TA1 proposal (regardless of whether focuses on TA1.1 or TA1.2, or both) will be required to:

Address multiple ML applications, such as object detection, object localization, classification, and prediction;

Choose between data poisoning attacks and inference attacks, or both;

HR001119S0026 GUARANTEEING AI ROBUSTNESS AGAINST DECEPTION (GARD) 10

Choose either audio, video, or both to study defenses beyond 2D still images; and Optionally address additional modalities together with metrics and methods of evaluating the success of defenses in the new modality

TA1.1: Theoretical Foundations for Defensible AI

TA1.1 will support theoretical developments that identify underlying vulnerabilities to attack, leading to well-analyzed defense algorithms with performance bounds explicitly dependent on threat scenarios and resource constraints. Performers under this TA will be expected to show progress during each phase in at least one of the following areas:

Creating metrics for robust generalization;

Finding factors of vulnerability and robustness;

Analyzing the defensibility of scenarios and algorithms; or Calculating economy of defense within threat scenarios.

Theory should assist in increasing depth of understanding in state-of-the-art attacks/defenses, as well as lay effective foundations for new defenses. Theories developed by TA1.1 performers must be clearly delineated and conveyed to the rest of the GARD performer groups to promote more effective analysis of ML vulnerabilities and robust generalization capabilities, enumerate attack/defense types and explore potential defenses. Algorithms developed in TA1.1 may be biologically inspired. The foundations and algorithms suggested under TA1.1 will be tested to establish their credibility and contribution to robust ML. Use of testbeds developed under TA2 is encouraged, but not required.

Proposed theories will have to account for multi-sensor, multi-modal, active, and adaptive sensors (See Table 1 below in Section I.G on metrics). “Multi-sensor” applies to either multiple sensors working in parallel or a single sensor used multiple times with resulting data fused into one coherent knowledge item (e.g., two cameras, camera plus microphone, or camera plus depth sensor). Theories should be functionally applicable to realistic scenarios and physical world attacks. Analyses should assess how a given attack scenario impacts defense method allocation of resources (e.g., computation, memory, energy consumption when available). Proposals must clarify why particular scenarios were selected, including outlining fundamental properties and relevance to well defined, realistic situations.

TA1.2: Principled Defenses

TA1.2 is the heart of the GARD program. While state-of-the-art attacks focus mainly on digitized image classification, TA1.2 performers will defend against potential detection, localization, classification, and prediction attacks. As TA1.2 projects advance in phase, performers will have to consider scenarios that allow for active / adaptive data collection by the defender system, and smart methods of using this capability to improve robustness.

Additionally, recognizing attacks and identifying unsafe situations also represent important defense components to be addressed under TA1.2. Performers must indicate which scenarios their algorithms can defend against, identify breaking points, and how to maximize success of their algorithm. Figure 1 (below) provides a high-level view of the challenges and processes to be addressed in TA1.2.

HR001119S0026 GUARANTEEING AI ROBUSTNESS AGAINST DECEPTION (GARD) 11

Proposers may include in their technical approaches investigation of topics such as how to defend systems incorporating reinforcement learning, strategy, real-time decision making, or substantial levels of autonomy;

TA2 teams and the Government Evaluator will suggest scenarios and attacks to defend against. All performers are invited to contribute to the discussion as well. Scenarios will vary in difficulty from checking algorithm function to exploring design challenges.

Evaluation scenarios will reflect current program phase foci (i.e., single sensor, multi-sensor, multimodal) and a range of possible knowledge about system attacks (e.g., blind, black-box, white-box), as well as attacks during different input acquiring stages (data set poisoning versus inference attacks). At least one physical world evaluation scenario will be conducted per phase; remaining scenarios will address digital scenarios or virtual simulations of the real world to support transition to physical scenarios. Performers will have ample notice prior to evaluation to consider different scenarios.

Runtime system Learned Model Output

Training Data

Learning Algorithm

Learning system

Deception Attacks (Expressed in

Scenarios)

ML Defenses

Sensors

Figure 1: TA1.2 high level view of ML defense processes and challenges to meet GARD challenges.

Additional TA1.2 requirements include:

Two to three sensor modalities must be selected:

o One modality, 2D still images, is required o Performers may choose between audio, video or both as a second or second and third modality o Optionally one modality from the following: text, IR, LIDAR, neuromorphic spiking hardware vision system, or other similar modality of high importance.

Department of Defense (DoD)-specific sensors are not the focus of GARD

Performers will participate in both classification and prediction evaluation tasks, or both object identification and location evaluations (or all, if desired);

Performers are encouraged to incorporate defenses to backdoor attacks (entrance without permission) induced by data poisoning that fit the modalities they choose;

HR001119S0026 GUARANTEEING AI ROBUSTNESS AGAINST DECEPTION (GARD) 12

TA1.2 will be tested on a variety of scenarios; scores will be based on the level of success defending against attacks, as well as the variety of attacks and scenarios being defended against; and

TA1.2 performers will propose their source datasets sufficient to support initial development in Phase 1, and explain their choice.

TA2: Evaluation Framework

TA2 performers will build a testbed to evaluate defenses under a wide-range of threat scenarios, and in particular, must reflect TA1.2 evaluation scenarios. The testbed will include TA1.1 metrics, as well as those designed by TA2 performer including the initial metrics discussed below. The TA2 performer will also be responsible for implementing the baseline defenses used to compute these metrics. The testbed will enable the evaluation of defense efforts against a specific set of published attacks and their combinations, including inference time and poisoning attacks, to be defined by the Government at the outset of the program, and revised periodically, in advance of evaluation events. TA2 performers will also incorporate new attacks as they are published during the program to test TA1.2 defenses, adapting them as necessary to the Evaluation Scenarios (for example, to the physical world or in black box conditions). Note, scenarios will not emulate any specific DoD operational system, but rather, will focus on essential parametrized factors central to real systems robustness. TA2 performers will be in regular contact with government evaluators for implemented attack and threat scenario details.

TA2 testbeds must include three sensor modalities: 2D multichannel still images, audio, and video. Images, videos, and audio can come from various sources, and the testbed will have to be flexible enough to support this variety. The testbed must also be flexible enough to host varied defense algorithms, and incorporate single, multiple, and multimodal sensors, as well as active and adaptive defenses and various features consistent with physical environments.

G. Metrics

The most direct measure of GARD success will be the continued improvement in model robustness to adversarial inputs compared to both (i) a system that is not under attack and without defense (baseline-not-attacked), as well as (ii) a system that is under attack and also uses current methods of defense (baseline-defended). These can be measured by comparing the reduction in accuracy induced by an attack on a system with GARD defenses to both baselines mentioned above, resulting in two metrics: Defense Figure of Merit (DFOM), and Defense Improvement over Baseline Defense (DIBD) as shown in Figure 2:

HR001119S0026 GUARANTEEING AI ROBUSTNESS AGAINST DECEPTION (GARD) 13

Figure 2: Components of two evaluation metrics

Auxiliary metrics defined from these components include:

Effectiveness of new Defense = M - P This represents the improvement in the accuracy of the operation once the GARD defense is implemented

Effectiveness of the Attack = N – P

Net Defense Effectiveness = N – M This represents restoration of accuracy when a defense mechanism is applied after an attack. Warning: this quantity can be gamed by use of a weak attack and weak defense.

DFOM = (M-P) / (N-P)

This represents the “success rate” of a defense mechanism against an adversarial attack;

when M = N, DFOM = 1 representing a perfect defense.

DIBD = (M-P) / (B-P)

This represents the “improvement” of a defense mechanism over the baseline defense under an adversarial attack; when DIBD is greater than 1 the proposed defense outperforms the baseline defense. This is the ultimate goal of defenses.

The DFOM is designed to capture the success of a defense at recovering accuracy while being subjected to adversarial attacks and may vary based on the defensibility of a scenario. The DIBD metric directly compares newly proposed defenses to known baseline defenses (e.g., the Madry defense with a perturbation size determined by the dataset). As DIBD values inherently account for the defensibility of a scenario by using a consistent baseline defense, it becomes more relevant to use DIBD to interpret a defense’s performance across multiple Evaluation Scenarios.

Note that the DFOM and DIBD will vary for different random input samples. It is therefore important to present these metrics as binned ensembles (e.g., in the form of a histogram) to convey their variability. To account for statistical variability, the DFOM and DIBD targets for

HR001119S0026 GUARANTEEING AI ROBUSTNESS AGAINST DECEPTION (GARD) 14

each Phase (see Table 1 below) must be in the 90th percentile of resulting values drawn from random input samples for a given attack.

While DFOM and DIBD serve as the primary initial metrics of robustness, GARD considers adversarial robustness as an inherently multi-objective problem. Defenses will also be evaluated using additional metrics, such as:

Characteristics of the attack (e.g., whether it requires white box access, vs. working blind, how many queries it requires, or the number of poisoned samples needed);

Errors during defense and errors during detection, which are the Type I, II, and III errors (over-predict, under-predict, and recover from targeted attacks) made at inference time when an attack detection mechanism is used;

Cost of defense in operation, which is the computational effort, time, access and knowledge required to perform an ML operation at inference time, separating real-time from offline detection/defense;

Cost of defense training, which covers the computational effort, time, access and knowledge required to train and prepare the ML system for detection of attack;

Computational effort, which is quantified both as a theoretical construct via number of operations in best, average, and worst case, and by experimentation (as CPU/hours, GPU/hours, and TPU/hours) using standard computing hardware. The potential for parallelization should be explicitly noted. And finally;

Energy consumption, which is a significant factor in size, weight, and power (SWAP) constrained applications.

Program goals based on the preceding explanation of metrics are expressed in Table 1 using the DFOM or DIBD metrics. Meeting, exceeding, or failing to meet goals will be considered in view of the additional metrics described above, as well as performer-suggested metrics of relevance. For example, it may be possible for a given defense mechanism to exceed a goal for a given phase, but computational resources could become too excessive to make the defense methodology practical. In short, meeting goals should be described in context to the nature of the attack, computational resources, and scenario.

Table 1: Program Defense Effectiveness Goals

H. Evaluation Scenarios

While the primary metrics DFOM and DIBD are straightforward to measure, meaningful assessment of the security implications depends on the threat context. GARD will assess defenses through a series of Evaluation Scenarios, each with a specific threat model and

Setting Phase 1 Phase 2 Phase 3 Scenario DFOM / DIBD DFOM / DIBD DFOM / DIBD

Image >0.3/1.3 >0.7/2 >0.95/3 Sound >0.3/1.3 >0.7/10 >0.95/100 Video >0.3/1.3 >0.7/10 >0.95/100

HR001119S0026 GUARANTEEING AI ROBUSTNESS AGAINST DECEPTION (GARD) 15

constraints on both attack and defense. These scenarios will be more complex and realistic than traditional adversarial examples. Many will feature physical world inputs, with complementary digital scenarios in between. Each Scenario will entail different priorities among the metrics described above; for example, placing a greater emphasis on computational costs or penalizing some types of error more than others.

The Government Evaluator will devise each scenario to be both challenging and informative about the key factors that govern model vulnerability. Scenarios are expected to cover visual and auditory modalities, initially independently. Later phases will incorporate greater complexity such as multi-sensors (e.g., two cameras), multi-modality (e.g., image plus LIDAR or depth sensors, or audio-video).

TA2 will work closely with the Government Evaluator to ensure that the testbed can support the Evaluation Scenarios. In addition, TA2 will be responsible for providing reference attacks to test TA1 defenses in each scenario. These scenarios are especially focused on TA1.2 performers, but TA1.1 teams will be encouraged to take advantage of scenario problems and TA2 testbed to test their work as well.

I. Out of Scope

The following are out of scope in this program:

Data theft, privacy, model inversion: GARD is focused on attacks that induce incorrect behavior in ML models by manipulating inputs.

Game theory, AI strategy, agent models, or the general use of AI in adversarial contexts:

These are broad problems in themselves and more appropriate to other programs.

Methods from these domains that can be shown to be relevant to the specific problems addressed by GARD are permissible.

Attacks on military or other government systems, extension to military datasets: GARD is a basic research program exploring the characteristics and limitations of ML methods under general adversarial assumptions.

General noise robustness for ML models: Adversarial inputs that could be ignored as having negligible probability under most noise models may be reliably produced by an attacker.

Generic cybersecurity of ML systems: GARD is concerned with the specific vulnerabilities introduced by the ML model itself. Application of traditional cybersecurity analysis to systems that happen to use ML is better addressed elsewhere.

Methods that focus solely on attack detection: Detection of adversarial inputs may be a useful component of defense, but does not fully address the problem.

J. Schedule, Milestones, and Evaluation

The program will conduct evaluations during each phase, as indicated in the schedule figure below (Figure 3). Performers will be evaluated twice a year in meetings; the main evaluations will be conducted toward the end of each phase, while the other evaluations near the middle of the phase will be less formal. For costing purposes, assume that meetings will alternate between the Washington D.C. area and San Francisco, CA, and that the main evaluations will require

HR001119S0026 GUARANTEEING AI ROBUSTNESS AGAINST DECEPTION (GARD) 16

four-day-long trips, while the other evaluations will require three-day-long trips. Cost proposals should assume a kickoff date of December 1, 2019.

A Government Evaluator will develop and conduct evaluations, utilizing the TA2 performer-developed testbed and possibly adding to it. Evaluation scenario complexity will increase over the course of the program. This BAA is not soliciting for the Government Evaluator role.

Month 1

Program Kickoff

Months 1 to 12

Month 29

Phase II Review

Month 48

Final Delivery

Phase I Phase II

Months 13 to 30 Months 31 to 48

Phase III

Month 12

Phase I Review

Month 6 TA 1,2 Interim

Review

Month 18 TA1,2 Interim

Review

Month 24 TA1,2 Interim

Review

Month 36 TA1,2 Interim

Review

Month 42 TA1,2 Interim

Review

TA1 / TA2 TA1 / TA2 TA1 / TA2

Figure 3: Overall schedule of the GARD program. Graphic assumes that GARD begins in Month 1 of a 48-month total effort.

Phase 1: (12 months) single sensor defenses

TA1.1 progress will include publishable papers describing theoretical findings, at least one of which must consider evaluation scenarios from any phase. TA1.1 progress will be assessed on their measurable contributions towards at least one of the following goals during each phase of the program: finding metrics, identifying factors of vulnerability and robustness, measuring capabilities, and relating these to existing or new algorithms. In addition, TA1.1 performers are encouraged to collaborate with performers addressing TA1.2 and consider evaluation scenarios.

TA1.2 progress will be assessed during evaluation scenarios. Scenarios will:

include object detection, object localization, and classification with parameterized metrics for inference and poisoning attacks; TA1 will choose between defending inference and poisoning attacks;

use 2D multichannel images, single-source audio or single-camera video;

measure progress with previously discussed metrics (Table 1);

involve large datasets (be prepared for ImageNet or larger);

evaluate audio TA1 methods on sound classification and speaker identification tasks from a set of 20 speakers with the same accent and language; and evaluate video TA1 methods on object identification and localization.

TA2 progress will include the development of a testbed that mirrors evaluation scenarios (including ML tasks, sensor modalities, inference, and poisoning), supports the most recent published attacks, and includes all required and appropriate metrics developed by TA1.1.

HR001119S0026 GUARANTEEING AI ROBUSTNESS AGAINST DECEPTION (GARD) 17

Phase 2: (18 months) multi-sensor defenses

TA1.1 progress will include papers describing theoretical findings, including at least one that reflects evaluation of TA1.2 scenarios from any phase. At least one finding must be applicable to scenarios from a GARD program phase and/or sensor modality that the performer has not yet addressed, or beyond the state of the art. At least two of the papers should provide general results to the community.

TA1.2 progress will be assessed with evaluation scenarios that introduce:

object detection, object localization, and classification and prediction;

multi-sensor datasets (multi-camera images/videos, multi-source audio);

increased attack capability (black-box attacks, more attack resources);

increased performance requirements (reduced type II error/miss rate);

combinations of state-of-the-art attacks;

audio scenarios with multi-source audio streams;

video scenarios with multi-view videos; and backdoor attacks, both poisoning and inference.

TA2 progress will include extension of the group’s Phase 1 testbed to include multi-sensor datasets, support for attacks published for each evaluation scenario, and support for combination attacks.

Phase 3: (18 months) multi-modality, active, and adaptive defenses

TA1.1 progress will include papers describing theoretical findings that consider evaluation scenarios of TA1.2 from any phase with modalities or scenarios beyond the state of the art.

TA1.2 progress will be assessed with evaluation scenarios that introduce:

object detection, object localization, and classification and prediction;

multimodal sensors (e.g., audio + video);

active and adaptive functionality (e.g., interactive defenses);

increased attack capability (white-box attacks, more attack resources); and increased performance requirements (reduce type III error/targeting).

TA2 progress will include extension of the group’s Phase 2 testbed to support Phase 3 TA1.2 evaluation scenarios, support for attacks published for each evaluation scenario, and active/adaptive functionality. Testbeds at this stage should be ready for transition.

HR001119S0026 GUARANTEEING AI ROBUSTNESS AGAINST DECEPTION (GARD) 18

K. Deliverables

Table 2 summarizes the anticipated deliverables by TA.

Technical Area Anticipated Deliverables

TA1.1: Theoretical Foundations

Properties of system design that reduce or increase vulnerability Metrics for vulnerability, attack effectiveness Algorithms for defense Computational resources required for defense against a given attack Methods for analysis of robust generalization and defended AI Results of experiments Bounds on reliability of test results; conclusions on effective test methods Conclusions regarding defensibility of scenarios of interest Recommendations on constraint conditions for TA2 challenge scenarios Publications

TA1.2: Principled Defenses

Implementations of existing and new defenses Defenses and defensible ML and scenarios of success Computational resources required for defense of a given attack scenario Analysis – both theoretical and practical Limits of algorithms and suggestions for defensible AI Results of tests and competitions Publications

TA2: Evaluation

Testbed, in a form usable by other organizations Definitions of new testing scenarios Results of tests of TA1 outputs Maintenance of a library of implemented attacks from the literature and defenses from GARD

All

Quarterly technical status reports Monthly financial reports Monthly phone/video meeting with PM Attendance at PI meetings Support for periodic reviews Source code and representative input data

Table 2: Deliverables

HR001119S0026 GUARANTEEING AI ROBUSTNESS AGAINST DECEPTION (GARD) 19

II. Award Information

A. Awards

DARPA anticipates multiple awards for TA1 and a single award for TA2. The level of funding for individual awards made under this solicitation has not been predetermined and will depend on the quality of the proposals received and the availability of funds. Awards will be made to proposers whose proposals are determined to be most advantageous to the Government, all factors considered, including the potential contributions of the proposed work, overall funding strategy, and availability of funding. See Section V for further information.

The Government reserves the right to:

select for negotiation all, some, one, or none of the proposals received in response to this solicitation;

make awards without discussions with proposers;

conduct discussions with proposers if it is later determined to be necessary;

segregate portions of resulting awards into pre-priced options;

accept proposals in their entirety or to select only portions of proposals for award;

fund proposals in increments and/or with options for continued work at the end of one or more phases;

request additional documentation once the award instrument has been determined (e.g., representations and certifications); and remove proposers from award consideration should the parties fail to reach agreement on award terms within a reasonable time or the proposer fails to provide requested additional information in a timely manner.

Proposals selected for award negotiation may result in a procurement contract, grant, cooperative agreement, or Other Transaction (OT) depending upon the nature of the work proposed, the required degree of interaction between parties, and other factors.

Proposers looking for innovative, commercial-like contractual arrangements are encouraged to consider requesting Other Transactions. To understand the flexibility and options associated with Other Transactions, consult http://www.darpa.mil/work-with-us/contract-management#OtherTransactions.

In accordance with 10 U.S.C. § 2371b(f), the Government may award a follow-on production contract or Other Transaction (OT) for any OT awarded under this BAA if: (1) that participant in the OT, or a recognized successor in interest to the OT, successfully completed the entire prototype project provided for in the OT, as modified; and (2) the OT provides for the award of a follow-on production contract or OT to the participant, or a recognized successor in interest to the OT.

In all cases, the Government contracting officer shall have sole discretion to select award instrument type, regardless of instrument type proposed, and to negotiate all instrument terms and conditions with selectees. DARPA will apply publication or other restrictions, as necessary, if it determines that the research resulting from the proposed effort will present a high likelihood of disclosing performance characteristics of military systems or manufacturing technologies that are unique and critical to defense. Any award resulting from such a determination will include a requirement for DARPA permission before publishing any information or results on the http://www.darpa.mil/work-with-us/contract-management#OtherTransactions http://www.darpa.mil/work-with-us/contract-management#OtherTransactions

HR001119S0026 GUARANTEEING AI ROBUSTNESS AGAINST DECEPTION (GARD) 20

program. For more information on publication restrictions, see the section below on Fundamental Research.

B. Fundamental Research

It is DoD policy that the publication of products of fundamental research will remain unrestricted to the maximum extent possible. National Security Decision Directive (NSDD) 189 defines fundamental research as follows:

‘Fundamental research’ means basic and applied research in science and engineering, the results of which ordinarily are published and shared broadly within the scientific community, as distinguished from proprietary research and from industrial development, design, production, and product utilization, the results of which ordinarily are restricted for proprietary or national security reasons.

As of the date of publication of this BAA, the Government expects that program goals as described herein may be met by proposers intending to perform fundamental research and does not anticipate applying publication restrictions of any kind to individual awards for fundamental research that may result from this BAA. Notwithstanding this statement of expectation, the Government is not prohibited from considering and selecting research proposals that, while perhaps not qualifying as fundamental research under the foregoing definition, still meet the BAA criteria for submissions. If proposals are selected for award that offer other than a fundamental research solution, the Government will either work with the proposer to modify the proposed statement of work to bring the research back into line with fundamental research or else the proposer will agree to restrictions in order to receive an award.

Proposers should indicate in their proposal whether they believe the scope of the research included in their proposal is fundamental or not. While proposers should clearly explain the intended results of their research, the Government shall have sole discretion to select award instrument type and to negotiate all instrument terms and conditions with selectees. Appropriate clauses will be included in resultant awards for non-fundamental research to prescribe publication requirements and other restrictions, as appropriate. This clause can be found at http://www.darpa.mil/work-with-us/additional-baa.

For certain research projects, it may be possible that although the research being performed by the awardee is restricted research, a subawardee may be conducting fundamental research. In those cases, it is the awardee’s responsibility to explain in their proposal why its subawardee’s effort is fundamental research

C. Disclosure of Information and Compliance with Safeguarding Covered Defense Information Controls

The following provisions and clause apply to all solicitations and contracts; however, the definition of “controlled technical information” clearly exempts work considered fundamental research and therefore, even though included in the contract, will not apply if the work is fundamental research.

DFARS 252.204-7000, “Disclosure of Information” DFARS 252.204-7008, “Compliance with Safeguarding Covered Defense Information Controls” http://www.darpa.mil/work-with-us/additional-baa

HR001119S0026 GUARANTEEING AI ROBUSTNESS AGAINST DECEPTION (GARD) 21

DFARS 252.204-7012, “Safeguarding Covered Defense Information and Cyber Incident Reporting”

The full text of the above solicitation provision and contract clauses can be found at http://www.darpa.mil/work-with-us/additional-baa#NPRPAC.

Compliance with the above requirements includes the mandate for proposers to implement the security requirements specified by National Institute of Standards and Technology (NIST) Special Publication (SP) 800-171, “Protecting Controlled Unclassified Information in Nonfederal Information Systems and Organizations” (see https://doi.org/10.6028/NIST.SP.800-171r1) that are in effect at the time the BAA is issued.

For awards where the work is considered fundamental research, the contractor will not have to implement the aforementioned requirements and safeguards; however, should the nature of the work change during performance of the award, work not considered fundamental research will be subject to these requirements.

http://www.darpa.mil/work-with-us/additional-baa#NPRPAC https://doi.org/10.6028/NIST.SP.800-171r1

HR001119S0026 GUARANTEEING AI ROBUSTNESS AGAINST DECEPTION (GARD) 22

III. Eligibility Information

A. Eligible Applicants

DARPA welcomes engagement from all responsible sources capable of satisfying the Government's needs, including academia (colleges and universities); businesses (large, small, small disadvantaged, etc.); other organizations (including non-profit); other entities (foreign, domestic, and government); FFRDCs; minority institutions; and others.

DARPA welcomes engagement from non-traditional sources in addition to current DARPA performers.

1. Federally Funded Research and Development Centers (FFRDCs) and Government Entities

a. FFRDCs FFRDCs are subject to applicable direct competition limitations and cannot propose to this BAA in any capacity unless they meet the following conditions: (1) FFRDCs must clearly demonstrate that the proposed work is not otherwise available from the private sector. (2) FFRDCs must provide a letter on official letterhead from their sponsoring organization citing the specific authority establishing their eligibility to propose to Government solicitations and compete with industry, and their compliance with the associated FFRDC sponsor agreement’s terms and conditions. This information is required for FFRDCs proposing to be awardees or subawardees.

b. Government Entities Government Entities (e.g., Government/National laboratories, military educational institutions, etc.) are subject to applicable direct competition limitations. Government entities must clearly demonstrate that the work is not otherwise available from the private sector and provide written documentation citing the specific statutory authority and contractual authority, if relevant, establishing their ability to propose to Government solicitations.

c. Authority and Eligibility At the present time, DARPA does not consider 15 U.S.C. § 3710a to be sufficient legal authority to show eligibility. While 10 U.S.C.§ 2539b may be the appropriate statutory starting point for some entities, specific supporting regulatory guidance, together with evidence of agency approval, will still be required to fully establish eligibility. DARPA will consider FFRDC and Government entity eligibility submissions on a case-by-case basis; however, the burden to prove eligibility for all team members rests solely with the proposer.

2. Foreign Participation Non-U.S. organizations and/or individuals may participate to the extent that such participants comply with any necessary nondisclosure agreements, security regulations, export control laws, and other governing…