TrojAI-V3.2.pdf

U.S. ARMY RESEARCH OFFICE

BROAD AGENCY ANNOUNCEMENT FOR

TrojAI

W911NF-19-S-0012

Issued by:

U.S. Army Contracting Command Aberdeen Proving Ground Research Triangle Park Division P. O. Box 12211 Research Triangle Park, NC 27709-2211

Issued: May 2, 2019 Concept Papers Due: May 31, 2019 Final Proposals by Invite Only Due: July 25, 2019

I. OVERVIEW OF THE FUNDING OPPORTUNITY: 4

B. Additional Overview Information 5

II. DETAILED INFORMATION ABOUT THE FUNDING OPPORTUNITY 6

A. 1 Program Overview 6 A.1.1 Problem Statement and Concept of Operations 6 A.1.2 Program Scope 7

A.1.3 Out of Scope 8 A.1.4 Method Considerations 9 A.1.5 relevant Expertise 9 A.1.6 Performer Team 9

A.2 Program Structure 10 A.2.1 Research Areas 10 A.2.2 Metrics 10 A.2.3 Testing and Evaluation 11 A.2.4 Program Stages and Performance Goals 11 A.2.5 Adversarial Examples 14 A.2.6 Timeline 15

A.3 Meetings and Travel Requirements 15 A.3.1 Teleconference meetings 15 A.3.2 Workshops 15 A.3.3 Site Visits 15

B. Federal Award Information 18

H. Agency Contacts: 48

I. OVERVIEW OF THE FUNDING OPPORTUNITY:

A. Required Overview Content

1. Federal Agency Name(s):

U.S. Army Research Office

Issuing Acquisition Office:

U.S. Army Contracting Command-Aberdeen Proving Ground, Research Triangle Park Division (ACC-APG RTP Division)

2. Funding Opportunity Title: TrojAI

3. Announcement Type Initial Announcement

4. Research Opportunity Number:

W911NF-19-S-0012

5. Catalog of Federal Domestic Assistance (CFDA) Number:

12.431 – Basic Scientific Research

6. Response Dates:

a. Concept Papers- May 31 2019 no later than 4:00 PM Eastern Time

b. Selection of Concept Papers for full proposal on June 10th 2019

c. Proposals Due – July 25 2019 no later than 4:00 PM Eastern Time

d. Selection of Proposal – Sept 1st, 2019

7. POINTS OF CONTACT

a. Contracting Officer: Kevin Bassler, kevin.j.bassler.civ@mail.mil

b. IARPA TrojAI Program Manager: Jeff Alstott, jeff.alstott@iarpa.gov C. ARO Program Manager: Cliff Wang, cliff.x.wang.civ@mail.mil

B. Additional Overview Information

This Broad Agency Announcement (BAA) which sets forth research areas of interest to the Army Research Office (ARO) and the Intelligence Advanced Research Projects Activity (IARPA) is issued under paragraph 6.102(d)(2) of the Federal Acquisition Regulation (FAR), and 10 USC 2358 which provides for the competitive selection of basic research proposals.

Proposals submitted in response to this BAA and selected for award are considered to be the result of full and open competition and in full compliance with the provision of Public Law 98- 369, "The Competition in Contracting Act of 1984" and subsequent amendments. This BAA focuses on basic research as defined at 32 CFR 22.105.

The Department of Defense agencies involved in this program reserve the right to select for award; all, some, or none of the proposals submitted in response to this announcement. The participating DoD agencies will provide no funding for direct reimbursement of proposal development costs. Technical and cost proposals (or any other material) submitted in response to this BAA will not be returned. It is the policy of participating DoD agencies to treat all proposal as sensitive, competitive information and to disclose their contents only for the purposes of evaluation.

Concept papers and technical and cost proposals (or any other material) submitted in response to this BAA will not be returned to the applicant. Unless noted in an applicant's proposal to the contrary, unsuccessful proposals will be retained for six (6) months from declination and then properly destroyed. It is the policy of participating DoD agencies to treat all proposals as sensitive, competitive information and to disclose their contents only for the purposes of evaluation.

This will be a two-step application process:

The application process under this BAA consists of a Concept Paper stage and a Proposal stage.

The purpose of this two-step approach is to facilitate pre-screening by the U.S. Government such that detailed proposals are only sought from applicants whose concept papers demonstrate the most promise for award (this also helps to reduce unnecessary bid and proposal effort). The government’s decision to invite a Proposal will be based upon the evaluation results of a timely and compliant Concept Paper submission. Only the most highly rated Concept Papers will receive an invitation from the government to submit a Proposal. An Applicant that does NOT submit a timely and compliant Concept Paper, is NOT eligible to submit a Proposal for consideration for funding. An Applicant that does NOT receive an invitation from the Government to submit a Proposal is NOT eligible to submit a Proposal. An Applicant invited to submit a Proposal will receive feedback on their Concept Paper that is expected to improve their Proposal submissions. To facilitate organizations who might want to come together to submit one proposal, all proposers invited for full proposal will receive each others' contact details and concept papers (sans budget section).

(End of Section)

II. DETAILED INFORMATION ABOUT THE FUNDING OPPORTUNITY

A.1 Program Overview The U.S. Army Research Office (ARO) in partnership with the Intelligence Advanced Research Projects Activity (IARPA) seeks research and development of technology and techniques for detection of Trojans in Artificial Intelligence. TrojAI is envisioned to be a 2-year effort with multiple awardees coming together as a group of performers (hereinafter referred to as the “performer team”), which will work together to achieve the common program goals set forth in this BAA.

A.1.1 Problem Statement and Concept of Operations Using current machine learning methods, an artificial intelligence (AI) is trained on data, learns relationships in that data, and then is deployed to the world to operate on new data. For example, an AI can be trained on images of traffic signs, learn what stop signs and speed limit signs look like, and then be deployed as part of an autonomous car. The problem is that an adversary that can disrupt the training pipeline can insert Trojan behaviors into the AI. For example, an AI learning to distinguish traffic signs can be given potentially just a few additional examples of stop signs with yellow squares on them, each labeled “speed limit sign”. If the AI were deployed in a self-driving car, an adversary could cause the car to run through the stop sign just by putting a sticky note on it, since the AI would incorrectly see it as a speed limit sign (see figure). The goal of the TrojAI Program is to combat such Trojan attacks by inspecting AIs for Trojans.

Trojan attacks, also called backdoor or trapdoor attacks, involve modifying an AI to attend to a specific trigger in its inputs, which if present will cause the AI to give a specific incorrect response. In the traffic sign case, the trigger is a sticky note. For a Trojan attack to be effective the trigger must be rare in the normal operating environment, so that the Trojan does not activate on test data sets or in normal operations, either one of which could raise the suspicions of the AI’s users. Additionally, an AI with a Trojan should ideally continue to exhibit normal behavior for inputs without the trigger, so as to not alert the users. Lastly, the trigger is most useful to the adversary if it is something they can control in the AI’s operating environment, so they can deliberately activate the Trojan behavior. Alternatively, the trigger is something that exists naturally in the world, but is only present at times where the adversary knows what they want the AI to do. Trojan attacks’ specificity differentiates them from the more general category of “data poisoning attacks”, whereby an adversary manipulates an AI’s training data to make it just generally ineffective.

In the initial example the Trojan was inserted by manipulating both the training data and its labels. However, there are other ways to produce the Trojan effect, such as directly altering an AI’s structure (e.g., manipulating a deep neural network’s weights)1 or adding to the training data that have correct labels but are specially-crafted to still produce the Trojan behavior2.

Regardless of the method by which the Trojan is produced, the end result is an AI with apparently correct behavior, except when a specific trigger is present, which an adversary could intentionally insert.

Obvious defenses against Trojan attacks include securing the training data (to protect data from manipulation), cleaning the training data (to make sure the training data is accurate), and protecting the integrity of a trained model (prevent further malicious manipulation of a trained clean model). Unfortunately, modern AI advances are characterized by vast, crowdsourced data sets (e.g., 109 data points) that are impractical to clean or monitor. Additionally, many bespoke AIs are created by transfer learning: take an existing, public AI published online and modify it a little for the new use case. Trojans can persist in an AI even after such transfer learning. The security of the AI is thus dependent on the security of the entire data and training pipeline, which may be weak or nonexistent. Furthermore, the user may not be the one doing the training. Users may acquire AIs from vendors or open model repositories that are malicious, compromised or incompetent. Acquiring an AI from elsewhere brings all of the problems with the data pipeline, as well as the possibility of the AI being modified directly while stored at a vendor or in transit to the user. Given the diffuse and unmanageable supply chain security, the focus for the TrojAI Program is on the operational use case where the complete AI is already in the would-be users’ hands: detect if an AI has a Trojan, to determine if it can be safely deployed.

A.1.2 Program Scope The performer team will develop and deliver software to automatically inspect an AI and predict if it has a Trojan. The AIs will be neural networks trained in a classification task. Initially, the

AIs will classify small images, but later stages of the program may expand to AIs that classify audio or text or perform other tasks. The AIs will classify input data into a small number of classes at near-human or super-human performance. As a reference, the image classification tasks will be analogous to the German Traffic Sign Recognition task3, but performers should not assume that the input data or classes correspond to any public data sets. Trojan attacks will modify a portion of the AIs to recognize input triggers and cause misclassification, similar to the “speed limit sign” example above. The performer team will deliver software that detects which AIs have been subject to a Trojan attack. Performer software will take as input the AI’s source code, architecture and compiled binary. The performer software may also receive a small number of examples of valid data from the AI’s problem domain, as described in section A.2.4 The performer team should be prepared to initially develop methods for AIs made of feedforward networks (e.g. Convolutional Neural Nets like ResNet for image classification) and then later develop methods for AIs made of recurrent networks (e.g. Long-Short Term Memory Networks for audio or text classification).

A.1.3 Out of Scope The scope of this Program is limited to inspecting a standalone piece of AI software with minimal information about the AI’s problem domain, divorced of software metadata or history. Out-of-scope methods would include:

Using side-channel information such as finding and inspecting the AI’s training data or inspecting log files of when and how the AI was trained.

Human-in-the-loop methods. The performer team’s delivered software will be run by a separate team to evaluate sequestered AIs, without the involvement of the performers or any other human. As such, a method that relied on explaining aspects of the AI’s decision making to a human, then the human making the prediction on whether the AI has a Trojan would be out of scope. However, performers could use human-in-the-loop methods to train the Trojan detection algorithm, which is then submitted to run autonomously. Similarly, developing methods for visualizing or explaining an AI’s actions may be useful auxiliary technologies that help create effective Trojan detectors.

Methods that attempt brute-force search by running the AI against a valid input and then systematically adding all possible Trojan triggers to the input to observe if the AI’s behavior changes. The stages of the Program (see A.2.4) will attempt to make such brute-force search computationally infeasible by using sufficiently large spaces for both the valid inputs and the possible Trojan triggers. However, in scope are non-brute-force search processes, such as leveraging heuristics or information gleaned from the AI’s behavior to dramatically decrease the search space.

Confirming that an AI exactly matches a gold standard, presumably untampered AI Methods that rely on knowing the manner in which the Trojan was inserted (e.g. mislabeled training data attacks, clean label training data attacks, or directly editing AI weights)

Lastly, out of scope is developing new forms of Trojan attacks, such as attacks that try to evade detection. However, any such attacks that are published elsewhere before or during the Program may also be incorporated as attacks used in the Program, and so developing methods to detect them is in scope.

A.1.4 Method Considerations Very recent research has developed methods detecting Trojans, given certain assumptions.

However, none of these currently solve the problem:

Examining the data set by using the AI’s own representations to help identify unusual clusters, which are likely Trojans4: In this Program the training data set will not be available for inspection.

Examining if a specific, known input has a Trojan trigger5: Requires observing an input that actually has a Trojan trigger, which will not be available in this Program. However, generating possible triggers and running such tests against them may be useful.

Manipulating valid inputs on the pixel level to search for inputs that switch the AI’s behavior, and thus are likely Trojans6: Relevant, but work so far has looked for Trojan triggers that exist in pixel space (e.g., a square in the lower-left corner) vs. in feature space (e.g., a yellow square in the middle of the stop sign, which may be at different places in the image, viewed askew, in low-light conditions, etc., which means the trigger will not consistently appear at the same pixels in the same way). The latter is the relevant paradigm for this Program.

A.1.5 Relevant Expertise IARPA anticipates offeror teams may include, but are not limited to, experts in the following technical areas:

● Deep learning

● Other machine learning methods

● Model inversion

● Model explainability, including visualization

● Cybersecurity

● Data mining

A.1.6 Performer Team This program is a 2-year effort (base year and 1 option year) and it is expected that awardees included in the performer team are likely to perform for the entire 2-year period. Awardees on the performer team are not competing against each other as the awardees are expected to work together as a whole to meet the program goals. Thus, all awardees will be strongly encouraged to openly share information and work together in order to achieve the overall objective of the TrojAI program. Members of the TrojAI performer team will be encouraged to regularly communicate, collaborate and support each other. This will be supported by the performer team’s weekly stand-up meetings (described in A.3.1) and regular public release of code (described in A.2.1).

Since awardees are expected to work together as described above, the awardee selection decision process will be based on how the individual contributions of the selected awardees come together as a whole to meet program goals. Thus, for example, development of supporting technology that does not directly detect Trojans, but demonstrably enables better Trojan- detection technologies by others on the performer team, may be proposed.

Awardees will be provided with dedicated channels to communicate with each other and IARPA about their research, such as a dedicated Slack workspace and dedicated email listserv.

These will be the channels by which IARPA communicates with the performers about research (but not, e.g., contracting). These records will be made public after the end of the Program.

A.2 Program Structure A.2.1 Deliverables The performer team will submit software that takes as an input an AI and outputs the probability the AI has a Trojan (a probability between 0 and 1). Information about the AI will include connection architecture, connection weights, the source code, the compiled software, etc.; the information will aim to match the white-box access that a customer who has commissioned or purchased an AI may have. The software will also be able to run the AI against data. Examples of valid (unattacked) data may also be available as inputs, as described in A.2.4.

The submitted software must be containerized, such as with Docker. IARPA seeks for the submitted software’s source code, along with ample documentation, to be posted to a public repository such as Github, to permit free and effective use by the public. IARPA anticipates that meeting these needs will necessitate a permissive use license, such as the MIT license.

A.2.2 Metrics The performance metric is accuracy in detecting whether an AI has been subject to a Trojan attack. The metric for accuracy will be the log score, or cross entropy, which is a proper scoring rule for measuring the accuracy of a probabilistic prediction. The log score is well-understood by machine learning researchers, as it is often the objective function used to train their AIs. The log score is calculated for an outcome y (0 or 1) and a forecast p (between 0 and 1):

– ∗ 1 ∗ 1

This simplifies to -log(p) if y=1 and -log(1-p) if y=0. The log score ranges from infinity (confident and wrong) to 0 (confident and correct).

If the Program is successful at detecting Trojans, a supplementary metric may be added: which classes have the Trojan, and which classes do they change to? For N classes, the number of possible Trojans between classes is N*(N-1). The performer software would return a probability for each possible Trojan, which would also be evaluated with the log score.

A.2.3 Testing and Evaluation The performer team’s software will be evaluated by an independent Testing and Evaluation (T&E) team, which will run the software on their hardware against a sequestered set of AIs and report the log score. The sequestered AIs will each be trained to the same classification task with the same class, but each will have different training data. Those AIs with Trojans will each have a different Trojan trigger, though each trigger will conform to certain known constraints (see A.2.4).

Performer software must process the entire set of sequestered AIs within 24 hours on a machine with specifications comparable to:

CPU: 10 physical cores, 20 logical cores of a Power9 running @3.5 GHz GPU: 1 Tesla V100, with 16 GB GPU memory Memory: 128 GB DDR4 memory Disk: 1.5TB temporary local scratch I/O: 2500 Gigabit / sec R/W

Performer software will be used to process each AI individually, and so will be unable to leverage shared or accumulated knowledge about the other sequestered AIs encountered during testing. However, the AIs can be processed in parallel to meet the time constraints.

Performer software may make use of public data sets, including novel data sets the performers create and publish. These data sets must be approved before submission by the T&E team, who will place the data on the T&E machines for all to use. Given that the Trojan-detection software will be run with hardware and time limits, the T&E team will reject data set submissions that are essentially attempts to use pre-calculation to circumvent the limits (such as attempting brute-force search, as described in 1.A.3). However, submitted software that generates new data on the fly, within the time limits, is permitted.

These constraints on hardware and data are only for running the Trojan-detection software at inference time. Performers may use different or greater quantities of hardware or data for training the Trojan-detection software, before deployment to the T&E team.

A.2.4 Program Stages and Performance Goals The Program will be organized by stages of increasing difficulty, which will push and demonstrate the state of the art. In each stage certain technical aspects of the Trojan-detection problem will be set and communicated to the performer team, which will then build and submit to T&E Trojan-detection software that will run against a roster of sequestered test AIs (see table below). Difficulty will be increased by manipulating aspects of the problem, such as:

Number of reference AIs provided: In each stage the T&E team may publish a population of reference AIs, each labeled as attacked or unattacked. These AIs' structures will be fully visible

(e.g., connection architecture, connection weights, source code, compiled code, etc.). These AIs will be generated by the same processes as the sequestered AIs used in T&E. In the first stage on the order of 1,000 AIs will be made public, but subsequent stages may decrease this number. In the first stage the true Trojan triggers and attacked classes for the reference AIs will also be released, but later stages may not include either.

Number of sequestered AIs for testing: Because the amount of time to evaluate the test AIs is fixed at 24 hours, more test AIs will lower the amount of time available to inspect each AI.

The first stage will have 100 test AIs (~15 minutes per AI), while later stages will likely increase to 1,000 test AIs (~1.5 minutes per AI).

Rarity of Trojans: In the first stage there will be a 50/50 class balance between AIs with and without Trojans. Later stages may increase the rarity of Trojans, such as to a 2/98 class balance.

Number of classes the AIs are trained to identify: In the first stage the AIs will be trained to classify images into 5 different classes. Later stages could increase this number.

Amount of test data for each class: Data points may be provided that are examples of each AI’s test data. The data points will be valid data points, without Trojan attacks in them. The test data will be available to the performers for the reference AIs, and to the software for the sequestered AIs. In the first stage 100 data points will be available for each class for each AI. In later stages fewer data points will be available for each class, including possibly 0 test data points for some classes.

Number of classes targeted by the Trojan: In the first stage, all classes will be attacked the same (e.g. A single trigger causes images from any class to go to the same class X). In later stages, more complex combinations of attacks may be present (e.g. Class X is triggered to go to class Y, class Y is triggered to go to class Z, and class A is triggered to go to class B). In all stages, general facts about the attack structures will be known to the performers, but not the identity of the attacked classes.

Variety of AIs: In the first stage the set of AIs will include at least 3 different types of deep neural network architectures. This number could increase in subsequent stages. Performer can assume that the architectures will be close to the current state of the art for training to human-level performance at the task while minimizing training times and cost. As an example, the current DAWNBench leaderboard for CIFAR10 training has ResNets of 9 to 50 layers.7

Regardless of the AIs’ architectures, they will be delivered in a well-specified format communicated to the performers, such as ONNX8.

Variability of Trojan triggers: In the first stage each attacked AI will have a Trojan trigger that is a polygon of uniform color with no more than 12 sides located on the surface of the classified object at specific (unknown) location. The classified object itself will make up the majority of the input image, and the trigger will be between 2% and 25% of the surface area of the classified object. Later stages may increase the space of possible triggers, such as by making them complex shapes with multiple colors. In all cases the space of possible triggers will be clearly communicated at the start of the round, but a fixed list of specific trigger images will not be provided. In no case will the triggers shrink to single-pixel size.

Trojan attack mechanism: The initial stage will employ Trojans created by manipulating training data and its labels. It is not anticipated that differences in the Trojan attack method will be material for detecting the resulting Trojan, but if this proves to be the case then later stages may use different attack methods (e.g., clean-label data attacks or directly manipulating network weights).

AI problem domain: In the initial stage the AIs will be classifying images. In later stages the

AIs may also classify audio or text. If classification as a whole is solved, then later stages may have AIs trained for more complex behaviors like question-answering or game playing.

In all stages these and any other aspects of the problem will be communicated to the performer team once the stage begins. The exact parameters of each stage will be determined over the course of the Program in response to the capabilities the performers develop. Examples of 6 possible stages are shown in the table below:

Stage Reference AIs (Public)

Test AIs (Sequestered for T&E)

# Classes in Data AI Trained to Classify

Test Data Points Provided

Problem Domain

1 1,000 AIs;

50% attacked

100 AIs;

50% attacked

5 100 Image Classification

2 1,000 AIs;

2% attacked

1,000 AIs;

2% attacked

5 2 Image Classification

3 3 AIs; 0% attacked

1,000 AIs;

50% attacked

5 1 Image Classification

4 1,000 AIs;

50% attacked

1,000 AIs;

50% attacked

10 1 for most classes, 0 for some classes

5 1,000 AIs;

2% attacked

1,000 AIs;

2% attacked

5 5 Audio Classification

6 1,000 AIs;

2% attacked

1,000 AIs;

2% attacked

5 5 Text Classification

The goal for each stage will be to close half the distance between random guessing and perfect performance. For a task with a 50/50 split between attacked and unattacked AIs, random guessing would yield a (natural) log score of .693 and the target would be .347. For a task with a 2/98 split between attacked and unattacked AIs, random guessing would yield a (natural) log score of .098 and the target would be .049.

Once a stage’s goal is reached the next stage may begin. The total number of stages in the Program could range from 1 to a large number; this will be determined by how quickly the target for each stage is reached by the performer team.

The entire performer team will move together between stages. It is expected, but not required, that the performer team will need to develop and deploy new solutions for each stage. During the Program it is acceptable for the performer team to develop a mix of methods, some of which are low-hanging fruit expected to quickly succeed at an early stage and some of which will take longer to develop but are needed to succeed at later stages.

A.2.5 Adversarial Examples Trojans are deliberate attacks, but AIs can also make incorrect classification due to endogenous errors. One well-studied error type is “adversarial examples”, in which AIs, typically deep neural networks, make misclassification errors based on input features that would never fool a human.9

The most philosophically extreme instances of adversarial examples are very small manipulations to an input, such as the addition of static to an image that a human eye couldn’t even detect, which cause the AI to confidently misclassify the input. However, adversarial examples can also involve features that are large and semantically meaningful to humans, and they can be somewhat robust to varying conditions such as viewing angle.10 Such adversarial examples can be the same kind of misclassification errors that Trojans create, with the only difference being they are “natural” instead of manufactured.

Adversarial examples are a problem for AI users, and there is very active research on how to build AIs such that they are infrequent or nonexistent. However, even if adversarial examples are solved, the problem of Trojan attacks will likely remain. As such, in the TrojAI Program adversarial examples are not the object of interest. Adversarial examples can and will occur as false positives during the course of the Program, but the Program includes several factors that will mitigate the influence of adversarial examples:

Defensive training

Where possible, the AIs will be trained with the latest published techniques for preventing adversarial examples from occurring. While no defensive techniques have conclusively removed adversarial examples, they have made them rarer.

Tighter definitions for what is considered an attack

Trojan triggers will have a minimum size and will be highly robust to multiple environmental effects such as viewing angles and lighting conditions. While adversarial examples can also have these properties, the heightened requirements mean there will be fewer of them.

In the first stage, Trojan attacks will be against all classes, meaning the presence of the trigger will cause any input to be misclassified to the same class. While adversarial examples can also have these properties11, the heightened requirements again mean there will be fewer of them.

Opportunities for new science

Both adversarial examples and Trojans are instances of the AI’s decision boundary between two classes being wrong, but the shape of those incorrect boundaries may be different. Recent research has shown how to detect simpler Trojans not because there were no other errors in the AI’s decision boundary, but because the Trojan attack was a more precise perturbation than the other, naturally occurring errors.12 It may be that it is possible to reliably distinguish Trojan attacks from adversarial examples.

A.2.6 Timeline IARPA intends to run the Program for 24 months, with 1 base year and 1 option year. It is expected that awardees included in the performer team are likely to perform for the entire 2-year period. Awardees on the performer team are not competing against each other as the awardees are expected to work together as a whole to meet the program goals.

The Program will support and require continuous software development, with the performer team rapidly pushing new developments to public code repositories and to T&E. The T&E team will support testing performer submissions at a rate of up to weekly. Results, including log scores and runtimes, will be made immediately visible to the entire performer team as soon as they are calculated. As such, there will be no pre-defined dates for delivering software, but there will be continuous evaluation of the state of the art for the performer team’s research directions.

This continuous evaluation will be accompanied by regular updates to IARPA Program management and the rest of the performer team, which will describe the technical methods behind submissions and current research problems.

A.3 Meetings and Travel Requirements Awardees are expected to assume responsibility for administration of their projects and to comply with contract/assistance award requirements for reporting, attendance at Program teleconferences and workshops, and availability for site visits by Program management.

A.3.1 Teleconference meetings The TrojAI Program intends to host weekly stand-up teleconference meetings for members of the performer team to provide updates on their progress to each other and to Program management.

A.3.2 Workshops The TrojAI Program intends to hold a Program-level Kick-Off meeting by the second month of the Program and then similar 2-day Workshops semi-annually thereafter. The dates and location of these are to be specified at a later date by the Government, but are anticipated to include an annual meeting in the Washington Metro Area (WMA) and an annual meeting satellite to a major machine learning conference. The Workshops will focus on technical aspects of the Program and on facilitating open technical exchanges, interaction, and sharing among the various Program participants. Performer team members will be expected to present the technical status and progress of their projects to other participants and invited guests.

A.3.3 Site Visits Site visits by the Contracting Officer Representative (to include the Grants Officers Representative and the Cooperative Agreements Manager) and the TrojAI Program Manager will generally take place up to twice yearly during the Program and will occur during the period between Program-level Workshops. These visits will occur at the performers’ facilities. Reports on technical progress, details of successes and issues, contributions to the Program goals, and technology demonstrations will be expected at such visits.

Activity Provisional Month into Performance Period

Workshop (WMA) 2 Site Visits 5 Workshop (ML Conference) 8 Site Visits 11 Workshop (WMA) 14 Site Visits 17 Workshop (ML Conference) 20 Site Visits 23

Delivery of Trojan Detection Software Continuously

Stand-up teleconference meeting Weekly

References

1) https://github.com/bryankim96/stux-DNN; Zou, Minhui, Yang Shi, Chengliang Wang, Fangyu Li, WenZhan Song, and Yu Wang. “PoTrojan: Powerful Neural-Level Trojan Designs in Deep Learning Models,” February 8, 2018. https://arxiv.org/abs/1802.03043.

2) Turner, Alexander, Dimitris Tsipras, and Aleksander Madry. “Clean-Label Backdoor Attacks,” September 27, 2018. https://openreview.net/forum?id=HJg6e2CcK7.

3) http://benchmark.ini.rub.de/?section=gtsrb&subsection=data set

4) Tran, Brandon, Jerry Li, and Aleksander Madry. “Spectral Signatures in Backdoor

Attacks.” ArXiv:1811.00636 [Cs, Stat], November 1, 2018.

http://arxiv.org/abs/1811.00636.; Chen, Bryant, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava.

“Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering.”

ArXiv:1811.03728 [Cs, Stat], November 8, 2018. http://arxiv.org/abs/1811.03728.

5) Chou, Edward, Florian Tramèr, Giancarlo Pellegrino, and Dan Boneh. “SentiNet:

Detecting Physical Attacks Against Deep Learning Systems.” ArXiv:1812.00292 [Cs], December 1, 2018. http://arxiv.org/abs/1812.00292.

6) https://people.cs.vt.edu/vbimal/publications/backdoor-sp19.pdf

7) https://dawn.cs.stanford.edu/benchmark/index.html#cifar10-train-cost

8) https://onnx.ai/

9) Szegedy, Christian, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian

Goodfellow, and Rob Fergus. “Intriguing Properties of Neural Networks.”

ArXiv:1312.6199 [Cs], December 20, 2013. http://arxiv.org/abs/1312.6199.

10) Eykholt, Kevin, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. “Robust Physical-World Attacks on Deep Learning Models.” ArXiv:1707.08945 [Cs], July 27, 2017.

http://arxiv.org/abs/1707.08945.

11) Brown, Tom B., Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer.

“Adversarial Patch.” ArXiv:1712.09665 [Cs], December 27, 2017.

http://arxiv.org/abs/1712.09665.

12) https://people.cs.vt.edu/vbimal/publications/backdoor-sp19.pdf

(end of section)

B. Federal Award Information

It is anticipated the awards will be made in the form of contracts, grants, and cooperative agreements. The awards will be made at funding levels commensurate with the proposed research, investigator/team type, as well as availability of funding. We realize the preparation of a research proposal often represents a substantial investment of time and effort by the applicant. Therefore, in an attempt to minimize this burden, we are requiring applicants interested in funding under this BAA to submit concept papers describing the type of research effort contemplated. These concept papers will be reviewed and ranked by a government panel. A detailed description of the concept paper submissions can be found in Section D.

Highest ranked applicants will be invited to submit full proposals. An Applicant that does NOT receive an invitation from the Government to submit a Proposal is NOT eligible to submit a Proposal. Only those applicants invited by a TPOC and/or the Program Manager will be eligible to submit a proposal. To facilitate those organizations that may want to come together in submitting a proposal, all proposers invited for full proposals will receive each others' contact details and concept papers (sans budget section).

Anticipated awards will be made in the form of procurement contracts, grants, or cooperative agreements and are subject to the availability of appropriations. Proposals must have a 24-month duration. Funding for the second year will be contingent upon satisfactory performance and the availability of funds.

The ACC-APG RTP Division has the authority to award a variety of instruments on behalf of ARO. The ACC-APG RTP Division reserves the right to use the type of instrument most appropriate for the effort proposed. Applicants should familiarize themselves with these instrument types and the applicable regulations before submitting a proposal. Following are brief descriptions of the possible award instruments.

1. Procurement Contract. A legal instrument, consistent with 31 U.S.C. 6303, which reflects a relationship between the Federal Government and a State Government, a local government, or other entity/contractor when the principal purpose of the instrument is to acquire property or services for the direct benefit or use of the Federal Government.

Contracts are primary governed by the following regulations:

a. Federal Acquisition Regulation (FAR) http://farsite.hill.af.mil/

b. Defense Federal Acquisition Regulation Supplement (DFARS) http://farsite.hill.af.mil/vmdfara.htm

c. Army Federal Acquisition Regulation Supplement (AFARS) http://farsite.hill.af.mil/vmafara.htm

2. Grant - A legal instrument that, consistent with 31 U.S.C. 6304, is used to enter into a relationship:

a. The principal purpose of which is to transfer a thing of value to the recipient to carry out a public purpose of support or stimulation authorized by a law or the United States, rather than to acquire property or services for the DoD's direct benefit or use.

b. In which substantial involvement is not expected between the DoD and the recipient when carrying out the activity contemplated by the grant.

c. No fee or profit is allowed.

3. Cooperative Agreement. A legal instrument which, consistent with 31 U.S.C. 6305, is used to enter into the same kind of relationship as a grant (see definition "grant"), except that substantial involvement is expected between the DoD and the recipient when carrying out the activity contemplated by the cooperative agreement. The term does not include "cooperative research and development agreements" as defined in 15 U.S.C. 3710a. No fee or profit is allowed.

4. Grants and cooperative agreements for Institutions of Higher Education and nonprofit organizations are primary governed by the following:

a. Federal statutes

b. Federal regulations

c. 2 CFR part 200, as modified and supplemented by DoD's interim Implementation found in

2 CFR part 1103

d. 32 CFR Parts 21, 22, 26, and 28.

e. DoD R&D General Terms and Conditions dated July 2018

f. ACC-APG-RTP Division Assistance, Research General Terms and Conditions dated

August 2016, hereinafter referred to as “Agency Specific Requirements”

g. Award-specific terms and conditions

5. Grants and cooperative agreements for for-profit and nonprofit organizations exempted from

Subpart E—cost principles of part 200, are primary governed by the following:

a. Federal statutes

b. Federal regulations

c. 32 CFR Part 34 - Administrative Requirements for Grants and Agreements with

For-Profit Organizations

d. 32 CFR Parts 21, 22, 26, and 28

e. DoD Research and Development General Terms and Conditions

f. Agency-specific Research Terms and Conditions

6. The following websites may be accessed to obtain an electronic copy of the governing regulations and terms and conditions:

a. FAR, DFARS, and AFARS: http://farsite.hill.af.mil/

b. Code of Federal Regulations (CFR): https://www.ecfr.gov/cgi-bin/ECFR?page=browse

c. DoD Research and Development General Terms and Conditions:

https://www.onr.navy.mil/work-with-us/manage-your-award/manage-grant-award/grants-terms-conditions

d. Agency-specific Research Terms and Conditions:

http://www.arl.army.mil/www/default.cfm?page=8

C. Eligibility Information

1. Eligible Applicants:

Eligible applicants under this BAA include Institutions of higher education (foreign and domestic), nonprofit organizations, and for profit concerns (large and small businesses). This BAA focuses on basic research as defined at 32 CFR 22.105.

2. Cost Sharing or Matching:

There is no requirement for cost sharing, matching, or cost participation to be eligible for award under this BAA and cost sharing and matching is not an evaluation factor used under this BAA.

In addition, if cost sharing is proposed on a grant or cooperative agreement proposal submitted by a nonprofit or institution of higher education, the award will be subject to the restrictions at 2 CFR 200.306. If cost sharing is proposed on a contract proposal, the award will be subject to the restrictions at FAR 35.003

3. Federally Funded Research and Development Centers (FFRDCs) and Government Entities:

a. FFRDCs FFRDCs are subject to applicable direct competition limitations and cannot propose to this BAA in any capacity unless they meet the following conditions: (1) FFRDCs must clearly demonstrate that the proposed work is not otherwise available from the private sector. (2) FFRDCs must provide a letter on official letterhead from their sponsoring organization citing the specific authority establishing their eligibility to propose to Government solicitations and compete with industry, and their compliance with the associated FFRDC sponsor agreement’s terms and conditions. This information is required for FFRDCs proposing to be awardees or subawardees.

b. Government Entities Government Entities (e.g., Government/National laboratories, military educational institutions, etc.) are subject to applicable direct competition limitations. Government entities must clearly demonstrate that the work is not otherwise available from the private sector and provide written documentation citing the specific statutory authority and contractual authority, if relevant, establishing their ability to propose to Government solicitations. This information is required for Government Entities proposing to be awardees or subawardees.

D. CONCEPT PAPER SUBMISSION INFORMATION

1. Overview

Concept papers should focus on describing details of the proposed research, including how it is innovative and how it could substantially increase the scientific state of the art.

Concept papers are limited to six (6) total pages: three (3) pages for technical content, one (1) cover page, one (1) page for personnel and one (1) page for budget, as discussed below.

Evaluators will only review the concept paper cover page, up to three concept paper technical content pages, and the one-page addendum. Any references may be placed on additional pages that will not count against the 6-page limit.

Concept papers must be in the following format but do not require any special forms:

• Page Size: 8 ½ x 11 inches

• Margins – 1 inch

• Spacing – single

• Font – Times New Roman, 12 point

Combine all files and forms into a single PDF before submitting.

2. Format and Content of Concept papers:

a. COVER PAGE (not to exceed one page):

The concept paper cover page shall include at a minimum: Title of the concept paper, name and contact information of the individual and organization submitting the concept paper , the BAA number of this announcement, and the TPOC name, if known.

b. TECHNICAL CONTENT (not to exceed three pages):

What is your basic idea and technical approach? What potential scientific understandings and new techniques would come out of the proposed research? Why is it innovative? Are there initial results? What are the hard technical challenges to this idea you will be focused on with your research?

c. PERSONNEL (not to exceed one page):

Include biographical sketches of the key personnel who will perform the research.

d. BUDGET (not to exceed one page)

Include a rough estimate of costs. It is expected that the entire TrojAI program, across all awardees, will be able to support 8-10 researchers (university faculty or industry PI) at the 20% load level and 16-20 graduate students/postdocs per year. Computational cost (either equipment purchase or cloud usage needed to support research), travel and overhead will need to be included in the budget plan.

3. Restrictive Markings on concept papers:

a. Concept papers should NOT contain any proprietary data. The applicant must also identify any technical data or computer software contained in the concept paper that is to be treated by the Government as limited rights in technical data and restricted rights in computer software.

In the absence of such identification, the Government will conclude there are no limitations or restrictions on technical data or computer software included in the concept paper. Records or data bearing a restrictive legend may be included in the concept paper.

Care must be exercised to ensure that classified, sensitive, and critical technologies are not included in a concept paper. If such information is required, appropriate restrictive markings and procedures should be applied prior to submission of the concept paper.

b. Applicants are cautioned, however, that portions of the concept papers may be subject to release under terms of the Freedom of Information Act, 5 U.S.C. 552, as amended.

4. Evaluation and Disposition of concept papers

(1) Evaluation Process: Applicants are advised that invitations for proposals will be made based on the concept paper submission and the availability of funding. The concept paper will be evaluated for the concept's scientific merit and technical and resource plausibility.

Applicants whose concept papers are evaluated as having significant scientific merit may be invited to submit a full proposal. An applicant may not submit a proposal without submitting a concept paper and receiving a proposal invite from the Government. All concept papers received after the deadline (specified in Section I.A.6) will not be considered.

(2) Disposition Process: The applicant will be notified in writing after completion of the evaluation. No formal feedback will be provided for submissions not invited for a full proposal. Concept papers will not be returned to applicants.

5. Concept paper Submission

All concept papers must be emailed directly to the following email address:

usarmy.rtp.rdecom-aro.mbx.baa2@mail.mil. In the email subject line, include the phrase “Concept paper Submission,” the BAA number W911NF-19-S-0012. Concept papers submitted via email must be in a single PDF formatted file as an email attachment.

(end of section)

E. PROPOSAL APPLICATION AND SUBMISSION INFORMATION

1. Address to View Broad Agency Announcement

This BAA may be accessed from the following:

1) Grants.gov (www.grants.gov)

2) FedBizOpps (www.fbo.gov)

3) ARL website https://www.arl.army.mil/www/default.cfm?page=8

Amendments, if any, to this BAA will be posted to these websites when they occur.

Interested parties are encouraged to periodically check these websites for updates and amendment.

The following information is for those wishing to respond to the BAA:

2. Content and Form of Application Submission

a. General Information

A proposal submitted under this BAA must address unclassified fundamental research.

Proposal submissions will be protected from unauthorized disclosure in accordance with applicable laws and DoD regulations. Applicants are expected to appropriately mark each page of their submission that contains proprietary information. Any proprietary data that the applicant intends to be used only by the Government for evaluation purposes must be clearly marked. The applicant must also identify any technical data or computer software contained in the proposal that is to be treated by the Government as limited rights in technical data and restricted rights in computer software. The participating DoD agencies will provide no funding for direct reimbursement of proposal development costs. Technical and cost proposals (or any other material) submitted in response to this BAA will not be returned.