HR001121S0015.pdf

Broad Agency Announcement Perceptually-enabled Task Guidance (PTG)

HR001121S0015

March 5, 2021

Defense Advanced Research Projects Agency Information Innovation Office 675 North Randolph Street Arlington, VA 22203-2114

HR001121S0015 PERCEPTUALLY-ENABLED TASK GUIDANCE (PTG) 2

Table of Contents Part I: Overview Information……………………………………………………………………...………3

Part II: Full Text of Announcement………………………………………………………………………..5

I. Funding Opportunity Description

II. Award Information

A. Awards

B. Fundamental Research

C. Disclosure of Information and Compliance with Safeguarding Covered Defense Information Controls

III. Eligibility Information

A. Eligible Applicants

B. Organizational Conflicts of Interest

C. Cost Sharing/Matching

D. Other Eligibility Requirements

IV. Application and Submission Information

A. Address to Request Application Package

B. Content and Form of Application Submission

C. Submission Dates and Times

D. Funding Restrictions

E. Other Submission Requirements

V. Application Review Information

A. Evaluation Criteria

B. Review and Selection Process

VI. Award Administration Information

A. Selection Notices

B. Administrative and National Policy Requirements

C. Reporting

VII. Agency Contacts

VIII. Other Information

A. Frequently Asked Questions (FAQs)

B. Collaborative Efforts/Teaming

C. Proposers Day

D. Submission Checklist

E. Associate Contractor Agreement (ACA)

HR001121S0015 PERCEPTUALLY-ENABLED TASK GUIDANCE (PTG) 3

PART I: OVERVIEW INFORMATION

Federal Agency Name: Defense Advanced Research Projects Agency (DARPA), Information Innovation Office (I2O)

Funding Opportunity Title: Perceptually-enabled Task Guidance (PTG)

Announcement Type: Initial Announcement

Funding Opportunity Number: HR001121S0015

Catalog of Federal Domestic Assistance Numbers (CFDA): 12.910 Research and Technology Development

Dates o Posting Date: March 5, 2021 o Proposers Day: March 18, 2021 o Abstract Due Date: March 31, 2021, 12:00 noon (ET) o Proposal Due Date: May 14, 2021, 12:00 noon (ET)

Anticipated Individual Awards: Multiple awards are anticipated (roughly six in Technical Area (TA) 1 and two in TA2), and contingent upon the number and quality of proposals received and the availability of funds.

Total Funding Available for Award: Approximately $30 million for TA1, and $10 million for TA2.

Types of Instruments that May be Awarded: Procurement contracts, cooperative agreements, or Other Transactions (OTs) (grants will not be awarded)

Agency Contacts:

o Technical POC: Bruce Draper, Program Manager, DARPA/I2O o BAA Email: PTG@darpa.mil o BAA Mailing Address:

DARPA/I2O

ATTN: HR001121S0015

675 North Randolph Street Arlington, VA 22203-2114 o I2O Solicitation Website: http://www.darpa.mil/work-with-us/opportunities http://www.darpa.mil/work-with-us/opportunities

HR001121S0015 PERCEPTUALLY-ENABLED TASK GUIDANCE (PTG) 4

PART II: FULL TEXT OF ANNOUNCEMENT

I. Funding Opportunity Description

DARPA is soliciting innovative fundamental research proposals in the area of perceptually-enabled task guidance. The goal is to enable mechanics, medics, and other specialists to perform tasks within and beyond their skillsets by providing just-in-time feedback and instructions for physical tasks. We envision artificial intelligence (AI) technology that perceives the environment, reasons about physical tasks, and models the user, all in real-time. The AI will leverage commercially-available sensor technologies to see what the user sees and hear what the user hears, and will provide contextually relevant information and instruction to the user using augmented reality (AR). DARPA is soliciting these proposals to investigate innovative approaches that enable revolutionary advances in science. Specifically excluded is research that primarily results in evolutionary improvements to the existing state of practice.

This Broad Agency Announcement (BAA) is being issued, and any resultant selection will be made, using procedures under Federal Acquisition Regulation (FAR) 6.102(d)(2) and 35.016.

Any negotiations and/or awards will use procedures under FAR 15.4. Proposals received as a result of this BAA shall be evaluated in accordance with evaluation criteria specified herein through a scientific review process.

DARPA posts BAAs on the System for Award Management, Contract Opportunities (Beta.Sam.Gov) website (https://beta.sam.gov/) and, when applicable, the Grants.gov website (https://www.grants.gov/).

The following information is provided for prospective proposers.

Introduction

The Perceptually-enabled Task Guidance (PTG) program will develop artificial intelligence (AI) technologies to help users perform complex physical tasks. Military personnel are expected to perform an increasing number of tasks and more complex tasks than ever before. Mechanics, for example, are asked to repair more types of increasingly sophisticated machines and platforms, and Medics are asked to perform more procedures over extended periods of time. The goal of the PTG program is to make users more versatile by expanding their skillset and more proficient by reducing their errors.

PTG will develop methods, techniques, and technology for artificially intelligent assistants that provide just-in-time visual and audio feedback to help users perform tasks, for example as depicted in Figure 1. Users of PTG assistants will wear both sensors (head-mounted cameras and microphones) that allow the assistant to see what they see and hear and what they hear, and users will also wear augmented reality (AR) headsets that allow assistants to

Figure 1: Perceptually-enabled Task Guidance https://beta.sam.gov/ https://www.grants.gov/

HR001121S0015 PERCEPTUALLY-ENABLED TASK GUIDANCE (PTG) 5

provide feedback in the form of generated speech and aligned graphics. The assistant will learn about tasks relevant to the user by ingesting knowledge from checklists, illustrated manuals, training videos, and other sources of information. The assistant will combine this task knowledge with a perceptual model of the environment to support mixed-initiative and task-focused user dialogs. These dialogs have at least three modes: (1) users may initiate the dialog by asking the assistant questions, for example “what do I do next?”, and should receive contextually-relevant answers; (2) when a user makes a critical mistake, for example skipping a step or performing a step incorrectly, the system should take the initiative to warn the user and suggest remedial action; and (3) when a task is new to a user, the assistant should walk the user through the steps of the task at an adaptive level of instruction appropriate to the user’s expertise. Note that assistants support not just one physical task, but a set of related tasks within a specialty, and that users may perform multiple tasks at once, interweaving the steps. For example, an assistant might know all the repair procedures in a manual and provide contextually-driven feedback to a skilled mechanic who is repairing multiple parts at once.

Program Scope PTG technology will exploit recent advances in deep learning for video and speech analysis, automated reasoning for task and/or plan monitoring, and augmented reality for human-computer interfaces. However, these technologies by themselves are insufficient. To create task guidance assistants, PTG is looking for novel approaches and integrated technologies that address four key (and interconnected) problems:

Knowledge Transfer. Assistants need to automatically acquire task knowledge from instructions intended for humans, with an emphasis on checklists, illustrated manuals, and training videos;

Perceptual Grounding. The objects, settings, actions, sounds, and words recognized by an assistant must be aligned with the terms it uses to describe and model tasks, so that observations can be mapped to its task knowledge;

Perceptual Attention. Assistants must pay attention to percepts that are relevant to current tasks, while ignoring extraneous stimuli. Assistants must also respond to unexpected, but salient, events that may alter a user’s goals or suggest a new task; and, User Modeling. Assistants must determine how much information to present to a user and when to do so. This requires an epistemic model of what the user knows, a physical model of what the user is doing, and a model of their attentional and emotional states.

These four problems are not independent of each other. To give just one example, there is a strong interaction between knowledge transfer and perceptual grounding. If knowledge transfer translates instructions into a small predetermined library of terms, then perceptual grounding becomes easy, whereas if knowledge transfer adopts whatever terms appear in a manual, perceptual grounding is challenging. Therefore, the PTG program does not divide task guidance technology into four separate research areas, but instead strong proposals will present integrated solutions that address all four problems.

As described in more detail below, the PTG program is seeking proposals in two Technical Areas (TAs). TA1 is for fundamental research into knowledge transfer, perceptual grounding, perceptual attention, and user modeling. TA2 is for integrated demonstrations of TA1 technologies on a militarily-relevant use case scenario in one of three broad areas: mechanical

HR001121S0015 PERCEPTUALLY-ENABLED TASK GUIDANCE (PTG) 6

repair (Mechanics), battlefield medicine (Medics), or pilot guidance (Pilots). TA3 is for testing and evaluation; proposals for TA3 are not solicited as part of this BAA.

Each proposal or abstract submitted in response to this BAA shall address only one TA.

Organizations may submit multiple abstracts and proposals to any one TA, or they may submit abstracts and proposals to multiple TAs, although ultimately no organization or individual will receive an award in both TA1 and TA2.

The goal of PTG is to develop software-based AI technology for perceptually-enabled task guidance. Development of sensors, computing hardware, and augmented reality headsets is outside the scope of PTG. PTG is not interested in supporting the development of new sensors.

Small and potentially wearable cameras and microphones are already available in the commercial market. Performers may exploit additional sensors, e.g., inertial sensors, as long as they are small (wearable), inexpensive, and widely available. Similarly, PTG will not support the development of computing hardware, which is already getting cheaper, smaller, and faster.

Proposers may use whatever commercial-off-the-shelf (COTS) hardware they need (e.g., Central Processing Units, Graphics Processing Units, Field-Programmable Gate Arrays, etc.), but the development of new hardware is outside the scope of the program. Finally, PTG is not interested in the development of AR headsets. AR headsets are already commercially available, with new and better headsets in development.

Program Structure

PTG is planned as a 48-month program divided into two 24-month phases. In Phase 1, TA1 performers will develop new technologies for knowledge transfer, perceptual grounding, perceptual attention, and user modeling in the proxy domain of cooking. They will show progress through technology demonstrations in months 12 and 24. TA2 performers, in Phase 1, will refine and collect labeled training data for an application use case of their choosing within the broad areas of mechanical maintenance (Mechanics), battlefield medicine (Medics), or pilot assistance (Pilots). They will create an initial prototype system that integrates existing technologies for perception, reasoning, and AR with new technologies developed by TA1 performers. Their initial prototypes will be demonstrated in month 18.

In the optional Phase 2, TA1 performers will continue to advance fundamental research, while working with TA2 performers and contributing to TA2 prototypes. The goal is an agile research methodology, analogous to agile software development methodologies, with rapid initial development followed by iterative improvement with tight interaction between TA1 and TA2 performers. In Phase 2, TA1 teams will iteratively improve technologies for knowledge transfer, perceptual grounding, perceptual attention, and user modeling in order to provide better user feedback for more tasks and more difficult tasks, driven in part by the needs of TA2 performers and TA2 prototypes. TA2 teams, on the other hand, will focus on the needs of end-users and create increasingly sophisticated military task guidance prototypes, where the improvements are driven by continuously integrating new advances in TA1 technology. In Phase 2, TA2 performers will demonstrate their prototype systems in months 30, 39, and 48 (see Schedule section below).

In Phase 2, TA1 and TA2 performers will be evaluated jointly. The final products of the PTG program will be the militarily-relevant TA2 prototypes. TA2 performers will be assessed both on how well their prototypes meet the needs of users and by how well they integrate technologies

HR001121S0015 PERCEPTUALLY-ENABLED TASK GUIDANCE (PTG) 7

from multiple TA1 performers. TA1 performers will be assessed by the impact of their technologies on TA2 prototypes. Note that DARPA will not direct TA1 performers to work with specific TA2 performers or vice versa. However, TA1 teams whose technology is not part of a TA2 prototype will be considered non-performing, as will TA2 teams that do not integrate technology from multiple TA1 performers.

Strong TA1 and TA2 proposals will present evidence or substantiated arguments to support why and how their proposed research approaches are likely to meet the quantitative milestones (described below in the Evaluation & Milestones section) for each phase. Both potential benefits (or performance payoff from a revolutionary technology development perspective) and risks related to the proposed approach should be defined.

TA1: Foundational Technologies

Performers in TA1 will develop foundational technologies via fundamental research approaches.

The goal is to create PTG systems that process first-person sensory streams, meaning input from head-mounted cameras and microphones, and interact with users through AR headsets. The PTG systems will observe both the environment and the actions of the user to infer (without being told) what tasks the user is performing and whether the user is performing those tasks correctly.

If the user asks a question, for example, “what should I do next?,” the PTG system will answer verbally and/or graphically based upon the current context. If the user appears unsure of what to do, the PTG system will interject with helpful information. If the user makes a critical mistake, either by performing actions in an impermissible order or by performing an action incorrectly, the PTG system will warn the user and suggest remedial steps. The goal is to create technology that makes users more versatile by helping them perform a wide range of complex physical tasks and more proficient by reducing mistakes.

Proposers to TA1 should describe novel approaches to creating perceptually-enabled task guidance technology. Proposed solutions will need to integrate perception, task-based reasoning, and human-machine communication, while at the same time addressing new issues that arise in the context of end-to-end task guidance, namely:

Knowledge Transfer. Generality is essential to PTG. DARPA is not interested in developing technology to support a single physical task, or even a dozen tasks. The goal is to create systems that can interpret instructions meant for humans with little or no human involvement. In the context of TA1 Phase 1, within the surrogate cooking domain, this initially means reading recipes from illustrated cookbooks and watching cooking videos. In Phase 2, within the military Pilots, Mechanics, and/or Medics use case context, it will mean reading checklists and illustrated military manuals, and watching training videos for the specialties identified by TA2 performers.

Perceptual Grounding. Human instructions often include domain specific terms that do not appear in general-purpose training sets. These terms may refer to hyper-specific objects, actions, or states, such as egg separator, sauté, or rue in cooking (surrogate domain context). Alternatively, these may be general terms that refer to a variety of objects, actions, or sounds. A simple term, such as bowl describes many objects with different sizes, color, and textures. Perceptual grounding is the challenge of learning to visually or aurally recognize the objects, settings, and actions used to model physical

HR001121S0015 PERCEPTUALLY-ENABLED TASK GUIDANCE (PTG) 8

tasks from limited training data, with little or no human interaction.

Perceptual Attention (and environmental modeling). Continuous video and audio streams produce vast amounts of data. Attention is the process by which a few percepts are selected for further processing in order to avoid overwhelming downstream components of the system. Percepts should be attended to if they are either relevant or salient. A percept is relevant if it is needed to monitor the current stage of a current task.

For example, in cooking a pan is relevant if the task is to sauté, but not if the task it to retrieve eggs from the refrigerator. A percept is salient if it might trigger a new, unexpected task. For example, falling objects are salient, if only because they may trigger an unexpected cleanup task. Included in this definition of attention is scene modeling. A new object entering a scene is generally salient, but head mounted cameras present continuously changing fields of view. By modeling the immediate environment, attention systems determine what objects are new, as opposed to merely reentering the field of view, and what objects are nearby, even when they cannot be seen at the moment.

User Modeling. Multi-level user models guide decisions about when and how much to communicate with the user. Physical user models, for example, can prevent a system from telling the user to grab an object they are already holding. Models of expertise prevent a system from using terms that the user doesn’t understand or providing unnecessary details. Stylistic models distinguish users who are reassured by being told they have performed a step correctly from those who find it annoying. Similarly, models of emotional and/or mental state may indicate whether a user is actively attending to the task or taking a brief break. Across all levels of abstraction, however, the goal of user monitoring in PTG is to guide mixed-initiative and multimedia dialogs with the user in order to make them more versatile and proficient without increasing their cognitive load.

Strong TA1 proposals should address all of the problems above in one form or another, but there are many ways to approach these interrelated problems, and some solutions may put more emphasis on some problems than others. DARPA is looking for novel and creative approaches, and will consider any proposed technology, so long as it can (1) ingest instructions meant for humans and (2) process first-person video and audio streams in order to (3) produce mixed-initiative dialogs and multimedia instruction.

At the task domain level, the PTG program is divided into two phases. TA1 Phase 1 research will be conducted in the surrogate cooking domain, where tasks are defined by recipes, instructions come in the form of illustrated cookbooks and cooking videos, and users are cooks of varying skill levels. Training data for the cooking domain is available through the Epic Kitchens data set (https://epic-kitchens.github.io/2020-100), which contains extensive hand-labeled first-person video and audio data. Proposers may choose to augment or replace the Epic Kitchens data with other sources of training data, but if they do they must share this data with the evaluation team and with all other PTG performers. We do not anticipate that TA1 performers will need to conduct extensive data collection in the proxy cooking domain, and TA1 proposals seeking funding for data collection in the cooking domain must provide a compelling justification.

TA1 Phase 2 research will be conducted in militarily-relevant domains defined by TA2 performers (see below). The TA2 performers will provide task descriptions and training data for their scenarios in the first year of the program. Therefore, strong TA1 proposals will describe https://epic-kitchens.github.io/2020-100

HR001121S0015 PERCEPTUALLY-ENABLED TASK GUIDANCE (PTG) 9

approaches that are general enough to be developed in the cooking domain and then transferred to new domains. In Phase 2, TA1 performers will be expected to work closely with TA2 performers to iteratively improve their technologies and integrate them into TA2 prototypes.

TA1 performers may choose to also continue work in the cooking domain in Phase 2; however, they will be evaluated according to the impact that their research has on TA2 prototypes (see Evaluation & Milestones section below for further details).

TA1 proposals must align with a base period covering Phase 1 (months 1 – 24) and with a Phase 2 option (months 24 – 48).

TA2: Prototype Development

TA2 performers will be tasked to produce militarily-relevant prototypes of perceptually-enabled task guidance systems to be used and evaluated by military service members. These prototypes should combine state-of-the-art perceptual, reasoning, and augmented reality technology with new technologies for knowledge transfer, perceptual grounding, perceptual attention, and user modeling developed by TA1 performers.

Strong TA2 proposals should define a military use case in one of three broad topic areas: (1) mechanical maintenance (Mechanics), (2) battlefield medicine (Medics), or (3) pilot assistance (Pilots). The use case should define a specific scenario; for example, a mechanic’s scenario might be to help junior Air Force mechanics maintain the landing gear and brakes of an F-16.

The use case should be precise enough to suggest specific tasks, as defined in manuals and/or training videos, and the proposal should explain how PTG technology will be integrated into existing procedures to accomplish these tasks. Strong proposals will include approaches involving current or former military specialists who are familiar with the use case and can evaluate the PTG prototype during its development to ensure military relevance.

In year 1 of Phase 1, TA2 performers will collect and label training data for their use case, and make this data available to TA1 performers. The labeled data must, therefore, be unclassified and not contain Controlled Unclassified Information (CUI). The data should include video and audio of multiple users performing 50 or more instances of each task for at least 50 tasks, and be labeled at the level of objects, actions, contexts (a.k.a. settings), words, and other sounds.

Overlapping this data collection effort, TA2 performers should create an initial PTG prototype system by month 18 of the program. This prototype should be capable of achieving the first TA2 demonstration milestone (see Evaluations & Milestones below). This prototype will include perception, reasoning, and AR components that may be modeled on pre-existing technology. It should also contain initial components for knowledge transfer, perceptual grounding, perceptual attention, and user modeling developed by TA1 performers, possibly re-implemented and/or modified to meet the needs of the TA2 performer’s prototype. Strong TA2 proposals will describe procedures for integrating new TA1 technologies into evolving TA2 prototypes, and will not propose to duplicate TA1 research efforts.

In Phase 2, DARPA anticipates TA2 performers will iteratively improve their prototypes to achieve the performance milestones scheduled for months 30, 39, and 48 (see Evaluation & Milestones below). During this time, TA2 performers are not expected to perform original research in the four TA1 problem areas (knowledge transfer, etc.). Instead they are expected to engage with three or more TA1 performers in what can be described as an agile research

HR001121S0015 PERCEPTUALLY-ENABLED TASK GUIDANCE (PTG) 10

methodology. The TA2 performer teams will concentrate on their prototypes and the end user experience, making their needs known to the TA1 teams and iteratively adopting improved TA1 technologies. TA2 performers will be evaluated based on the quality of their prototype, the success of their prototype in achieving milestones, and their ability to select and adopt the best TA1 technologies.

TA2 proposals must align with a base period covering Phase 1 (months 1 – 24) and with a Phase 2 option (months 24 – 48).

Evaluation & Milestones

TA1 and TA2 performers will be evaluated by a TA3 Government evaluation team, not solicited by this BAA, anticipated to be composed of experts from a Federally Funded Research and Development Center (FFRDC). To evaluate new technologies, the TA3 team will create specialized demonstration scenarios for the cooking use case and for the military use cases defined by TA2 performers, based on the descriptions in Table 1 below.

Scenario #1 Scenario #2 Scenario #3 Scenario #4 Description Monitor users performing one task, warn about mistakes.

Monitor users performing five interleaved tasks.

Walk users through 1 of 10 novel (to them) tasks.

Answer questions while monitoring users on up to 50 tasks.

Performance Target

Error recognition with F1 > 0.75

Error & task recognition with F1 > 0.75

Faster task completion, user acceptance

Faster tasks, fewer errors, user acceptance

Use Case TA1: cooking TA2: military

TA1: cooking TA2: military

TA1: cooking TA2: military

TA1: cooking TA2: military

Evaluator TA3 team member

TA3 team member

Military user (TA2 only)

Military user (TA2 only)

Schedule (in months)

TA1: 12

TA2: 18

TA1: 24

TA2: 30

TA1: 36 (opt.)

TA2: 39

TA1: 48 (opt.)

TA2: 48

Table 1: Demonstration Scenarios

In Phase 1, TA1 performers are expected to make progress in knowledge transfer, perceptual grounding, perceptual attention, and user modeling technologies as demonstrated in the context of end-to-end systems in the cooking domain, applied to scenario 1 (month 12) and scenario 2 (month 24). These technology demonstrations are the Phase 1 milestones for TA1 performers. In Phase 2, TA1 performers are expected to work with TA2 performers and contribute technology to TA2 prototype systems. TA1 performers will be evaluated according to their contributions to TA2 demonstrations. In addition, TA1 performers have the option of demonstrating new enhancements of their technology on versions of scenarios 3 and 4 in the cooking domain, but such optional demonstrations are not replacements for contributing technology to TA2 demonstrations.

TA2 performers have two milestones in Phase 1 (see Schedule and Figure 2 below). The first is to produce labeled training data for their use case and a use case description that can be distributed to TA1 performers. Ideally, the training data should support the evolving training requirements of TA1 performers. Since is it difficult to know in advance exactly what these

HR001121S0015 PERCEPTUALLY-ENABLED TASK GUIDANCE (PTG) 11

requirements will be, the Government evaluation team will evaluate the TA2 training data according to how well it supports baseline deep learning algorithms for object, scene, sound, and activity recognition, as selected by the TA3 Government evaluation team.

The second milestone is to create the first prototype PTG system for their selected use case. This prototype should be able to demonstrate success in scenario 1 by month 18, where the users are members of the Government evaluation team. This first PTG prototype (and all subsequent PTG prototypes) should incorporate technology for knowledge transfer, perceptual grounding, perceptual attention, and user modeling developed by three or more TA1 performers.

In Phase 2, TA2 performers have three milestones, all defined in terms of their iteratively improving prototypes. Prototypes should succeed in scenarios 2, 3, and 4 by months 30, 39, and 48 respectively. In scenario 2, a member of the Government evaluation team will act as the user;

in scenarios 3 and 4, military specialists selected by the TA3 Government evaluation team will act as the users.

Schedule

PTG is planned as a 48-month project divided into two 24-month phases, as shown in Figure 2 below. In Phase 1, TA1 performers develop new technologies in the surrogate cooking domain while TA2 performers work in their selected military domain. In Phase 2, TA1 and TA2 teams are expected to cooperate, with technology being evaluated in the TA2 military application domains. No recompete or additional proposal will be needed for Phase 2 contract options;

performers will be judged on the quality of their Phase 1 work and availability of funds.

Figure 2: PTG Schedule & Milestones

TA2 teams will present their selected military use cases to the TA1 performers at the program kickoff. At the 12-month mark, TA2 performers will deliver training data for their military use case data to TA1 performers, and begin to integrate TA1 technology into their first prototype. As discussed above, TA2 milestones are prototype demonstrations in months 18, 30, 39, and 48.

HR001121S0015 PERCEPTUALLY-ENABLED TASK GUIDANCE (PTG) 12

All performers are expected to attend the kickoff meeting followed by Principal Investigator (PI) meetings every six months. These meetings are planned to be held at different locations throughout the country, with the kickoff meeting being in Arlington, VA. For budgeting purposes, assume half the meetings will be held in Arlington and half in Northern California, although meeting locations may vary. DARPA reserves the right to conduct some or all meetings virtually if necessitated by circumstances.

Deliverables

The primary deliverables for this program are the following:

TA1 technology transfers to TA2 after each of the 12, 24, 36, and 48 month demos (including ongoing interactions with TA2 performers to support integration)

TA2 military use cases delivered to TA1 performers no later than month 6

TA2 data collections delivered to TA1 performers by month 12

TA2 prototype systems demonstrated and delivered to TA3 Government evaluators in months 18, 30, 39, and 48

All performers (TA1, TA2, and TA3) shall be required to provide the following deliverables via DARPA’s Technical-Financial Information Management System (TFIMS) database:

Technical briefings and reports - Kickoff presentations with changes or updates shall be submitted within 1 month of the program kickoff meeting. Principal Investigator Meeting briefings (roughly one every six months over the duration of the program).

Quarterly progress reports - A quarterly progress report describing progress made, resources expended, and issues requiring the attention of the Government team shall be provided within 15 calendar days of the end of each Department of Defense fiscal year quarter

Monthly financial reporting, including spend plans Final report - The final report shall concisely summarize the effort

Intellectual Property

A key goal of the program is to establish an open, standards-based, multi-source, plug-and-play architecture that allows for interoperability and integration. This includes the ability to easily add, remove, substitute, and modify software and hardware components. This will facilitate rapid innovation by providing a base for future users or developers of program technologies and deliverables. Therefore, it is desired that all noncommercial software (including source code), software documentation, hardware designs and documentation, and technical data generated by the program be provided as open source deliverables to the Government, as lesser rights may adversely impact the lifecycle costs of affected items, components, or processes.

Intellectual property rights asserted by proposers are strongly encouraged to be aligned with open source regimes. Any proposer claiming the use of proprietary technology or choosing to explicitly exclude their technology from the open source regime, will need to provide adequate justification in their proposal, and will need to describe how use of the proprietary technology

HR001121S0015 PERCEPTUALLY-ENABLED TASK GUIDANCE (PTG) 13

will be granted to other performers in the project for the purposes of the project. See Section VI.B.1 for more details on intellectual property.

II. Award Information

A. Awards

Multiple awards are anticipated (roughly six in TA1 totaling approximately $30 million and two in TA2 totaling approximately $10 million), contingent upon the number and quality of proposals received and the availability of funds. The level of funding for individual awards made under this solicitation has not been predetermined and will depend on the quality of the proposals received and the availability of funds. Awards will be made to proposers whose proposals are determined to be the most advantageous to the Government, all factors considered, including the potential contributions of the proposed work, overall funding strategy, and availability of funding. See Section V for further information.

The Government reserves the right to:

select for negotiation all, some, one, or none of the proposals received in response to this solicitation;

make awards without discussions with proposers;

conduct discussions with proposers if it is later determined to be necessary;

segregate portions of resulting awards into pre-priced options;

accept proposals in their entirety or to select only portions of proposals for award;

fund proposals in increments and/or with options for continued work at the end of one or more phases;

request additional documentation once the award instrument has been determined (e.g., representations and certifications); and remove proposers from award consideration should the parties fail to reach agreement on award terms within a reasonable time or the proposer fails to provide requested additional information in a timely manner.

Proposals selected for award negotiation may result in a procurement contract, cooperative agreement, or Other Transaction (OT) depending upon the nature of the work proposed, the required degree of interaction between parties, and other factors. Grants will NOT be awarded under this program.

Proposers looking for innovative, commercial-like contractual arrangements are encouraged to consider requesting Other Transactions. To understand the flexibility and options associated with Other Transactions, consult http://www.darpa.mil/work-with-us/contract-management#OtherTransactions.

In accordance with 10 U.S.C. § 2371b(f), the Government may award a follow-on production contract or Other Transaction (OT) for any OT awarded under this solicitation if: (1) that participant in the OT, or a recognized successor in interest to the OT, successfully completed the entire prototype project provided for in the OT, as modified; and (2) the OT provides for the award of a follow-on production contract or OT to the participant, or a recognized successor in interest to the OT.

In all cases, the Government contracting officer shall have sole discretion to select award instrument type, regardless of instrument type proposed, and to negotiate all instrument terms http://www.darpa.mil/work-with-us/contract-management#OtherTransactions http://www.darpa.mil/work-with-us/contract-management#OtherTransactions

HR001121S0015 PERCEPTUALLY-ENABLED TASK GUIDANCE (PTG) 14

and conditions with selectees. DARPA will apply publication or other restrictions, as necessary, if it determines that the research resulting from the proposed effort will present a high likelihood of disclosing performance characteristics of military systems or manufacturing technologies that are unique and critical to defense. Any award resulting from such a determination will include a requirement for DARPA permission before publishing any information or results on the program. For more information on publication restrictions, see the section below on Fundamental Research.

B. Fundamental Research

It is DoD policy that the publication of products of fundamental research will remain unrestricted to the maximum extent possible. National Security Decision Directive (NSDD) 189 defines fundamental research as follows:

‘Fundamental research’ means basic and applied research in science and engineering, the results of which ordinarily are published and shared broadly within the scientific community, as distinguished from proprietary research and from industrial development, design, production, and product utilization, the results of which ordinarily are restricted for proprietary or national security reasons.

As of the date of publication of this solicitation, the Government expects that program goals as described herein may be met by proposers intending to perform fundamental research and does not anticipate applying publication restrictions of any kind to individual awards for fundamental research that may result from this solicitation. Notwithstanding this statement of expectation, the Government is not prohibited from considering and selecting research proposals that, while perhaps not qualifying as fundamental research under the foregoing definition, still meet the solicitation criteria for submissions. If proposals are selected for award that offer other than a fundamental research solution, the Government will either work with the proposer to modify the proposed statement of work to bring the research back into line with fundamental research or else the proposer will agree to restrictions in order to receive an award.

Proposers should indicate in their proposal whether they believe the scope of the research included in their proposal is fundamental or not. While proposers should clearly explain the intended results of their research, the Government shall have sole discretion to determine whether the proposed research shall be considered fundamental and to select the award instrument type. Appropriate language will be included in resultant awards for non-fundamental research to prescribe publication requirements and other restrictions, as appropriate. This language can be found at http://www.darpa.mil/work-with-us/additional-baa.

For certain research projects, it may be possible that although the research to be performed by a potential awardee is non-fundamental research, its proposed subawardee’s effort may be fundamental research. It is also possible that the research performed by a potential awardee is fundamental research while its proposed subawardee’s effort may be non-fundamental research.

In all cases, it is the potential awardee’s responsibility to explain in its proposal which proposed efforts are fundamental research and why the proposed efforts should be considered fundamental research.

C. Disclosure of Information and Compliance with Safeguarding Covered Defense Information Controls

The following provisions and clause apply to all solicitations and contracts; however, the definition of “controlled technical information” clearly exempts work considered fundamental http://www.darpa.mil/work-with-us/additional-baa

HR001121S0015 PERCEPTUALLY-ENABLED TASK GUIDANCE (PTG) 15

research and therefore, even though included in the contract, will not apply if the work is fundamental research.

DFARS 252.204-7000, “Disclosure of Information” DFARS 252.204-7008, “Compliance with Safeguarding Covered Defense Information Controls” DFARS 252.204-7012, “Safeguarding Covered Defense Information and Cyber Incident Reporting” The full text of the above solicitation provision and contract clauses can be found at http://www.darpa.mil/work-with-us/additional-baa#NPRPAC.

Compliance with the above requirements includes the mandate for proposers to implement the security requirements specified by National Institute of Standards and Technology (NIST) Special Publication (SP) 800-171, “Protecting Controlled Unclassified Information in Nonfederal Information Systems and Organizations” (see https://doi.org/10.6028/NIST.SP.800-171r1) and DoDI 8582.01 that are in effect at the time the solicitation is issued.

For awards where the work is considered fundamental research, the contractor will not have to implement the aforementioned requirements and safeguards. However, should the nature of the work change during performance of the award, work not considered fundamental research will be subject to these requirements.

III. Eligibility Information

A. Eligible Applicants

DARPA welcomes engagement from all responsible sources capable of satisfying the Government's needs, including academia (colleges and universities); businesses (large, small, small disadvantaged, etc.); other organizations (including non-profit); other entities (foreign, domestic, and government); FFRDCs; minority institutions; and others.

DARPA welcomes engagement from non-traditional sources in addition to current DARPA performers.

1. Federally Funded Research and Development Centers (FFRDCs) and Government Entities

a. FFRDCs FFRDCs are subject to applicable direct competition limitations and cannot propose to this solicitation in any capacity unless they meet the following conditions. (1) FFRDCs must clearly demonstrate that the proposed work is not otherwise available from the private sector. (2) FFRDCs must provide a letter, on official letterhead from their sponsoring organization, that (a) cites the specific authority establishing their eligibility to propose to Government solicitations and compete with industry, and (b) certifies the FFRDC’s compliance with the associated FFRDC sponsor agreement’s terms and conditions. These conditions are a requirement for FFRDCs proposing to be awardees or subawardees.

b. Government Entities Government Entities (e.g., Government/National laboratories, military educational institutions, etc.) are subject to applicable direct competition limitations. Government Entities must clearly demonstrate that the work is not otherwise available from the http://www.darpa.mil/work-with-us/additional-baa#NPRPAC https://doi.org/10.6028/NIST.SP.800-171r1

HR001121S0015 PERCEPTUALLY-ENABLED TASK GUIDANCE (PTG) 16

private sector and provide written documentation citing the specific statutory authority and contractual authority, if relevant, establishing their ability to propose to Government solicitations and compete with industry. This information is required for Government Entities proposing to be awardees or subawardees.

c. Authority and Eligibility At the present time, DARPA does not consider 15 U.S.C. § 3710a to be sufficient legal authority to show eligibility. While 10 U.S.C.§ 2539b may be the appropriate statutory starting point for some entities, specific supporting regulatory guidance, together with evidence of agency approval, will still be required to fully establish eligibility. DARPA will consider FFRDC and Government Entity eligibility submissions on a case-by-case basis; however, the burden to prove eligibility for all team members rests solely with the proposer.

2. Foreign Participation Non-U.S. organizations and/or individuals may participate to the extent that such participants comply with any necessary nondisclosure agreements, security regulations, export control laws, and other governing statutes applicable under the circumstances.

B. Organizational Conflicts of Interest

FAR 9.5 Requirements In accordance with FAR 9.5, proposers are required to identify and disclose all facts relevant to potential OCIs involving the proposer’s organization and any proposed team member (subawardee, consultant). Under this Section, the proposer is responsible for providing this disclosure with each proposal submitted to the solicitation. The disclosure must include the proposer’s, and as applicable, proposed team member’s OCI mitigation plan. The OCI mitigation plan must include a description of the actions the proposer has taken, or intends to take, to prevent the existence of conflicting roles that might bias the proposer’s judgment and to prevent the proposer from having unfair competitive advantage. The OCI mitigation plan will specifically discuss the disclosed OCI in the context of each of the OCI limitations outlined in FAR 9.505-1 through FAR 9.505-4.

Agency Supplemental OCI Policy In addition, DARPA has a supplemental OCI policy that prohibits contractors/performers from concurrently providing Scientific Engineering Technical Assistance (SETA), Advisory and Assistance Services (A&AS) or similar support services and being a technical performer.

Therefore, as part of the FAR 9.5 disclosure requirement above, a proposer must affirm whether the proposer or any proposed team member (subawardee, consultant) is providing SETA, A&AS, or similar support to any DARPA office(s) under: (a) a current award or subaward; or (b) a past award or subaward that ended within one calendar year prior to the proposal’s submission date.

If SETA, A&AS, or similar support is being or was provided to any DARPA office(s), the proposal must include:

The name of the DARPA office receiving the support;

The prime contract number;

Identification of proposed team member (subawardee, consultant) providing the support; and An OCI mitigation plan in accordance with FAR 9.5.

Government Procedures

HR001121S0015 PERCEPTUALLY-ENABLED TASK GUIDANCE (PTG) 17

In accordance with FAR 9.503, 9.504 and 9.506, the Government will evaluate OCI mitigation plans to avoid, neutralize or mitigate potential OCI issues before award and to determine whether it is in the Government’s interest to grant a waiver. The Government will only evaluate OCI mitigation plans for proposals that are determined selectable under the solicitation evaluation criteria and funding availability.

The Government may require proposers to provide additional information to assist the Government in evaluating the proposer’s OCI mitigation plan.

If the Government determines that a proposer failed to fully disclose an OCI; or failed to provide the affirmation of DARPA support as described above; or failed to reasonably provide additional information requested by the Government to assist in evaluating the proposer’s OCI mitigation plan, the Government may reject the proposal and withdraw it from consideration for award.

C. Cost Sharing/Matching

Cost sharing is not required; however, it will be carefully considered where there is an applicable statutory condition relating to the selected funding instrument (e.g., OTs under the authority of 10 U.S.C. § 2371).

For more information on potential cost sharing requirements for Other Transactions for Prototype, see https://www.darpa.mil/work-with-us/contract-management#OtherTransactions.

D. Other Eligibility Requirements

1. Ability to Receive Awards in Multiple Technical Areas (TAs) - Conflicts of Interest No single proposal may address more than one TA. Proposers may submit proposals to both TA1 and TA2, however, proposers selected for TA2 cannot be selected for any portion of TA1, or vice versa, whether as a prime, subcontractor, or in any other capacity from an organizational to individual level. This is to avoid conflicts of interest as the TA2 performers will select which TA1 technologies they will adopt, and TA1 performers will be evaluated, in part, on whether their technologies are incorporated into TA2 prototypes. If an institution or individual submits proposals to both TA1 and TA2, the decision as to which, if either, proposal to select for award will be at the discretion of the Government.

IV. Application and Submission Information

A. Address to Request Application Package

This document contains all information required to submit a response to this solicitation. No additional forms, kits, or other materials are needed except as referenced herein. No request for proposal (RFP) or additional solicitation regarding this opportunity will be issued, nor is additional information available except as provided at the System for Award Management, Contract Opportunities website (https://beta.sam.gov), the Grants.gov website (https://www.grants.gov/), or referenced herein.

B. Content and Form of Application Submission

1. Abstracts https://www.darpa.mil/work-with-us/contract-management#OtherTransactions

HR001121S0015 PERCEPTUALLY-ENABLED TASK GUIDANCE (PTG) 18

Proposers are highly encouraged to submit an abstract in advance of a proposal to minimize effort and reduce the potential expense of preparing an out of scope proposal. The abstract provides a synopsis of the proposed project, including brief answers to the following questions:

What is the proposed work attempting to accomplish or do?

How is it done today, and what are the limitations?

Who will care and what will the impact be if the work is successful?

How much will it cost, and how long will it take?

DARPA will respond to abstracts with a statement as to whether DARPA is interested in the idea. If DARPA does not recommend the proposer submit a full proposal, DARPA will provide feedback to the proposer regarding the rationale for this decision. Regardless of DARPA’s response to an abstract, proposers may submit a full proposal. DARPA will review all conforming full proposals using the published evaluation criteria and without regard to any comments resulting from the review of an abstract.

Abstract Format: Abstracts shall not exceed a maximum of 5 pages including the cover sheet and all figures, tables, and charts. The page limit does not include a submission letter (optional).

Reminder – Each abstract submitted in response to this BAA shall address only one TA.

Organizations may submit multiple abstracts to any one TA, and/or they may submit abstracts to multiple TAs.

All pages shall be formatted for printing on 8-1/2 by 11-inch paper with 1-inch margins and font size not smaller than 12 point. Font sizes of 8 or 10 point may be used for figures, tables, and charts. Document files must be in .pdf, .odx, .doc, .docx, .xls, or .xlsx formats.

Submissions must be written in English. All pages should be numbered.

Abstracts must include the following components:

Cover Sheet: Provide…