HR001119S0030.pdf

HR001119S0030 CAML 1

Broad Agency Announcement Competency-Aware Machine Learning (CAML)

Defense Sciences Office

HR001119S0030

February 19, 2019

HR001119S0030 CAML 2

Table of Contents I. Funding Opportunity Description

A. Introduction B. Background C. Program Description/Scope D. Program Structure E. Technical Area Descriptions F. Schedule/Milestones G. Deliverables

II. Award Information A. General Award Information B. Fundamental Research

III. Eligibility Information A. Eligible Applicants B. Organizational Conflicts of Interest C. Cost Sharing/Matching

IV. Application and Submission Information A. Address to Request Application Package B. Content and Form of Application Submission C. Submission Dates and Times D. Funding Restrictions E. Other Submission Requirements

V. Application Review Information A. Evaluation Criteria B. Review and Selection Process C. Federal Awardee Performance and Integrity Information (FAPIIS)

VI. Award Administration Information A. Selection Notices B. Administrative and National Policy Requirements C. Reporting

VII. Agency Contacts VIII. Other Information

A. Frequently Asked Questions (FAQs) B. Collaborative Efforts/Teaming C. Proposers Day

BAA Attachments:

Attachment A: ABSTRACT SUMMARY SLIDE TEMPLATE Attachment B: ABSTRACT TEMPLATE Attachment C: PROPOSAL SUMMARY SLIDE TEMPLATE Attachment D: PROPOSAL TEMPLATE VOLUME 1 TECHNICAL & MANAGEMENT VOLUME Attachment E: PROPOSAL TEMPLATE VOLUME 2 COST VOLUME Attachment F: PROPOSAL TEMPLATE VOLUME 2 COST SUMMARY SPREADSHEET Attachment F-2: PROPOSAL VOLUME 2 COST BREAKDOWN Attachment G: PROPOSAL TEMPLATE VOLUME 3 ADMINISTRATIVE & NATIONAL POLICY

REQUIREMENTS VOLUME

HR001119S0030 CAML 3

PART I: OVERVIEW INFORMATION

Federal Agency Name: Defense Advanced Research Projects Agency (DARPA), Defense Sciences Office (DSO)

Funding Opportunity Title: Competency-Aware Machine Learning (CAML)

Announcement Type: Initial Announcement

Funding Opportunity Number: HR001119S0030

Catalog of Federal Domestic Assistance (CFDA) Number(s): 12.910 Research and Technology Development

Dates (All times listed herein are Eastern Time.)

o Posting Date: February 19, 2019 o Proposers Day: February 20, 2019. See Section VIII.C.

o Abstract Due Date: March 6, 2019, 4:00 p.m.

o FAQ Submission Deadline: April 5, 2019, 4:00 p.m. See Section VIII.A.

o Full Proposal Due Date: April 22, 2019, 4:00 p.m.

Anticipated Individual Awards: DARPA anticipates multiple awards.

Types of Instruments that may be awarded: Procurement contracts, cooperative agreements or other transactions

Agency contacts o Technical POC: Dr. Jiangying Zhou, Program Manager, DARPA/DSO o BAA Email: CAML@darpa.mil o BAA Mailing Address:

DARPA/DSO

ATTN: HR001119S0030

675 North Randolph Street Arlington, VA 22203-2114 o DARPA/DSO Opportunities Website: http://www.darpa.mil/work-with-us/opportunities

Frequently Asked Questions (FAQ): FAQs for this solicitation may be viewed on the DARPA/DSO Opportunities Website. See Section VIII.A for further information.

http://www.grants.gov/?tFilter=&oFilter=2&sort=name?tFilter=&oFilter=2&sort=name http://www.grants.gov/?tFilter=&oFilter=2&sort=name?tFilter=&oFilter=2&sort=name

HR001119S0030 CAML 4

PART II: FULL TEXT OF ANNOUNCEMENT

I. Funding Opportunity Description

This Broad Agency Announcement (BAA) constitutes a public notice of a competitive funding opportunity as described in Federal Acquisition Regulation (FAR) 6.102(d)(2) and 35.016 as well as 2 CFR § 200.203. Any resultant negotiations and/or awards will follow all laws and regulations applicable to the specific award instrument(s) available under this BAA, e.g., FAR

15.4 for procurement contracts.

A. Introduction The Defense Sciences Office (DSO) at the Defense Advanced Research Projects Agency (DARPA) is soliciting innovative research proposals in the area of competency-awareness machine learning, whereby an autonomous system can self-assess its task competency and strategy and express both in a human-understandable form. This competency-awareness capability contributes to the goal of transforming autonomous systems from tools into trusted, collaborative partners. The resulting competency-aware machine learning systems will enable machines to control their behaviors to match user expectations and allow human operators to quickly and accurately gain insight into a system’s competence in complex, time-critical, dynamic environments. The Competency-Aware Machine Learning (CAML) program will, in this way, improve the efficiency and effectiveness of human-machine teaming. Proposed research should investigate innovative approaches that enable revolutionary advances in science.

DSO will exclude proposals that propose evolutionary improvement to the existing state of practice.

B. Background In order to transform machine learning systems from tools into partners, users need to trust their machine counterparts. One component to building trust is understanding machine competence (i.e., an accurate insight into the machine’s skills, experience, and reliability). State-of-the-art machine learning systems operate in a complex state space, and continuously develop emergent behaviors based on their internal structures and learning experiences. While such systems can perform well when their behaviors are applied in contexts that similarly match their structures and learning experiences, they are unable to communicate their task strategies, the completeness of their training relative to a given task, the factors that may influence their actions, or their likelihood to succeed under specific conditions.

Therefore, current systems require users to rely on traditional verification and validation (V&V) methodologies to predict a machine system’s competency. The complexity of learning systems and operating environments has made such V&V increasingly unrealistic for end users. This leads to a poor “economy of force” as human operators must mediate the system’s actions. This deficiency is especially significant in the Department of Defense (DoD) domain where machine systems often deal with high-stake decisions, and must cope with highly dynamic, fast-changing conditions.

CAML addresses this challenge by enabling learning systems that are self-aware of competency, have knowledge of learned abilities and the conditions under which those abilities were learned, HR001119S0030 CAML 5 and have knowledge of resultant task strategies and the situations for which those strategies are applicable.

CAML will significantly improve human-machine teaming capabilities and the task synergies expected of autonomous systems. By creating a fundamentally new machine learning approach, CAML will facilitate mission planning by choosing the appropriate available asset based on task requirements, determining the level of autonomy to be granted based on an asset-proposed strategy, and controlling behaviors to adapt for operating conditions.

C. Program Description/Scope The goal of CAML is to develop a competency-aware machine learning framework to support transitioning machine learning systems from tools into trusted partners. Achieving this goal requires the development of new elements in machine learning, including memory mechanisms, knowledge abstraction and representation, and behavior self-modeling.

CAML is interested in a diverse range of DoD-relevant machine learning problems including, but not limited to, learning systems for object recognition, robotic navigation, action planning, and decision-making. Developed technologies will be tested using realistic test vignettes with actual applications. Initial testing will be performed on machine learning platforms and applications chosen by the proposer, while final tests will involve Government experimental platforms and applications. CAML will evaluate the accuracy of the machine’s communication of its competency, not the human-perception of the machine’s competency.

Methodologies developed by proposers should be generalizable and applicable to a broad class of machine learning systems. Handcrafted solutions tailored to specific machine learning applications are not of interest. Since the emphasis is on a competency-aware framework, the development of new machine learning platforms is outside of the program scope.

D. Program Structure CAML will be a 48-month program divided into two phases and four technical areas (TA). Phase I will be a 36-month basic research period encompassing TA1-3, with milestones and evaluations. CAML performers will be expected to make measurable progress in individual TAs, but the performance of the complete, integrated system is of critical importance. Proposers must propose their own machine learning problems for experimentation and demonstration in Phase I.

Phase II will be a 12-month technology demonstration effort encompassing TA4 and will focus on validating the utility of the Phase I capabilities for DoD-relevant applications using Government platforms.

Summary descriptions of the four TAs are as follows. Details of the TAs are described in Section E. Proposals must address all four technical areas.

TA1: Self-knowledge of Experiences, will develop mechanisms for learning systems to discover conditions encountered during operation, and maintain a memory of experiences.

HR001119S0030 CAML 6

TA2: Self-knowledge of Task Strategies, will enable a machine learning system to analyze its task behaviors, summarize them into generalized patterns (task strategies), and identify dependencies that control its task behavior.

TA3: Competency-Aware Learning, will integrate the component technologies developed in TA1 and TA2 to establish a competency-aware learning framework, communicated with machine-derived, human-understandable, competency statements. TA3 will conclude with an experimental demonstration on a proposer-provided platform.

TA4: Capability Demonstrations will be a demonstration of the developed competency-aware machine learning systems on DoD platforms. TA4 will be addressed in Phase II.

It is at the discretion of the Government to exercise the Phase II option for all, some, or none, of the Phase 1 awardees.

E. Technical Area Descriptions TA1: Self-Knowledge of Experiences CAML defines experiences as any situation encountered during task learning and execution. The subset of experiences that affect task behavior are referred to here as meta-knowledge. The objective of TA1 is to introduce a memory mechanism and enable learning systems to capture and encode experiences and meta-knowledge. As humans, we remember our experiences; we identify similar situations/conditions and our previous success or failure to assess our ability to perform tasks at hands. TA1 seeks to enable machine learning systems to make similar evaluations of competency by developing the ability to maintain a memory of their previous task experiences.

Mechanisms developed in TA1 should support modeling of user-defined experience elements as well as those that are self-discovered. The former are specified prior to learning. The latter includes meta-knowledge that is emergent from the discovery of the dependencies of task behaviors (see TA2). Proposers should discuss how their model would both incorporate the pre-defined experience elements and interact with the mechanisms developed in TA2 to dynamically incorporate the emergent meta-knowledge.

Proposers should discuss the types of experiences relevant to their selected machine learning problems, how their approach would condense the experience elements into a compact representation in memory, and how the representations may be updated, refined, and potentially pruned. Proposers should also discuss how their approaches would monitor the experiences in the input data and characterize their occurrence during task learning and execution.

A challenge for encoding emergent meta-knowledge elements is that they are unforeseen and no prior representation exists. Proposers should discuss how their approach would enable learning such new experience elements. Potential machine learning techniques that proposers might consider for enabling continuous learning of new experience elements include, but are not limited to, incremental architectures, few-shot learning, multitask learning, and compositional modular networks.

HR001119S0030 CAML 7

Proposers should discuss how their experience memory would allow the retrieval of specific meta-knowledge to support competency modeling in TA2. Potential approaches for memory representation and retrieval include, but are not limited to, index-code based sparse representation, adaptive memory techniques, and differential neural computing techniques.

TA2: Self-Knowledge of Task Strategies TA2 will develop mechanisms to enable a machine learning system to analyze its task behaviors, summarize them into generalized patterns (i.e. task strategies), and identify the meta-knowledge elements that control task behavior. The objective is to establish descriptions of conditional task behaviors that provide concise insight into learning system behaviors and their internal/external dependencies. For example, proposers might use symbolic rules to express a learned task behavior and corresponding dependencies. In the case of a self-driving car, an example of such a rule could be “IF dirt road, THEN drive slowly.”

There are potentially a very large number of complex and interdependent learned task behaviors within a machine learning system. Proposers should discuss how their approach would delineate different task strategies and discover the meta-knowledge dependencies. Proposers may consider, but are not limited to, techniques such as inductive reasoning, rule abstraction, and on-going research in the area of symbolic knowledge extraction, to derive task strategies.

A challenge to developing machine self-knowledge of task strategies is determining the appropriate level of semantic abstraction for task behavior that balances the specificity and the level of generality. Proposers should discuss the types of task behaviors to be explored with respect to their selected learning problems and define the vocabulary and representation schemes for their abstraction. Proposers should also discuss how their task strategy representations would deal with potentially fragmented, incomplete, ambiguous, and overlapping behavior patterns.

Proposals should describe the interaction between TA1 and TA2 mechanisms such that new meta-knowledge elements are identified and integrated into the experience model.

TA3: Competency-Aware Learning TA3 will develop new machine learning systems that can assess their own competency for a given task and express that competency in a human-understandable form. TA3 seeks to achieve these goals by integrating machine analysis of its experiences (TA1) and task strategies (TA2) to determine competency, and providing machine-derived, human-understandable, descriptions of its task strategies and expected performance via competency statements.

Competency statements should facilitate accurate human insight into machine system capabilities and enable machine systems to achieve self-maintenance of performance based on a human-partner’s expectations. Competency statements could be both quantitative and qualitative. There could be multiple competency statements for a given task. An example of a competency statement is the Receiver Operating Characteristic (ROC) curve for a target recognition/ classification task. Another example could be measures of completeness or level of experience for tasks (e.g. frequency of task execution under given condition). Proposers should describe the forms of competency statements that are appropriate for their machine learning problems.

HR001119S0030 CAML 8

Competency-aware learning requires the integration of several different forms of processing at different time scales. Task-learning responds to sensory stimuli over short-timescales while meta-knowledge extraction and task rule consolidation occur over longer timescales. New machine learning frameworks need to be developed to support such diverse computing.

Proposals should discuss how their learning frameworks would allow for an efficient management of complex interactions between task learning, experience modeling, and task strategy discovery. Proposals should discuss how the learning system would provide a semantic interpretation of the emergent meta-knowledge.

Proposers are expected to describe the methods that their learning system will use to represent its competency internally. Proposers should also discuss methods the machine will use to determine what information to communicate to a human user and the form of that communication.

TA3 will conclude with a demonstration of competency-aware machine learning using performer-provided platforms and applications. Proposers should discuss the platforms they intend to use and the applications that will be demonstrated. Proposers should discuss how they plan to integrate their competency-aware machine learning system into their platforms. Proposals should describe demonstration plans and vignettes. Proposed vignettes should align with defense-relevant vignettes, examples of which are given in the TA4 description below.

Proposers should describe how they plan to test and evaluate the competency statements against the program metrics outlined in Section F. Additionally, proposers are encouraged to propose additional evaluation metrics to demonstrate the effectiveness of their model. Proposers are expected to develop an appropriate interface to support the examination and evaluation of the competency statements. In particular, the competency-aware machine learning system must provide evidence of the correctness of the task strategy stated in the competency statement.

TA4: Capability Demonstration

TA4 in Phase II will focus on demonstrations of the developed competency-aware machine learning on DoD platforms. The objective is to evaluate and validate the utility of the competency-aware learning system in operations relevant to DoD missions.

The specific experimental platforms and the details of the demonstration plan will be developed during Phase I by the Government team and will be provided to performers in Phase II. DARPA envisions that the TA4 demonstrations will involve multiple DoD platforms. Examples of potential platforms include autonomous ground resupply vehicle platforms, UAV intelligence, surveillance, reconnaissance (ISR) platforms, and mission planning systems.

The performance demonstrations will require performers to integrate their machine learning technology with the provided Government experimental platforms, and may require working with Government users to adapt their competency components to Government applications. It is therefore important that the competency-aware learning architectures developed in Phase I have well-designed APIs and modular structures. Proposers should discuss how their system design would facilitate integration into Government test platforms. Proposers should discuss their

HR001119S0030 CAML 9

porting and integration plan. Proposers should also discuss the type of human-user interaction controls their systems will provide to facilitate demonstrations.

Proposers must provide a draft Statement of Work and rough order of magnitude (ROM) estimate of the level-of-effort for the Phase II TA4 Capability Demonstration as part of their proposal submission. For planning purposes, proposers may use the autonomous ground resupply vehicles as the reference platform and provide the ROM estimate of the development effort supporting all three demonstration vignettes discussed below. Proposers whose machine learning problems do not fit the autonomous vehicle platform may use an alternative platform from the examples discussed above or another DoD relevant platform as the reference platform.

In the following, DARPA provides a few scenarios illustrating the potential demonstration vignettes contemplated by the program. However, DARPA emphasizes that these only serve to help proposers ground their proposal plan for Phase II. The actual performance demonstrations may be different from any of these cases.

Persistent-behaviors This scenario involves demonstrating a competency-aware machine learning system’s ability to maintain desired operating objectives. For example, an ISR operator may desire that a specific target detection probability (PD) is maintained. The competency-aware system would monitor the condition of the environment and inform the operator when the conditions cause the minimum PD threshold to be crossed. In addition, the operator would make potential adjustments to the target classification threshold (according to CAML-provided ROC curves for different conditions) or other operating parameters that would enable it to maintain a constant PD and alert operator of any resulting undesirable behaviors (e.g., excessive false alarm rate).

Adaptive-behaviors This scenario involves demonstrating a competency-aware machine learning system’s ability to reason with environmental conditions and deliberately use task strategies in action selection. For example, in a leader-follower convoy system platform there is an operator command (e.g., “catch up with the leader”). The competency-aware system would monitor the environment and its task strategies, and alert the operator if its expected behavior would deviate from the desired behavior (e.g., in rugged terrain its learned behavior is to reduce speed). Furthermore, the system would alert the operator when their task strategies are not applicable (e.g., it has not driven on muddy roads).

Informed role-assignments This scenario requires a competency-aware machine learning system to provide operators insights into its expected behaviors and knowledge of its competencies to enable proper task role assignments. For example, the operator queries multiple ISR learning systems about their competency in image recognition classification tasks (i.e., ROC and level of experiences) for different subject categories (e.g., cars, trucks, tanks, etc.). Upon receiving the competency statements, the operator can task the appropriate competency-aware learning system to execute autonomous decisions for frequently encountered, well-performed subject categories, but request operator confirmation of classifications made for lesser performing subject categories or for less practiced categories.

HR001119S0030 CAML 10

Prior to the completion of Phase I (at approximately Month 30), DARPA intends to provide a detailed Phase II demonstration plan and request updated technical and cost proposals for Phase II per specifications and guidance to be provided by DARPA. Participation in the competition for Phase II is optional and will be limited to Phase I performers. Associated proposal preparation costs for Phase II will not be reimbursed under Phase I awards. Evaluation of Phase II proposals will be based on the evaluation criteria specified in Section V of this BAA, and Phase II proposal evaluations will be conducted through a scientific and technical review process in accordance with the procedures outlined in this BAA. The Government reserves the right to change the award instrument or issue a new solicitation for Phase II if programmatic circumstances dictate.

Participation in an early phase does not guarantee funding in subsequent phases, and progression to Phase II will be contingent on evaluation of the Phase II proposal and availability of funding.

F. Schedule/Milestones The CAML program will focus on a machine learning system’s ability to self-assess its competency. The program will evaluate the quality of the machine reported competency with robust objective measures but will not address the human perception of the AI/ML system’s competency.

The Government will assess the developed competency-aware machine learning models using the following performance metrics:

Coverage Correctness Fidelity Reliability

Completeness of identified conditions in the learning data that lead to different task strategies

The accuracy of generalized rules for describing task behaviors

The accuracy of machine competency statements for describing actual performance

The system’s ability to maintain behaviors consistent with its self-assessed competency

Progress on CAML will be measured against the following ramp-up schedule:

Performance evaluations will use Government-provided test data. The Government team will work with each performer to define test data in accordance with performers’ applications and competency models.

HR001119S0030 CAML 11

In addition to the above metrics, proposers may propose additional performance measurement schemes tailored to their selected machine learning problems to highlight the unique and novel capabilities of their competency-aware learning technology.

Milestones:

Figure 1 shows the CAML program schedule and milestones.

Program Schedule Proposers should provide a technical and programmatic strategy that conforms to the entire program schedule and presents an aggressive plan to fully address all program goals, metrics, milestones and deliverables.

The task structure must be consistent across the proposed schedule, Statement of Work, and cost volume.

A target start date of October 1, 2019 may be assumed for planning purposes.

All proposals must include the following meetings and travel in the proposed schedule and costs:

Program kick-off meeting Quarterly program reviews: the meetings will alternate between the performers’ facilities and DARPA facility in Arlington, VA. During which they will have the opportunity to demonstrate progress towards agreed-upon milestones.

To continue integration and development between TAs, foster collaboration between teams, and disseminate program developments, a two-day Principal Investigator (PI) meeting will be held in Phase I in year 1 and year 2, with locations split between the East and West Coasts of the United States. For budgeting purposes between Washington, D.C. and San Francisco, CA.

Site visits: performers in Phase II should expect two site visits for performance demonstrations

Regular teleconference meetings will be scheduled with the Government team for

HR001119S0030 CAML 12

progress reporting as well as problems identification and mitigation.

G. Deliverables Performers will be expected to provide at a minimum the following deliverables:

Comprehensive quarterly technical reports due within ten days of the end of the given quarter, describing progress made on the specific milestones as laid out in the SOW

A phase completion report submitted within 30 days of the end of each phase, summarizing the research done

Monthly technical and financial status reports Other presentation materials as requested by the Government Software code and documentations Final technical report Other negotiated deliverables specific to the objectives of the individual efforts. These may include registered reports, experimental protocols, publications, data management plan, intermediate and final versions of software libraries, code, and APIs, including documentation and user manuals, and/or a comprehensive assemblage of design documents, models, modeling data and results, and model validation data.

Reporting as outlined in Section VI.C.

II. Award Information A. General Award Information

DARPA anticipates multiple awards.

The level of funding for individual awards made under this BAA will depend on the quality of the proposals received and the availability of funds. Awards will be made to proposers1 whose proposals are determined to be the most advantageous to the Government, all evaluation factors considered. See Section V for further information.

The Government reserves the right to:

select for negotiation all, some, one, or none of the proposals received in response to this solicitation;

make awards without discussions with proposers;

conduct discussions with proposers if it is later determined to be necessary;

segregate portions of resulting awards into pre-priced options;

accept proposals in their entirety or to select only portions of proposals for award;

fund awards in increments with options for continued work at the end of one or more

1 As used throughout this BAA, “proposer” refers to the lead organization on a submission to this BAA. The proposer is responsible for ensuring that all information required by a BAA--from all team members--is submitted in accordance with the BAA. “Awardee” refers to anyone who might receive a prime award from the Government, including recipients of procurement contracts, cooperative agreements, or Other Transactions. “Subawardee” refers to anyone who might receive a subaward from a prime awardee (e.g., subawardee, consultant, etc.).

HR001119S0030 CAML 13

phases;

request additional documentation once the award instrument has been determined (e.g., representations and certifications); and remove proposers from award consideration should the parties fail to reach agreement on award terms within a reasonable time or the proposer fails to provide requested additional information in a timely manner.

Proposals identified for negotiation may result in a procurement contract, cooperative agreement, or other transaction (OT), depending upon the nature of the work proposed, the required degree of interaction between parties, and other factors.

Proposers looking for innovative, commercial-like contractual arrangements are encouraged to consider requesting Other Transactions. To understand the flexibility and options associated with Other Transactions, consult http://www.darpa.mil/work-with-us/contract-management#OtherTransactions.

In accordance with 10 U.S.C. § 2371b(f), the Government may award a follow-on production contract or Other Transaction (OT) for any OT awarded under this BAA if: (1) that participant in the OT, or a recognized successor in interest to the OT, successfully completed the entire prototype project provided for in the OT, as modified; and (2) the OT provides for the award of a follow-on production contract or OT to the participant, or a recognized successor in interest to the OT.

In all cases, the Government contracting officer shall have sole discretion to select award instrument type, regardless of instrument type proposed, and to negotiate all instrument terms and conditions with selectees. DARPA will apply publication or other restrictions, as necessary, if it determines that the research resulting from the proposed effort will present a high likelihood of disclosing performance characteristics of military systems or manufacturing technologies that are unique and critical to defense. Any award resulting from such a determination will include a requirement for DARPA permission before publishing any information or results on the program. For more information on publication restrictions, see the section below on Fundamental Research.

B. Fundamental Research It is DoD policy that the publication of products of fundamental research will remain unrestricted to the maximum extent possible. National Security Decision Directive (NSDD) 189 defines fundamental research as follows:

‘Fundamental research’ means basic and applied research in science and engineering, the results of which ordinarily are published and shared broadly within the scientific community, as distinguished from proprietary research and from industrial development, design, production, and product utilization, the results of which ordinarily are restricted for proprietary or national security reasons.

As of the date of publication of this BAA, the Government expects that program goals as described herein may be met by proposers intending to perform fundamental research and does http://www.darpa.mil/work-with-us/contract-management#OtherTransactions http://www.darpa.mil/work-with-us/contract-management#OtherTransactions

HR001119S0030 CAML 14

not anticipate applying publication restrictions of any kind to individual awards for fundamental research that may result from this BAA. Notwithstanding this statement of expectation, the Government is not prohibited from considering and selecting research proposals that, while perhaps not qualifying as fundamental research under the foregoing definition, still meet the BAA criteria for submissions. If proposals are selected for award that offer other than a fundamental research solution, the Government will either work with the proposer to modify the proposed statement of work to bring the research back into line with fundamental research or else the proposer will agree to restrictions in order to receive an award.

As of the date of publication of this BAA, the Government cannot identify whether the work to be performed in Phase II may be considered fundamental research. Based on the nature of the assets and the nature of the work, the Government anticipates that some awards will include restrictions in Phase II on the resultant research that will require the awardee to seek DARPA permission before publishing any information or results relative to the program.

Proposers should indicate in their proposal whether they believe the scope of the research included in their proposal is fundamental or not. While proposers should clearly explain the intended results of their research, the Government shall have sole discretion to determine whether the proposed research shall be considered fundamental. Appropriate clauses will be included in resultant awards for non-fundamental research to prescribe publication requirements and other restrictions, as appropriate. This clause can be found at www.darpa.mil/work-with-us/additional-baa.

For certain research projects, it may be possible that although the research to be performed by a potential awardee is restricted research, their subawardee’s effort may be fundamental research.

In those cases, it is the awardee’s responsibility to explain in their proposal why its subawardee’s effort is fundamental research.

III. Eligibility Information A. Eligible Applicants

All responsible sources capable of satisfying the Government's needs may submit a proposal DARPA’s consideration.

1. Federally Funded Research and Development Centers (FFRDCs) and Government Entities

a. FFRDCs

FFRDCs are subject to applicable direct competition limitations and cannot propose to this BAA in any capacity unless they meet the following conditions: (1) FFRDCs must clearly demonstrate that the proposed work is not otherwise available from the private sector. (2) FFRDCs must provide a letter on official letterhead from their sponsoring organization citing the specific authority establishing their eligibility to propose to Government solicitations and compete with industry, and their compliance with the associated FFRDC sponsor agreement’s http://www.darpa.mil/work-with-us/additional-baa

HR001119S0030 CAML 15

terms and conditions. This information is required for FFRDCs proposing to be awardees or subawardees.

b. Government Entities

Government Entities (e.g., Government/National laboratories, military educational institutions, etc.) are subject to applicable direct competition limitations. Government entities must clearly demonstrate that the work is not otherwise available from the private sector and provide written documentation citing the specific statutory authority and contractual authority, if relevant, establishing their ability to propose to Government solicitations. This information is required for Government Entities proposing to be awardees or subawardees.

c. Authority and Eligibility

At the present time, DARPA does not consider 15 U.S.C. § 3710a to be sufficient legal authority to show eligibility. While 10 U.S.C.§ 2539b may be the appropriate statutory starting point for some entities, specific supporting regulatory guidance, together with evidence of agency approval, will still be required to fully establish eligibility. DARPA will consider FFRDC and Government entity eligibility submissions on a case-by-case basis; however, the burden to prove eligibility for all team members rests solely with the proposer.

2. Foreign Participation Non-U.S. organizations and/or individuals may participate to the extent that such participants comply with any necessary nondisclosure agreements, security regulations, export control laws, and other governing statutes applicable under the circumstances. For classified submissions, this includes mitigating any Foreign Ownership Control and Influence (FOCI) issues prior to transmitting the submission to DARPA. Additional information on these subjects can be found at http://www.dss.mil/isp/foci/foci_faqs.html.

B. Organizational Conflicts of Interest

FAR 9.5 Requirements In accordance with FAR 9.5, proposers are required to identify and disclose all facts relevant to potential OCIs involving the proposer’s organization and any proposed team member (subawardee, consultant). Under this Section, the proposer is responsible for providing this disclosure with each proposal submitted to the BAA. The disclosure must include the proposer’s, and as applicable, proposed team member’s OCI mitigation plan. The OCI mitigation plan must include a description of the actions the proposer has taken, or intends to take, to prevent the existence of conflicting roles that might bias the proposer’s judgment and to prevent the proposer from having unfair competitive advantage. The OCI mitigation plan will specifically discuss the disclosed OCI in the context of each of the OCI limitations outlined in FAR 9.505-1 through FAR 9.505-4.

Agency Supplemental OCI Policy

HR001119S0030 CAML 16

In addition, DARPA has a supplemental OCI policy that prohibits contractors/performers from concurrently providing Scientific Engineering Technical Assistance (SETA), Advisory and Assistance Services (A&AS) or similar support services and being a technical performer.

Therefore, as part of the FAR 9.5 disclosure requirement above, a proposer must affirm whether the proposer or any proposed team member (subawardee, consultant) is providing SETA, A&AS, or similar support to any DARPA office(s) under: (a) a current award or subaward; or (b) a past award or subaward that ended within one calendar year prior to the proposal’s submission date.

If SETA, A&AS, or similar support is being or was provided to any DARPA office(s), the proposal must include:

The name of the DARPA office receiving the support;

The prime contract number;

Identification of proposed team member (subawardee, consultant) providing the support; and An OCI mitigation plan in accordance with FAR 9.5.

Government Procedures In accordance with FAR 9.503, 9.504 and 9.506, the Government will evaluate OCI mitigation plans to avoid, neutralize or mitigate potential OCI issues before award and to determine whether it is in the Government’s interest to grant a waiver. The Government will only evaluate OCI mitigation plans for proposals that are determined selectable under the BAA evaluation criteria and funding availability.

The Government may require proposers to provide additional information to assist the Government in evaluating the proposer’s OCI mitigation plan.

If the Government determines that a proposer failed to fully disclose an OCI; or failed to provide the affirmation of DARPA support as described above; or failed to reasonably provide additional information requested by the Government to assist in evaluating the proposer’s OCI mitigation plan, the Government may reject the proposal and withdraw it from consideration for award.

C. Cost Sharing/Matching Cost sharing is not required; however, it will be carefully considered where there is an applicable statutory condition relating to the selected funding instrument (e.g., OTs under the authority of 10 U.S.C. § 2371). Cost sharing is encouraged where there is a reasonable probability of a potential commercial application related to the proposed research and development effort.

For more information on potential cost sharing requirements for Other Transactions for Prototype, see http://www.darpa.mil/work-with-us/contract-management#OtherTransactions.

IV. Application and Submission Information Prior to submitting a full proposal, proposers are strongly encouraged to first submit an abstract as described below. This process allows a proposer to ascertain whether the

HR001119S0030 CAML 17

proposed concept is: (1) applicable to the Competency-Aware Machine Learning BAA and

(2) currently of interest. For the purposes of this BAA, applicability is defined as follows:

The proposed concept is applicable to the technical areas described herein.

The proposed concept is important to DSO’s current investment portfolio.

The proposed concept investigates an innovative approach that enables revolutionary advances, i.e., will not primarily result in evolutionary improvements to the existing state of practice.

The proposed work has not already been completed (i.e., the research element is complete but manufacturing/fabrication funds are required).

The proposer has not already received funding or a positive funding decision for the proposed concept (whether from DARPA or another Government agency).

Abstracts and full proposals that are not found to be applicable to the Competency-Aware Machine Learning BAA as defined above may be deemed non-conforming2 and removed from consideration. All abstracts and full proposals must provide sufficient information to assess the validity/feasibility of their claims as well as comply with the requirements outlined herein for submission formatting, content and transmission to DARPA. Abstracts and full proposals that fail to do so may be deemed non-conforming and removed from consideration. Proposers will be notified of non-conforming determinations via letter.

A. Address to Request Application Package This document contains all information required to submit a response to this solicitation. No additional forms, kits, or other materials are needed except as referenced herein. No request for proposal or additional solicitation regarding this opportunity will be issued, nor is additional information available except as provided at the Federal Business Opportunities website (http://www.fbo.gov), the Grants.gov website (http://www.grants.gov/), or referenced herein.

B. Content and Form of Application Submission

1. Abstract Information

As stated above, proposers are strongly encouraged to submit an abstract in advance of a full proposal to minimize effort and reduce the potential expense of preparing an out of scope proposal. The abstract provides a synopsis of the proposed project by briefly answering the following questions:

What is the proposed work attempting to accomplish or do?

How is it done today, and what are the limitations?

Who will care and what will the impact be if the work is successful?

How much will it cost, and how long will it take?

2 “Conforming” is defined as having been submitted in accordance with the requirements outlined herein.

http://www.fbo.gov/ http://www.onr.navy.mil/Contracts-Grants/submit-proposal/grants-proposal/grants-terms-conditions.aspx

HR001119S0030 CAML 18

DARPA will respond to abstracts with a statement as to whether DARPA is interested in the idea. If DARPA does not recommend the proposer submit a full proposal, DARPA will provide feedback to the proposer regarding the rationale for this decision. Regardless of DARPA’s response to an abstract, proposers may submit a full proposal. DARPA will review all conforming full proposals using the published evaluation criteria and without regard to any comments resulting from the review of an abstract.

Proposers should note that a favorable response to an abstract is not a guarantee that a proposal based on the abstract will ultimately be selected for award negotiation.

While it is DARPA policy to attempt to reply to abstracts within thirty calendar days, proposers to this solicitation may anticipate a response within approximately three weeks. These official notifications will be sent via email to the Technical POC and/or Administrative POC identified on the abstract coversheet.

Abstract Format All proposers are required to use Attachment A: Abstract Summary Slide Template and Attachment B: Abstract Template provided to this solicitation on http://www.fbo.gov and http://www.grants.gov. Attachment A Abstract Summary Slide Template described herein must be in .ppt or .pptx format and should be attached as a separate file to this document.

2. Full Proposal Information Proposals consist of Volume 1: Technical and Management Volume, Volume 2: Cost Volume, and Volume 3: Administrative and National Policy Requirements Volume.

To assist in proposal development, various attachments have been provided along with the BAA posted on http://www.fbo.gov (Attachment C: Proposal Summary Slide Template; Attachment D: Proposal Template Volume 1 Technical & Management Volume; Attachment E: Proposal Template Volume 2 Cost Volume; Attachment F: Proposal Template Volume 2 Cost Summary Spreadsheet; Attachment F-2 Proposal Template Volume 2 Cost Breakdown and Attachment G:

Proposal Template Volume 3 Administrative & National Policy Requirements Volume).

Full proposals requesting a procurement contract or other transaction (OT) must use the following attachments:

Attachment C Attachment D Attachment E Attachment F Attachment F-2 Attachment G https://wawf.eb.mil/ http://www.grants.gov/

HR001119S0030 CAML 19

Full proposals requesting a cooperative agreement must use the following attachments in addition to the Grants.gov application package:

Attachment C Attachment D Attachment F Attachment F-2 Attachment G *Note – Budget Justification should be provided as Section L of the SF 424 Research & Related Budget form provided via Grants.gov. The Budget Justification should include the following information for the recipient and all subawardees: (1) Direct Labor: Detail the total number of persons and their level of commitment for each position listed (in Sections A and B), as well as which specific tasks (as described in the SOW) they will support. (2) Equipment (Section C) Provide an explanation for listed requested equipment exceeding $5,000, properly justifying their need to meet the objectives of the program. (3) Travel (Section D) Provide the purpose of the trip, number of trips, number of days per trip, departure and arrival destinations, number of people, etc. (4) Other Direct Costs (Section F). Provide a justification for the items requested and an explanation of how the estimates were obtained.

Proposals not meeting the format prescribed herein may not be reviewed.

Full Proposal Format

All proposers are required to use the templates provided as attachments to this solicitation on http://www.fbo.gov and http://www.grants.gov. Formatting instructions are provided therein.

3. Proprietary Information Proposers are responsible for clearly identifying proprietary information. Submissions containing proprietary information must have the cover page and each page containing such information clearly marked with a label such as “Proprietary” or “Company Proprietary.” NOTE:

“Confidential” is a classification marking used to control the dissemination of U.S. Government National Security Information as dictated in Executive Order 13526 and should not be used to identify proprietary business information. See Section V.B.1 for additional information.

4. Security Information DARPA anticipates that submissions received under this BAA will be unclassified. However, should a proposer wish to submit classified information, an unclassified email must be sent to the BAA mailbox requesting submission instructions from the DARPA/DSO Program Security Officer (PSO).

Security classification guidance and direction via a Security Classification Guide (SCG) and/or DD Form 254, “DoD Contract Security Classification Specification,” will not be provided at this time, since DARPA is soliciting ideas only. If a determination is made that the award instrument may result in access to classified information, a SCG and/or DD Form 254 will be http://www.darpa.mil/work-with-us/contract-management

HR001119S0030 CAML 20

issued by DARPA and attached as part of the award.

a. Classified Submission Requirements and Procedures

Classified submissions shall be marked and transmitted in accordance with the guidance outlined herein. Submissions containing Classified National Security Information (as defined by Executive Order 13526) must be conspicuously marked with the appropriate classification level and declassification date of both the submitted materials and that of the anticipated award.

i. Undetermined Classification Level: Submissions requiring DARPA to make a final classification determination shall be marked as follows:

“CLASSIFICATION DETERMINATION PENDING. Protect as though classified____________________________” (insert the recommended classification level, e.g., Top Secret, Secret or Confidential)

Such submissions shall be transmitted in accordance with the appropriate instructions below applicable to the proposed classification level.

ii. Confidential and Secret Information: Use transmission, classification, handling, and marking guidance provided by previously issued SCGs, the DoD Information Security Manual (DoDM 5200.01, Volumes 1 - 4), and the National Industrial Security Program Operating Manual, including the Supplement Revision 1, (DoD 5220.22-M and DoD 5200.22-M Sup. 1) when submitting Confidential and/or Secret classified information.

Confidential and Secret classified information may be submitted via ONE of the two following methods:

Hand-carried by an appropriately cleared and authorized courier to the DARPA CDR. Prior to traveling, the courier shall contact the DARPA Classified Document Registry (CDR) at 703-526-4052 to coordinate arrival and delivery.

OR

Mailed via U.S. Postal Service (USPS) Registered Mail or USPS Express Mail. All classified information will be enclosed in opaque inner and outer covers and double-wrapped. The inner envelope shall be sealed and plainly marked with the assigned classification and addresses of both sender and addressee.

HR001119S0030 CAML 21

The inner envelope…