HR001118S0044.pdf

Broad Agency Announcement Learning with Less Labels (LwLL)

HR001118S0044

August 6, 2018

Defense Advanced Research Projects Agency Information Innovation Office 675 North Randolph Street Arlington, VA 22203-2114

HR001118S0044 LEARNING WITH LESS LABELS (LWLL) 2

Table of Contents I. Funding Opportunity Description

A. Introduction/Background

B. Program Description

C. Program Structure

D. Schedule, Milestones and Evaluation

E. Deliverables

F. Government-furnished Property/Equipment/Information

G. Intellectual Property

II. Award Information

A. Awards

B. Fundamental Research

C. Disclosure of Information and Compliance with Safeguarding Covered Defense Information Controls

III. Eligibility Information

A. Eligible Applicants

B. Organizational Conflicts of Interest

C. Cost Sharing/Matching

D. Other Eligibility Requirements

IV. Application and Submission Information

A. Address to Request Application Package

B. Content and Form of Application Submission

C. Submission Dates and Times

D. Funding Restrictions

E. Other Submission Requirements

V. Application Review Information

A. Evaluation Criteria

B. Review and Selection Process

VI. Award Administration Information

A. Selection Notices

B. Administrative and National Policy Requirements

C. Reporting

VII. Agency Contacts

VIII. Other Information

HR001118S0044 LEARNING WITH LESS LABELS (LWLL) 3

A. Frequently Asked Questions (FAQs)

B. Collaborative Efforts/Teaming

C. Proposers Day

D. Submission Checklist

HR001118S0044 LEARNING WITH LESS LABELS (LWLL) 4

PART I: OVERVIEW INFORMATION

Federal Agency Name: Defense Advanced Research Projects Agency (DARPA), Information Innovation Office (I2O)

Funding Opportunity Title: Learning with Less Labels (LwLL)

Announcement Type: Initial Announcement

Funding Opportunity Number: HR001118S0044

Catalog of Federal Domestic Assistance Numbers (CFDA): 12.910 Research and Technology Development

Dates o Posting Date: August 6, 2018 o Proposers Day: July 13, 2018 o Abstract Due Date: August 21, 2018, 12:00 noon (ET) o Proposal Due Date: October 2, 2018, 12:00 noon (ET) o BAA Closing Date: October 2, 2018, 12:00 noon (ET)

Anticipated Individual Awards: DARPA anticipates multiple awards under this solicitation.

Types of Instruments that May be Awarded: Procurement contracts, cooperative agreements or Other Transactions

Agency Contacts o Technical POC: Wade Shen, Program Manager, DARPA/I2O o BAA Email: LwLL@darpa.mil o BAA Mailing Address:

DARPA/I2O

ATTN: HR001118S0044

675 North Randolph Street Arlington, VA 22203-2114 o I2O Solicitation Website: http://www.darpa.mil/work-with-us/opportunities mailto:LwLL@darpa.mil http://www.darpa.mil/work-with-us/opportunities

HR001118S0044 LEARNING WITH LESS LABELS (LWLL) 5

PART II: FULL TEXT OF ANNOUNCEMENT

I. Funding Opportunity Description

DARPA is soliciting innovative research proposals in the area of machine learning and artificial intelligence. Proposed research should investigate innovative approaches that enable revolutionary advances in science, devices, or systems. Specifically excluded is research that primarily results in evolutionary improvements to the existing state of practice.

This Broad Agency Announcement (BAA) is being issued, and any resultant selection will be made, using procedures under Federal Acquisition Regulation (FAR) 6.102(d)(2) and 35.016.

Any negotiations and/or awards will use procedures under FAR 15.4 (or 32 CFR § 200.203 for cooperative agreements). Proposals received as a result of this BAA shall be evaluated in accordance with evaluation criteria specified herein through a scientific review process.

DARPA BAAs are posted on the Federal Business Opportunities (FBO) website (https://www.fbo.gov/) and the Grants.gov website (http://www.grants.gov/).

The following information is for those wishing to respond to this BAA.

A. Introduction/Background

In supervised machine learning (ML), the ML system learns by example to recognize things, such as objects in images or speech. Humans provide these examples to ML systems during their training in the form of labeled data. With enough labeled data, we can generally build accurate pattern recognition models.

The problem is that training accurate models currently requires lots of labeled data. For tasks like machine translation, speech recognition or object recognition, deep neural networks (DNNs) have emerged as the state of the art, due to the superior accuracy they can achieve. To gain this advantage over other techniques, however, DNN models need more data, typically requiring 109 or 1010 labeled training examples to achieve good performance.

The commercial world has harvested and created large sets of labeled data for training models.

These datasets are often created via crowdsourcing: a cheap and efficient way to create labeled data. Unfortunately, crowdsourcing techniques are often not possible for proprietary or sensitive data. Creating data sets for these sorts of problems can result in 100x higher costs and 50x longer time to label.

To make matters worse, machine learning models are brittle, in that their performance can degrade severely with small changes in their operating environment. For instance, the performance of computer vision systems degrades when data is collected from a new sensor and new collection viewpoints. Similarly, dialog and text understanding systems are very sensitive to changes in formality and register. As a result, additional labels are needed after initial training to adapt these models to new environments and data collection conditions. For many problems, the labeled data required to adapt models to new environments approaches the amount required to train a new model from scratch.

https://www.fbo.gov/ http://www.grants.gov/

HR001118S0044 LEARNING WITH LESS LABELS (LWLL) 6

B. Program Description

The goal of this program is to make the process of training machine learning models more efficient by reducing the amount of labeled data required to build a model by six or more orders of magnitude, and by reducing the amount of data needed to adapt models to new environments to tens to hundreds of labeled examples.

In order to achieve the massive reductions of labeled data needed to train accurate models, the Learning with Less Labels program (LwLL) will divide the effort into two technical areas (TAs).

TA1 will focus on the research and development of learning algorithms that learn and adapt efficiently; and TA2 will formally characterize machine learning problems and prove the limits of learning and adaptation.

TA1: Learn and Adapt Efficiently

The goal of TA1 is to develop learning algorithms that: (1) reduce the amount of labeled data required to build a model from scratch by at least a factor of 106; and (2) adapt to new environments with hundreds of labeled examples. These algorithms can make use of as much unlabeled data as they wish and they may choose specific examples for labeling. The resulting algorithms may make use of existing models developed with publicly available corpora or pre-existing data (labeled and/or unlabeled) from publicly available datasets (which specific datasets will be allowed will be determined in advance of challenge problem evaluations). Algorithms must work autonomously. This means that given a dataset, algorithms must be able to automatically determine which exemplars they would like to have labeled, select from existing corpora or existing models for potential transfer, and create models of a given task without human intervention. Algorithms can create data as part of this process, but they cannot manually create labels.

In order to achieve these ambitious goals, it is likely that new methods will be needed to focus on salient aspects of input data while reducing nuisance variation, and to exploit unlabeled data via implicit or indirect supervision will be needed. DARPA anticipates advances in methods such as meta-learning, automated (and potentially distant) transfer learning, reinforcement learning, active learning, unsupervised or semi-supervised learning, and/or k-shot learning. Novel combinations of these techniques may be needed to achieve program performance targets. That said, any approach that can meet program objectives is in scope, including those not listed above.

To support TA1 performers, DARPA will curate and make available to program performers a corpus of at least four hundred machine learning problems and associated models.

TA1 techniques will be evaluated on an annual basis against a set of challenge problems such as computer vision, video recognition, and/or machine translation. These evaluation events will test the ability of TA1 algorithms to train models from scratch (task TA1.1) and adapt existing models to a new domain (TA1.2). For any given challenge, active labeling methods can be used to decide which examples will receive labels. DARPA will define the training and adaptation tasks and will provide guidelines and restrictions on data usage (e.g., subsets of the supplied corpus that can be used for a given challenge problem) for each annual evaluation.

HR001118S0044 LEARNING WITH LESS LABELS (LWLL) 7

TA2: Limits of Machine Learning and Adaptation

The goal of TA2 is to formally prove limits on the amount of labeled data needed to solve a given machine learning problem. To do this, methods are needed to formally characterize machine learning problems, both in terms of their decision difficulty and the true complexity of the data from which decisions are made. The resulting characterizations should enable the proof of limits on training and adaptation for different classes of machine learning problems, different models, and different kinds of data associated with said problems. DARPA seeks theories that prove tight bounds on learning in the presence of transfer and meta-transfer learning. The scope of this TA includes extensions to PAC (Probably Approximately Correct) learning theory (and variants) or alternative formalisms to prove tight class-specific problem bounds (e.g., extensions to VC theory), and statistical theory needed to characterize data complexity (e.g., extensions to Johnson-Lindenstrauss) and domain mismatch.

TA2 theories will be applied during annual evaluations to provide upper-bound estimates on performance of challenge problems as defined by DARPA. We anticipate refinement of program characterization and theories based on interactions with TA1 systems during annual evaluation events.

C. Program Structure

The period of performance for LwLL will be 36 months. The timeline for this program is shown in Figure 1. LwLL is divided into two phases, each 18 months in duration. Evaluations will be held every 12 months and at the end of each phase (i.e., an additional end-of-first-phase evaluation will occur at the 18-month point of the program timeline). While teams are required to submit systems individually for evaluation, teams are also encouraged to work with each other to build better systems through combined submissions during program evaluations. Between evaluations, the program will hold a series of collaboration events to allow performer teams to explore cross-team techniques and prepare for program evaluations. Program performance targets for each phase are detailed in Figure 2 and Figure 3.

For TA1, the objectives of Phase 1 are a 103x reduction in required training data when building models from scratch, and adaptation with thousands of examples. These objectives will be evaluated on image object detection and classification. The final objectives for TA1 match the program’s overall goal: 106x reduction in labeled data needed to build a model from scratch and 102 labeled examples needed to adapt to new environments. TA1 performers will need to prove minimal performance loss with respect to current state of the art, and must demonstrate this on all three challenge problems by the end of each phase of the program.

The objectives described in TA2 will be evaluated based on the tightness of provable bounds, and number and quality of peer-refereed articles for each of the topical areas listed in Figure 3.

HR001118S0044 LEARNING WITH LESS LABELS (LWLL) 8

GovTeam: Annual evaluation problems

TA2: Limits of generalized learning

TA1: Learn and adapt with less labels

Preliminary TA1 toolkit + publications

Phase 1 (18 months) Phase 2 (18 months)

Intermediate collaboration event

Initial methods for reinforcement learning

Similarity metrics for problem transfer

End of phase evaluation

Video activity recognition Machine translationImage object recognition

Problem clustering and generalization I

Initial methods for GAN proxy discovery

Annual challenge problem evaluation

Initial methods for active learning

Initial data complexity bounds

Initial problem complexity bounds

Formal problem complexity measures

Formal characterization problem/domain attributes

Final data complexity bounds

Final problem complexity bounds

Final TA1 toolkit + publications

Reward discovery for reinforcement learning

Generalized adversarial proxy discovery

Automated optimal experimentation for active learning

Similarity metrics / problem embedding (problem2vec)

Problem clustering and generalization II

Figure 1: Program Schedule

Challenges Train

(TA1.1)

Adapt

(TA1.2)

Train

(TA1.1)

Adapt

(TA1.2)

Object detection1

Train: LSVRC (open) Adapt: TBD Metric: mAP @ # labels

80% @ 105 80% @ 103 80% @ 102 80% @ 102

Object classification2

Train: LSVRC (open) Adapt: TBD Metric: mAP @ # labels

97% @ 106 97% @ 103 97% @ 103 97% @ 102

Activity recognition3

Train: TRECVid MED task Adapt: TBD Metric: mAP @ # labels

41% @ 103 41% @ 102

Machine translation4

Train: OpenMT task Adapt: TBD Metric: BLEU @ # labels

47% @ 103 47% @ 102

Phase 1 Phase 2

Figure 2: Program Goals for TA1 (accuracy @ N labeled examples. See terms here1,2,3,4)

1 LSVRC = ImageNet Large Scale Visual Recognition Challenge 2 mAP = Mean Average Precision 3 TRECVid MED = NIST TREC Video Multimedia Event Detection 4 BLEU = BiLingual Evaluation Understudy metric

HR001118S0044 LEARNING WITH LESS LABELS (LWLL) 9

Figure 3: Program Goals for TA2

D. Schedule, Milestones and Evaluation

The program will conduct a series of annual evaluations and collaboration events in the greater Washington DC area as shown in the schedule (see Figure 1). These events will last 2 weeks and will be conducted in every summer and winter (timed to the academic calendar). In addition to these events, semi-annual site visits (at or near performer locations) will be conducted by DARPA management to assess intermediate progress.

E. Deliverables

The Government anticipates receiving the following deliverables throughout the program:

Any technical papers covering work funded by LwLL;

Any data created during program work on program challenge problems;

Commented source code, any other necessary data, and documentation (including at a minimum a user manual and a detailed software design document) for all software developed under this program;

Quarterly technical status reports detailing progress made, tasks accomplished, major risks, planned activities, trip summaries, changes to key personnel, and any potential issues or problem areas that require the attention of the Government Team must be provided within 10 days after the end of each month;

Monthly financial status reports must be provided within 10 days after the end of each month;

HR001118S0044 LEARNING WITH LESS LABELS (LWLL) 10

A final phase report for each program phase that must concisely summarize the effort conducted, technical achievements, and remaining technical challenges, and will be due one calendar month after the end of each phase; and

A final report at the end of the overall period of performance that summarizes the project

Award instrument type may alter this list.

F. Government-furnished Property/Equipment/Information

In order to support program evaluations (annual and end-of-phase), the Government will make available training data and adaptation data with associated labels. Government furnished information will be made available as indicated below:

For TA1 performers who intend to do active labeling or active learning, an application programming interface (API)/mechanism for requesting labels for specific examples will be accessible during (and after) evaluations for each challenge problem. This API will be made available for testing and integration two months prior to the first evaluation.

Datasets and problems derived from open, on-line data sources such as OpenML, kaggle, dataverse, and NIST evaluations, etc. will also be available to LwLL performers for meta-learning and transfer learning purposes at program kickoff. These datasets and any corresponding models cover more than 400 problems across multiple domains and multiple tasks.

G. Intellectual Property

The program will emphasize creating and leveraging open source technology and architecture.

Intellectual property rights asserted by proposers are strongly encouraged to be aligned with open source regimes. See Section VI.B.1 for more details on intellectual property.

A key goal of the program is to create open algorithms that enable future applied research efforts to make use of the results of this program. This includes the ability to easily add, remove, substitute, and modify software components. This will facilitate rapid innovation by providing a base for future users or developers of program technologies and deliverables. Therefore, it is desired that all noncommercial software (including source code), software documentation, hardware designs and documentation, and technical data generated by the program be provided as deliverables to the Government, with a minimum of Government Purpose Rights (GPR), as lesser rights may adversely impact the lifecycle costs of affected items, components, or processes.

HR001118S0044 LEARNING WITH LESS LABELS (LWLL) 11

II. Award Information

A. Awards

Multiple awards are anticipated. The level of funding for individual awards made under this solicitation has not been predetermined and will depend on the quality of the proposals received and the availability of funds. Awards will be made to proposers whose proposals are determined to be the most advantageous to the Government, all factors considered, including the potential contributions of the proposed work, overall funding strategy, and availability of funding. See Section V for further information.

The Government reserves the right to:

select for negotiation all, some, one, or none of the proposals received in response to this solicitation;

make awards without discussions with proposers;

conduct discussions with proposers if it is later determined to be necessary;

segregate portions of resulting awards into pre-priced options;

accept proposals in their entirety or to select only portions of proposals for award;

fund proposals in increments and/or with options for continued work at the end of one or more phases;

request additional documentation once the award instrument has been determined (e.g., representations and certifications); and remove proposers from award consideration should the parties fail to reach agreement on award terms within a reasonable time or the proposer fails to provide requested additional information in a timely manner.

Proposals selected for award negotiation may result in a procurement contract, cooperative agreement, or Other Transaction (OT) depending upon the nature of the work proposed, the required degree of interaction between parties, and other factors.

Proposers looking for innovative, commercial-like contractual arrangements are encouraged to consider requesting Other Transactions. To understand the flexibility and options associated with Other Transactions, consult http://www.darpa.mil/work-with-us/contract-management#OtherTransactions.

In accordance with 10 U.S.C. § 2371b(f), the Government may award a follow-on production contract or Other Transaction (OT) for any OT awarded under this BAA if: (1) that participant in the OT, or a recognized successor in interest to the OT, successfully completed the entire prototype project provided for in the OT, as modified; and (2) the OT provides for the award of a follow-on production contract or OT to the participant, or a recognized successor in interest to the OT.

In all cases, the Government contracting officer shall have sole discretion to select award instrument type, regardless of instrument type proposed, and to negotiate all instrument terms and conditions with selectees. DARPA will apply publication or other restrictions, as necessary, if it determines that the research resulting from the proposed effort will present a high likelihood of disclosing performance characteristics of military systems or manufacturing technologies that are unique and critical to defense. Any award resulting from such a determination will include a requirement for DARPA permission before publishing any information or results on the http://www.darpa.mil/work-with-us/contract-management#OtherTransactions http://www.darpa.mil/work-with-us/contract-management#OtherTransactions

HR001118S0044 LEARNING WITH LESS LABELS (LWLL) 12

program. For more information on publication restrictions, see the section below on Fundamental Research.

B. Fundamental Research

It is DoD policy that the publication of products of fundamental research will remain unrestricted to the maximum extent possible. National Security Decision Directive (NSDD) 189 defines fundamental research as follows:

‘Fundamental research’ means basic and applied research in science and engineering, the results of which ordinarily are published and shared broadly within the scientific community, as distinguished from proprietary research and from industrial development, design, production, and product utilization, the results of which ordinarily are restricted for proprietary or national security reasons.

As of the date of publication of this BAA, the Government expects that program goals as described herein may be met by proposers intending to perform fundamental research and does not anticipate applying publication restrictions of any kind to individual awards for fundamental research that may result from this BAA. Notwithstanding this statement of expectation, the Government is not prohibited from considering and selecting research proposals that, while perhaps not qualifying as fundamental research under the foregoing definition, still meet the BAA criteria for submissions. If proposals are selected for award that offer other than a fundamental research solution, the Government will either work with the proposer to modify the proposed statement of work to bring the research back into line with fundamental research or else the proposer will agree to restrictions in order to receive an award.

Proposers should indicate in their proposal whether they believe the scope of the research included in their proposal is fundamental or not. While proposers should clearly explain the intended results of their research, the Government shall have sole discretion to select award instrument type and to negotiate all instrument terms and conditions with selectees. Appropriate clauses will be included in resultant awards for non-fundamental research to prescribe publication requirements and other restrictions, as appropriate. This clause can be found at http://www.darpa.mil/work-with-us/additional-baa.

For certain research projects, it may be possible that although the research being performed by the awardee is restricted research, a subawardee may be conducting fundamental research. In those cases, it is the awardee’s responsibility to explain in their proposal why its subawardee’s effort is fundamental research

C. Disclosure of Information and Compliance with Safeguarding Covered Defense Information Controls

The following provisions and clause apply to all solicitations and contracts; however, the definition of “controlled technical information” clearly exempts work considered fundamental research and therefore, even though included in the contract, will not apply if the work is fundamental research.

DFARS 252.204-7000, “Disclosure of Information” DFARS 252.204-7008, “Compliance with Safeguarding Covered Defense Information Controls” http://www.darpa.mil/work-with-us/additional-baa

HR001118S0044 LEARNING WITH LESS LABELS (LWLL) 13

DFARS 252.204-7012, “Safeguarding Covered Defense Information and Cyber Incident Reporting”

The full text of the above solicitation provision and contract clauses can be found at http://www.darpa.mil/work-with-us/additional-baa#NPRPAC.

Compliance with the above requirements includes the mandate for proposers to implement the security requirements specified by National Institute of Standards and Technology (NIST) Special Publication (SP) 800-171, “Protecting Controlled Unclassified Information in Nonfederal Information Systems and Organizations” (see https://doi.org/10.6028/NIST.SP.800-171r1) that are in effect at the time the BAA is issued.

For awards where the work is considered fundamental research, the contractor will not have to implement the aforementioned requirements and safeguards; however, should the nature of the work change during performance of the award, work not considered fundamental research will be subject to these requirements.

http://www.darpa.mil/work-with-us/additional-baa#NPRPAC https://doi.org/10.6028/NIST.SP.800-171r1

HR001118S0044 LEARNING WITH LESS LABELS (LWLL) 14

III. Eligibility Information

A. Eligible Applicants

DARPA welcomes engagement from all responsible sources capable of satisfying the Government’s needs, including academia (colleges and universities); businesses (large, small, small disadvantaged, etc.); other organizations (including non-profit); entities (foreign, domestic, and government); FFRDCs; minority institutions; and others.

DARPA welcomes engagement from non-traditional sources in addition to current DARPA performers.

1. Federally Funded Research and Development Centers (FFRDCs) and Government Entities

a. FFRDCs FFRDCs are subject to applicable direct competition limitations and cannot propose to this BAA in any capacity unless they meet the following conditions: (1) FFRDCs must clearly demonstrate that the proposed work is not otherwise available from the private sector. (2) FFRDCs must provide a letter on official letterhead from their sponsoring organization citing the specific authority establishing their eligibility to propose to Government solicitations and compete with industry, and their compliance with the associated FFRDC sponsor agreement’s terms and conditions. This information is required for FFRDCs proposing to be awardees or subawardees.

b. Government Entities Government Entities (e.g., Government/National laboratories, military educational institutions, etc.) are subject to applicable direct competition limitations. Government entities must clearly demonstrate that the work is not otherwise available from the private sector and provide written documentation citing the specific statutory authority and contractual authority, if relevant, establishing their ability to propose to Government solicitations.

c. Authority and Eligibility At the present time, DARPA does not consider 15 U.S.C. § 3710a to be sufficient legal authority to show eligibility. While 10 U.S.C.§ 2539b may be the appropriate statutory starting point for some entities, specific supporting regulatory guidance, together with evidence of agency approval, will still be required to fully establish eligibility. DARPA will consider FFRDC and Government entity eligibility submissions on a case-by-case basis; however, the burden to prove eligibility for all team members rests solely with the proposer.

2. Foreign Participation Non-U.S. organizations and/or individuals may participate to the extent that such participants comply with any necessary nondisclosure agreements, security regulations, export control laws, and other governing statutes applicable under the circumstances.

HR001118S0044 LEARNING WITH LESS LABELS (LWLL) 15

B. Organizational Conflicts of Interest

FAR 9.5 Requirements In accordance with FAR 9.5, proposers are required to identify and disclose all facts relevant to potential OCIs involving the proposer’s organization and any proposed team member (subawardee, consultant). Under this Section, the proposer is responsible for providing this disclosure with each proposal submitted to the BAA. The disclosure must include the proposer’s, and as applicable, proposed team member’s OCI mitigation plan. The OCI mitigation plan must include a description of the actions the proposer has taken, or intends to take, to prevent the existence of conflicting roles that might bias the proposer’s judgment and to prevent the proposer from having unfair competitive advantage. The OCI mitigation plan will specifically discuss the disclosed OCI in the context of each of the OCI limitations outlined in FAR 9.505-1 through FAR 9.505-4.

Agency Supplemental OCI Policy In addition, DARPA has a supplemental OCI policy that prohibits contractors/performers from concurrently providing Scientific Engineering Technical Assistance (SETA), Advisory and Assistance Services (A&AS) or similar support services and being a technical performer.

Therefore, as part of the FAR 9.5 disclosure requirement above, a proposer must affirm whether the proposer or any proposed team member (subawardee, consultant) is providing SETA, A&AS, or similar support to any DARPA office(s) under: (a) a current award or subaward; or (b) a past award or subaward that ended within one calendar year prior to the proposal’s submission date.

If SETA, A&AS, or similar support is being or was provided to any DARPA office(s), the proposal must include:

The name of the DARPA office receiving the support;

The prime contract number;

Identification of proposed team member (subawardee, consultant) providing the support; and An OCI mitigation plan in accordance with FAR 9.5.

Government Procedures In accordance with FAR 9.503, 9.504 and 9.506, the Government will evaluate OCI mitigation plans to avoid, neutralize or mitigate potential OCI issues before award and to determine whether it is in the Government’s interest to grant a waiver. The Government will only evaluate OCI mitigation plans for proposals that are determined selectable under the BAA evaluation criteria and funding availability.

The Government may require proposers to provide additional information to assist the Government in evaluating the proposer’s OCI mitigation plan.

If the Government determines that a proposer failed to fully disclose an OCI; or failed to provide the affirmation of DARPA support as described above; or failed to reasonably provide additional information requested by the Government to assist in evaluating the proposer’s OCI mitigation plan, the Government may reject the proposal and withdraw it from consideration for award.

HR001118S0044 LEARNING WITH LESS LABELS (LWLL) 16

C. Cost Sharing/Matching

Cost sharing is not required; however, it will be carefully considered where there is an applicable statutory condition relating to the selected funding instrument (e.g., OTs under the authority of 10 U.S.C. § 2371).

D. Other Eligibility Requirements

Each proposal submitted in response to this BAA shall address only one technical area.

Organizations may submit multiple proposals to any one TA, or they may propose to multiple TAs. The decision as to which proposal to consider for award is at the discretion of the Government; however, there are no conflicts between technical areas.

HR001118S0044 LEARNING WITH LESS LABELS (LWLL) 17

IV. Application and Submission Information

A. Address to Request Application Package

This document contains all information required to submit a response to this solicitation. No additional forms, kits, or other materials are needed except as referenced herein. No request for proposal (RFP) or additional solicitation regarding this opportunity will be issued, nor is additional information available except as provided at the Federal Business Opportunities website (https://www.fbo.gov), the Grants.gov website (http://www.grants.gov/), or referenced herein.

B. Content and Form of Application Submission

1. Abstracts Proposers are highly encouraged to submit an abstract in advance of a proposal to minimize effort and reduce the potential expense of preparing an out of scope proposal. The abstract provides a synopsis of the proposed project, including brief answers to the following questions:

What is the proposed work attempting to accomplish or do?

How is it done today, and what are the limitations?

Who will care and what will the impact be if the work is successful?

How much will it cost, and how long will it take?

DARPA will respond to abstracts with a statement as to whether DARPA is interested in the idea. If DARPA does not recommend the proposer submit a full proposal, DARPA will provide feedback to the proposer regarding the rationale for this decision. Regardless of DARPA’s response to an abstract, proposers may submit a full proposal. DARPA will review all full proposals submitted using the published evaluation criteria and without regard to any comments resulting from the review of an abstract.

Abstract Format: Abstracts shall not exceed a maximum of four (4) pages per technical area (maximum of 8 pages if submitting to TA1 and TA2), including the cover sheet and all figures, tables, and charts. The page limit does not include a submission letter (optional).

All pages shall be formatted for printing on 8-1/2 by 11-inch paper with 1-inch margins and font size not smaller than 12 point. Font sizes of 8 or 10 point may be used for figures, tables, and charts. Document files must be in .pdf, .odx, .doc, .docx, .xls, or .xlsx formats.

Submissions must be written in English. All pages should be numbered.

Abstracts must include the following components:

Cover Sheet: Provide the administrative and technical points of contact (name, address, phone, email, lead organization). Include the BAA number, title of the proposed project, primary subcontractors, estimated cost, duration of the project, and the label “Abstract.”

Goals and Impact: Describe what is being proposed and what difference it will make (qualitatively and quantitatively) if successful. Describe the innovative aspects of the project in the context of existing capabilities and approaches, clearly delineating the https://www.fbo.gov/ http://www.grants.gov/

HR001118S0044 LEARNING WITH LESS LABELS (LWLL) 18

relationship of this work to any other projects from the past and present.

Technical Plan: Outline and address all technical challenges inherent in the approach and possible solutions for overcoming potential problems. Provide appropriate specific milestones (quantitative, if possible) at intermediate stages of the project to demonstrate progress.

Capabilities/Management Plan: Provide a brief summary of expertise of the team, including subcontractors and key personnel. Identify a principal investigator for the project and include a description of the team’s organization including roles and responsibilities. Describe the organizational experience in this area, existing intellectual property required to complete the project, and any specialized facilities to be used as part of the project. List Government-furnished property, facilities, or data assumed to be available. If desired, include a brief bibliography with links to relevant papers, reports, or resumes of key performers. Do not include more than two resumes as part of the abstract. Resumes count against the abstract page limit.

Statement of Work, Cost and Schedule: Provide a cost estimate for resources over the proposed timeline of the project, broken down by year. Include labor, materials, a list of deliverables and delivery schedule. Provide cost estimates for each subcontractor (may be a rough order of magnitude).

2. Proposals Proposals consist of Volume 1: Technical and Management Proposal (including mandatory Appendix A and optional Appendix B); Volume 2: Cost Proposal; the Level of Effort Summary by Task Excel spreadsheet; and the PowerPoint summary slide.

All pages shall be formatted for printing on 8-1/2 by 11-inch paper with 1-inch margins, single-line spacing, and a font size not smaller than 12 point. Font sizes of 8 or 10 point may be used for figures, tables, and charts. Document files must be in .pdf, .odx, .doc, .docx, .xls, or .xlsx formats. Submissions must be written in English. All pages of Volume 1 should be numbered.

A summary slide of the proposed effort, in PowerPoint format, should be submitted with the proposal. A template slide is provided as an attachment to the BAA. Submit this PowerPoint file in addition to Volumes 1 and 2 of your full proposal, and the Level of Effort Summary by Task Excel spreadsheet. This summary slide does not count towards the total page count.

Reminder – Each proposal submitted in response to this BAA shall address only one TA.

Organizations may submit multiple proposals to any one TA, or they may propose to multiple TAs.

Proposals not meeting the format prescribed herein may not be reviewed.

a. Volume 1: Technical and Management Proposal The maximum page count for Volume 1 is 30 pages for proposals addressing one technical area and 40 pages for proposals addressing both technical areas, including all figures, tables, HR001118S0044 LEARNING WITH LESS LABELS (LWLL) 19 and charts but not including the cover sheet, table of contents or appendices. A submission letter is optional and is not included in the page count. Appendix A does not count against the page limit and is mandatory. Appendix B does not count against the page limit and is optional. Additional information not explicitly called for here must not be submitted with the proposal, but may be included in the bibliography in Appendix B. Such materials will be considered for the reviewers’ convenience only and not evaluated as part of the proposal.

Volume 1 must include the following components:

i. Cover Sheet: Include the following information.

Label: “Proposal: Volume 1” BAA number (HR001118S0044) Technical Area Proposal title Lead organization (prime contractor) name Type of organization, selected from the following categories: Large Business, Small Disadvantaged Business, Other Small Business, HBCU, MI, Other Educational, or Other Nonprofit

Technical point of contact (POC) including name, mailing address, telephone, and email

Administrative POC including name, mailing address, telephone number, and email address

Award instrument requested: procurement contract (specify type), cooperative agreement or OT.5

Total amount of the proposed effort Place(s) and period(s) of performance Other team member (subcontractors and consultants) information (for each, include Technical POC name, organization, type of organization, mailing address, telephone number, and email address)

Proposal validity period (minimum 180 days) Data Universal Numbering System (DUNS) number6 Taxpayer identification number7 Commercial and Government Entity (CAGE) code8 Proposer’s reference number (if any)

ii. Table of Contents

5 Information on award instruments can be found at http://www.darpa.mil/work-with-us/contract-management.

6 The DUNS number is used as the Government's contractor identification code for all procurement-related activities. Go to http://fedgov.dnb.com/webform/index.jsp to request a DUNS number (may take at least one business day). For further information regarding this subject, please see www.darpa.mil/work-with-us/additional-baa for further information.

7 See http://www.irs.gov/businesses/small/international/article/0,,id=96696,00.html for information on requesting a TIN. Note, requests may take from 1 business day to 1 month depending on the method (online, fax, mail).

8 A CAGE Code identifies companies doing or wishing to do business with the Federal Government. For further information regarding this subject, please see www.darpa.mil/work-with-us/additional-baa.

http://www.darpa.mil/work-with-us/contract-management http://fedgov.dnb.com/webform/index.jsp http://www.darpa.mil/work-with-us/additional-baa http://www.darpa.mil/work-with-us/additional-baa http://www.irs.gov/businesses/small/international/article/0,,id=96696,00.html

HR001118S0044 LEARNING WITH LESS LABELS (LWLL) 20

iii. Executive Summary: Provide a synopsis of the proposed project, including answers to the following questions:

What is the proposed work attempting to accomplish or do?

How is it done today, and what are the limitations?

Who or what will be affected and what will be the impact if the work is successful?

How much will it cost, and how long will it take?

The executive summary should include a description of the key technical challenges, a concise review of the technologies proposed to overcome these challenges and achieve the project’s goal, and a clear statement of the novelty and uniqueness of the proposed work.

iv. Innovative Claims and Deliverables: Describe the innovative aspects of the project in the context of existing capabilities and approaches, clearly delineating the uniqueness and benefits of this project in the context of the state of the art, alternative approaches, and other projects from the past and present. Describe how the proposed project is revolutionary and how it significantly rises above the current state of the art.

Describe the deliverables associated with the proposed project and any plans to commercialize the technology, transition it to a customer, or further the work. Discuss the mitigation of any issues related to sustainment of the technology over its entire lifecycle, assuming the technology transition plan is successful.

v. Technical Plan: Outline and address technical challenges inherent in the approach and possible solutions for overcoming potential problems. Demonstrate a deep understanding of the technical challenges and present a credible (even if risky) plan to achieve the project’s goal. Discuss mitigation of technical risk. Provide appropriate measurable milestones (quantitative if possible) at intermediate stages of the project to demonstrate progress and a plan for achieving the milestones.

vi. Management Plan: Provide a summary of expertise of the proposed team, including any subcontractors/consultants and key personnel who will be executing the work.

Resumes count against the proposal page limit so proposers may wish to include them in Appendix B. Identify a principal investigator (PI) for the project. Provide a clear description of the team’s organization including an organization chart that includes, as applicable, the relationship of team members; unique capabilities of team members; task responsibilities of team members; teaming strategy among the team members; and key personnel with the amount of effort to be expended by each person during the project.

Provide a detailed plan for coordination including explicit guidelines for interaction among collaborators/subcontractors of the proposed project. Include risk management approaches. Describe any formal teaming agreements that are required to execute this project. List Government-furnished materials or data assumed to be available.

vii. Personnel, Qualifications, and Commitments: List key personnel (no more than one page per person), showing a concise summary of their qualifications, discussion of previous accomplishments, and work in this or closely related research areas. Indicate the level of effort in terms of hours to be expended by each person during each contract

HR001118S0044 LEARNING WITH LESS LABELS (LWLL) 21

year and other (current and proposed) major sources of support for them and/or commitments of their efforts. DARPA expects all key personnel associated with a proposal to make a substantial time commitment to the proposed activity, and the proposal will be evaluated accordingly. It is DARPA’s intention to put key personnel conditions into the awards, so proposers should not propose personnel that are not anticipated to execute the award.

Include a table of key individual time commitments as follows:

Hours on Project Key

Individual Project

Status (Current, Pending, Proposed) Phase 1 Phase 2 LwLL Proposed x x

Project Name 1 Current x xName 1 Project Name 2 Pending n/a x

LwLL Proposed x x Name 2

Project Name 3 Proposed x x

viii. Capabilities: Describe organizational experience in relevant subject area(s), existing intellectual property, or specialized facilities. Discuss any work in closely related research areas and previous accomplishments.

ix. Statement of Work (SOW): The SOW must provide a detailed task breakdown, citing specific tasks and their connection to the interim milestones and metrics, as applicable. Each year of the project should be separately defined. The SOW must not include proprietary information. For each defined task/subtask, provide:

A general description of the objective.

A detailed description of the approach to be taken to accomplish each defined task/subtask.

Identification of the primary organization responsible for task execution

(prime contractor, subcontractor[s], consultant[s]), by name.

A measurable milestone, (e.g., a deliverable, demonstration, or other event/activity that marks task completion).

A definition of all deliverables (e.g., data, reports, software) to be provided to the Government in support of the proposed tasks/subtasks.

Identify any tasks/subtasks (by the prime or subcontractor) that will be accomplished at a university and believed to be fundamental research.

x. Schedule and Milestones: Provide a detailed schedule showing tasks (task name, duration, work breakdown structure element as applicable, performing organization), milestones, and the interrelationships among tasks. The task structure must be consistent with that in the SOW. Measurable milestones should be clearly articulated and defined in time relative to the start of the project.

xi. Appendix A: This section is mandatory and must include all of the following components. If a particular subsection is not applicable, state “NONE”.

(1). Team Member Identification: Provide a list of all team members including the

HR001118S0044 LEARNING WITH LESS LABELS (LWLL) 22

prime, subcontractor(s), and consultant(s), as applicable. Identify specifically whether any are a non-US organization or individual, FFRDC and/or Government entity. Use the following format for this list:

Non-US?

Individual

Name

Role (Prime, Subcontractor or Consultant)

Organization Org. Ind.

FFRDC

or

Govt?

(2). Government or FFRDC Team Member Proof of Eligibility to Propose: If none of the team member organizations (prime or subcontractor) are a Government entity or FFRDC, state “NONE”.

If any of the team member organizations are a Government entity or FFRDC, provide documentation (per Section III.A.1) citing the specific authority that establishes the applicable team member’s eligibility to propose to Government solicitations to include: 1) statutory authority; 2) contractual authority;

3) supporting regulatory guidance; and 4) evidence of agency approval for applicable team member participation.

(3). Government or FFRDC Team Member Statement of Unique Capability: If none of the team member organizations (prime or subcontractor) are a Government entity or FFRDC, state “NONE”.

If any of the team member organizations are a Government entity or FFRDC, provide a statement (per Section III.A.1) that demonstrates the work to be performed by the Government entity or FFRDC team member is not otherwise available from the private sector.

(4). Organizational Conflict of Interest Affirmations and Disclosure: If none of the proposed team members is currently providing SETA or similar support as described in Section III.B, state “NONE”.

If any of the proposed team members (individual or organization) is currently performing SETA or similar support, furnish the following information:

Prime Contract Number

DARPA

Technical Office supported

A description of the action the proposer has taken or proposes to take to avoid, neutralize, or mitigate the conflict

(5). Intellectual Property (IP): If no IP restrictions are intended, state “NONE”.

The Government will assume unlimited rights to all IP not explicitly identified as having less than unlimited rights in the proposal.

HR001118S0044 LEARNING WITH LESS LABELS (LWLL) 23

For all technical data or computer software that will be furnished to the Government with other than unlimited rights, provide (per Section VI.B.1) a list describing all proprietary claims to results, prototypes, deliverables or systems supporting and/or necessary for the use of the research, results, prototypes and/or deliverables. Provide documentation proving ownership or possession of appropriate licensing rights to all patented inventions (or inventions for which a patent application has been filed) to be used for the proposed project. Use the following format for these lists:

NONCOMMERCIAL

Technical Data and/or Computer Software To be Furnished With Restrictions

Summary of Intended Use in the Conduct of the Research

Basis for Assertion

Asserted Rights

Category

Name of Person Asserting Restrictions

(List) (Narrative) (List) (List) (List) (List) (Narrative) (List) (List) (List)

COMMERCIAL

Technical Data and/or Computer Software To be Furnished With Restrictions

Summary of Intended Use in the Conduct of the Research

Basis for Assertion

Asserted Rights

Category

Name of Person Asserting Restrictions

(List) (Narrative) (List) (List) (List) (List) (Narrative) (List) (List) (List)

(6). Human Subjects Research (HSR): If HSR is not a factor in the proposal, state

“NONE”.

If the proposed work will involve human subjects, provide evidence of or a plan for review by an institutional review board (IRB). For further information on this subject, see Section VI.B.2.

(7). Animal Use: If animal use is not a factor in the proposal, state “NONE”.

If the proposed research will involve animal use, provide a brief description of the plan for Institutional Animal Care and Use Committee (IACUC) review and approval. For further information on this subject, see Section VI.B.2.

(8). Representations Regarding Unpaid Delinquent Tax Liability or a Felony Conviction under Any Federal Law: For further information…