DARPA-BAA-16-32.pdf

DARPA-BAA-16-32/NGS2 BAA 1

Broad Agency Announcement Next Generation Social Science (NGS2)

Defense Sciences Office

DARPA-BAA-16-32

March 18, 2016

DARPA-BAA-16-32/NGS2 BAA 2

Table of Contents I. Funding Opportunity Description II. Award Information

A. Awards B. Fundamental Research

III. Eligibility Information A. Eligible Applicants B. Procurement Integrity, Standards of Conduct, Ethical Considerations, and Organizational Conflicts of Interest C. Cost Sharing/Matching D. Other Eligibility Requirements

IV. Application and Submission Information A. Address to Request Application Package B. Content and Form of Application Submission C. Submission Dates and Times D. Funding Restrictions E. Other Submission Requirements

V. Application Review Information A. Evaluation Criteria B. Proposal Review and Selection Process

VI. Award Administration Information A. Selection Notices B. Administrative and National Policy Requirements C. Reporting

VII. Agency Contacts VIII. Other Information

A. Frequently Asked Questions (FAQs) B. Proposers Day C. Collaborative Efforts/Teaming

ATTACHMENTS – Mandatory Templates

Attachment 1: Abstract Template – Executive Summary Slide Attachment 2: Proposal Template Slides o Slide 1: Executive Summary o Slide 2: Concept o Slide 3: Cost Summary by Phase o Slide 4: Cost Summary by Task

Attachment 3: Proposal Template – Milestones and Deliverables Summary Table Attachment 4: Proposal Template – Technical and Management Volume Attachment 5: Proposal Template – Cost Volume Attachment 6: Proposal Template – Administrative and National Policy Requirements

DARPA-BAA-16-32/NGS2 BAA 3

PART I: OVERVIEW INFORMATION

Federal Agency Name: Defense Advanced Research Projects Agency (DARPA), Defense Sciences Office (DSO)

Funding Opportunity Title: Next Generation Social Science (NGS2)

Announcement Type: Initial Announcement

Funding Opportunity Number: DARPA-BAA-16-32

Catalog of Federal Domestic Assistance (CFDA) Number(s): 12.910 Research and Technology Development

Dates (All times listed herein are Eastern Time.)

o Posting Date: March 18, 2016 o Proposers Day: March 22, 2016. See Section VIII.B.

o Teaming Profile Due Date: March 28, 2016. See Section VIII.C.

o Abstract Due Date: March 31, 2016, 4:00 p.m.

o FAQ Submission Deadline: May 11, 2016, 4:00 p.m. See Section VIII.A.

o Full Proposal Due Date: May 18, 2016, 4:00 p.m.

Anticipated Funding Available for Award: DARPA anticipates awards to multiple development teams with the majority of funding reserved for teams with an integrated, end-to-end research approach. Only one award is anticipated for a Testing and Evaluation team, as described herein.

Types of Instruments that May be Awarded: Procurement contracts, grants, cooperative agreements or other transactions.

Agency contacts o Technical POC: Adam Russell, Program Manager, DARPA/DSO o Solicitation Email: NGS2@darpa.mil o Solicitation Mailing Address:

DARPA/DSO

ATTN: DARPA-BAA-16-32

675 North Randolph Street Arlington, VA 22203-2114 o DARPA/DSO Solicitation Website: http://www.darpa.mil/work-with-us/opportunities

Frequently Asked Questions (FAQ): FAQs for this solicitation may be viewed on the DSO Solicitation Website. See Section VIII.A for further information.

mailto:NGS2@darpa.mil http://www.darpa.mil/work-with-us/opportunities?tFilter=&oFilter=2&sort=name http://www.darpa.mil/work-with-us/opportunities?tFilter=&oFilter=2&sort=name

DARPA-BAA-16-32/NGS2 BAA 4

PART II: FULL TEXT OF ANNOUNCEMENT

I. Funding Opportunity Description This Broad Agency Announcement (BAA) is being issued, and any resultant selection will be made, using procedures under Federal Acquisition Regulation (FAR) 35.016. Any negotiations and/or awards will use procedures under FAR 15.4 (or 32 CFR 22 for grants and cooperative agreements). Any amendments to this BAA will be posted to the Federal Business Opportunities (FBO) website (http://www.fbo.gov/) and, as applicable, the Grants.gov website (http://www.grants.gov/).

A. Introduction

The Defense Sciences Office at the Defense Advanced Research Projects Agency (DARPA) is soliciting innovative research proposals to build a new capability (methods, models, tools, and a community of researchers) to perform rigorous, reproducible experimental research at scales necessary to understand emergent properties of human social systems. DARPA anticipates that the Next Generation Social Science (NGS2) program may require a fundamental reimagining of the social science research cycle and encourages participation from a wide and diverse combination of disciplines and skill sets – to include social sciences, but also physics, computer science, biology, game design, mathematics, and others. Specifically excluded is research that primarily results in incremental improvements to the existing state of practice.

Performers in the NGS2 program will work to determine fundamental measures and causal mechanisms that explain and predict the emergence of “collective identities.” A focus on “what matters most” for emergent social phenomena like collective identity presents an important and complex challenge to focus and validate new NGS2 research communities, tools, and methods.

Note that while the NGS2 program will focus on collective identity formation as an exemplar research question, DARPA anticipates that successful NGS2 capabilities will have benefits for tackling other complex problems and topics, including (but not limited to): resilience in social networks and structures, changes in cultural norms or beliefs, emergence of cooperation/competitions, social influences on preferences and cognition, etc.

B. Program Structure

The NGS2 program is divided into two phases, a 24-month base period (Phase 1) with one 18-month option period (Phase 2). Each phase will consist of two research cycles. For the sake of clarity, this BAA will reference the conceptual elements of each research cycle in terms of the following Technical Areas (TAs), described in detail below:

TA1: Predictive Modeling and Hypothesis Generation;

TA2: Experimental Methods and Platforms; and TA3: Interpretation and Reproducibility.

In Phase 1, performers will develop and demonstrate research tools and methods for rapidly testing and evaluating the accuracy of experimental hypotheses and predictions in multiple populations.

http://www.fbo.gov/ http://www.grants.gov/

DARPA-BAA-16-32/NGS2 BAA 5

Teams that demonstrate progress or promising technical approaches in support of NGS2 goals and TAs during Phase 1 may be encouraged to continue their research efforts in the next phase. In Phase 2, performers will prove out their capabilities and demonstrate the replicability and rigor of their methodologies and models. Performers will evaluate their models’ external validity and robustness by scoring their predictive accuracy across the experiments of other Phase 2 performers.

All proposers are expected to fully address how the proposed research will contribute to NGS2 program goals. The proposal narrative should clearly explain the technical approach and fully describe the research and development required to build and test proposed methods, models and tools. Proposers must also describe how the work advances the state of the art as applied to social science in general and the exemplar problem in particular. Proposals should also identify the highest development, integration, and scaling risks involved and clearly describe how those risks will be mitigated early in the proposed effort. A summary of the NGS2 program structure is provided in Figure 1, below.

1. Performer Categories

Performance in the NGS2 program will occur in one of three categories: End-to-End (ETE), Enablers, and Test & Evaluation (T&E). As described below, performers in the first two categories will focus on research and development and the third will perform independent testing and evaluation. Proposers may submit abstracts and proposals in any of these three categories.

However, to avoid any real or perceived conflicts of interest between development and T&E, proposers participating on a T&E submission may not participate on an Enabler or ETE team submission (See Section III.D for further information.). Proposers must specify the category to which they are applying on their submission cover sheet(s).

ETE: One or more complete multidisciplinary teams executing an end-to-end research approach that addresses each of the TAs described below. These teams are expected to engage in multiple cycles of modeling, hypothesis generation, prediction, experimentation, analysis, and reevaluation beginning in Phase 1 and continuing through Phase 2.

Enablers: Smaller, targeted efforts to develop and test new or early stage high-risk, high-payoff “enabling” technologies in one or more of the TAs. At the end of Phase 1, this second category of performer should enable some new capability (e.g., greater speed, more rigor, higher transparency, more dynamic informed consent, larger numbers of participants) and be poised for integration into a complete end-to-end approach. Proposals in this category must detail the unique enabling technical or methodical approach to be developed in regard to: the next generation research capability made possible by the proposed approach; the feasibility of achieving this goal during Phase 1; and the effort required for maturation, including a risk-reducing technical development plan with clear progress milestones. Proposers should also illustrate how they anticipate the enabling approach integrating with, and thereby impacting, Phase 2 research cycles.

T&E: A single team responsible for assisting the Government with testing and evaluation of the other performers’ programmatic and technical progress towards NGS2 goals. Proposers in

DARPA-BAA-16-32/NGS2 BAA 6

this category must have demonstrated experience supporting scientific experimentation, diverse collaboration and evaluation of the accuracy of experimental predictions. They must also have a history of effectively interacting with multiple research communities, including the social sciences, to help advance multidisciplinary, reproducible research in line with NGS2 goals. Proposers should discuss their capabilities in managing experiment registration, innovative forecasting and prediction platforms, engagement with appropriate scientific publications for registered reports1 of performer protocols as a form of early peer review, and providing consultation, infrastructure support, and collaboration enhancement across scientific teams and geographies. In addition to these activities, the T&E team will provide feedback to the ETE and Enabler performers and DARPA on plans and reports, assist DARPA with program research, and provide overarching documentation of program progress, implementation, and outcomes. As directed by DARPA, the T&E team will also help support transition of successful program products to DoD partners as well as the wider social science research community.

All proposers must describe their approach—and structure their Statements of Work (SOW)—in ways that reflect and address the following TAs and the research process considerations outlined in each description.

1 “Registered Reports emphasize the importance of the research question and the quality of methodology by conducting the peer review prior to data collection and analysis. Accepted papers then are virtually guaranteed publication in the journal if the authors follow through with the registered methodology.” Source: https://osf.io/8mpji/wiki/home/ https://osf.io/8mpji/wiki/home/

DARPA-BAA-16-32/NGS2 BAA 7

Figure 1: Program Structure

DARPA-BAA-16-32/NGS2 BAA 8

2. Technical Area Descriptions

TA1 - Predictive Modeling and Hypothesis Generation: TA1 seeks to enhance capabilities to rapidly identify, formalize, and empirically test hypotheses derived from a wide range of models and theories, each of which may make competing claims about what factors matter most, when, and why, for the emergence of collective identify. A major goal of TA1 is to formalize different variables and parameters from multiple models and theories – including social and behavioral sciences, but also potentially models or theories from epidemiology, biology, ecology, physics, network science, etc., or combinations thereof – in order to compare their accuracy in predicting the direction and size of effects of different experimental interventions. This TA also seeks innovative approaches for accounting for biases across research bodies and literatures, and to increase models’ responsiveness to changing evidence, to include ensemble modeling. Performers should therefore be able to tackle a number of current challenges, including: how to formalize and compare time-dependent predictions from various stochastic, deterministic, and/or agent-based models; how to push theoretical or descriptive models towards quantitative prediction, while pushing quantitative but often atheoretical models towards greater description; how to connect low level, quantifiable data to higher order theory while managing dimensionality; and how to derive sufficiently precise, falsifiable predictions to help disconfirm otherwise plausible models/theories.

ETE proposers should address TA1 by detailing their proposed approaches, tools, and/or methods for (at a minimum):

Identifying multiple theories and models, and potentially combinations thereof, that speak to collective identity formation from a diverse set of disciplines and literatures;

Formalizing parametric implementations of those models;

Recognizing and accounting for various biases in research and publications to appropriately re-weight specific model variables, parameters, and/or dynamics;

Articulating and registering specific, testable predictions of the direction and size of effects of interventions on different outcome measures; and, Documenting models to facilitate reproducibility, while updating models based on empirical results.

As relevant, Enabler and T&E proposers should detail their proposed tools, approaches, and/or methods to contribute to one or more TA1 goals. They should also describe the anticipated impact and the expected outcome(s) and benefit(s) of their research for the NGS2 program, specifically, and social science more generally.

TA2 - Experimental Methods and Platforms: TA2 seeks to enable new capabilities for the design and conduct of multifactorial experimental research of sufficient complexity, time, number of participants, etc., to test the accuracy and robustness of models’ predictions related to the emergence of collective identity. For NGS2 , the term “experimental” is used to highlight a focus on predicting and testing hypothesized causal mechanisms by using ethical and empirically sound experimental approaches (including interventions, randomization, control groups, etc.), as opposed to approaches based largely on observation and correlation. ETE proposers should address TA2 by detailing their proposed approaches, tools and/or methods for (at a minimum):

Experimental design

DARPA-BAA-16-32/NGS2 BAA 9

o Designing multifactorial (many-to-many) experiments that effectively and credibly operationalize dependent measures of collective identity formation;

o Creating or leveraging experimental platforms and narratives that engage participants, that are easy to understand, have face and ecological-validity to elicit relevant behaviors and knowable ground truth, and provide sufficient sample sizes and diversity;

o Running experiments that allow multiple independent variables to be manipulated in order to test a wide range of different model predictions—potentially with “rolling predictions” for particular time-dependent interventions or manipulations—while facilitating reproducibility of results;

o Communicating experimental plans to various Human Subject Research Protection organizations, to include DoD Secondary Review (see Section VI.B.2 for more detail) and international partners as appropriate;

o Replicating and extending experiments across multiple research cycles;

Participant recruitment and retention o Determining and justifying the numbers of participants that a proposer will seek to include, sufficient to test model predictions across at least 3 different and representatively diverse populations, with multiple intervention and control groups in each research cycle (i.e., a total of 12 different populations over the entire NGS2 program). Proposers should include their rationale for choosing these populations, inclusion/exclusion criteria, etc.;

o Developing experimental narratives and/or tasks of sufficient engagement and interest to attract the number of volunteer participants required for testing;

o Approaches to informed consent and participant recruitment;

o Negotiating potential language differences, demographic variation, cultural variables, etc., that may substantively impact experimental methods and data collection;

Data collection o Capturing and integrating subjective and objective measures using mixed methods;

o Quantifying and accounting for various biases in experimental collection, such as platform- or media-specific bias, sampling bias, biases in existing data corpora, etc.;

and, o Approaches to protecting PII and other potentially sensitive data.

As relevant, Enabler and T&E proposers should detail their proposed tools, approaches, and/or methods to contribute to one or more TA2 goals. They should also describe the anticipated impact and the expected outcome(s) and benefit(s) of their research for the NGS2 program, specifically, and social science more generally.

TA3 - Interpretation and Reproducibility: TA3 seeks to advance capabilities for processing, analyzing, and visualizing data collected through next generation experiments, scoring experimental predictions for accuracy, determining the level (or lack) of evidentiary support of models, and enabling reproducibility (i.e., strictly reproducing results given access to the original datasets, code, and analytic procedures) as well as replication (i.e., the ability to replicate an experiment in another condition, with another population, and/or by different researchers). ETE

DARPA-BAA-16-32/NGS2 BAA 10

proposers should address TA3 by detailing their proposed approaches, tools and/or methods for (at a minimum):

Analyzing data collected over long(er) periods of time, involving potentially complex, non-linear, and recursive relationships of variables with large numbers of co-variates and mediators, etc.;

Conducting confirmatory and exploratory analyses in order to enhance interpretation, reproducibility, replication, and efforts to explain residuals related to specific research questions;

Dealing with noisy and/or missing data across potentially large multi-modal data sets;

Quantifying, incorporating and communicating different types of uncertainty in behaviors, measures, and sampling;

Determining, validating and visualizing effect sizes and variance explained/predicted by different models; and Data management, including approaches to documenting and sharing data, meta-data and code(s) for other performers to reproduce results, as well as preserving data for future use by a wider research community (see also Section I.C.1., Data Management).

As relevant, Enabler and T&E proposers should detail their proposed tools, approaches, and/or methods to contribute to one or more TA3 goals. They should also describe the anticipated impact and the expected outcome(s) and benefit(s) of their research for the NGS2 program, specifically, and social science more generally.

3. Phases and Research Cycles

NGS2 is organized into two phases as shown in Table 1. For Phase 1, ETE performers will be expected to design and conduct two research cycles (Cycle 1 and Cycle 2), with each cycle involving at least one experiment to test multiple hypotheses regarding collective identity formation with at least three diverse and representative populations. Each cycle also includes analysis and data sharing to ensure reproducibility of results (explained in further detail below in Section I.C.5), as each ETE performer will reproduce the analysis of other ETE teams using the data and tools provided to them by those other teams. They will in turn have their results reproduced by the other ETE teams. For reproducibility budgeting and scheduling purposes, ETE teams should assume at least two other ETE teams. After Cycle 1 is complete and all results have been reproduced, ETE performers will then replicate their experiments in Cycle 2 with three new participant populations. Performers may update their model predictions based on Cycle 1 outcomes, but minimal experimental design changes are expected.

In Phase 2, ETE performers will both replicate and accelerate their Phase 1 approaches in two more research cycles (Cycles 3 and 4). Phase 2 seeks to mature experimental accuracy and external validity of performer models by adjusting parameters to account for Phase 1 results from all performers. As currently envisioned, Phase 2 ETE performers will be evaluated on the accuracy of their experimental predictions for all Phase 2 performers’ experiments.

ETE proposals should therefore consider process changes and coordination potentially required for their Phase 2 experiments to enable other teams to make predictions of effect direction and size for various interventions and manipulations.

DARPA-BAA-16-32/NGS2 BAA 11

Table 1: Technical Goals by Phase Performer Category Technical Area Phase 1 Phase 2

TA1: Predictive Modeling & Hypothesis Generation

• Identify multiple models and formalize predictions for own Phase 1 experiments

• Develop methods to adjust models & account for bias

• Adjust successful models based on Phase 1 results

• Enhance explanation and prediction of models

• Make specific model-based predictions for all Phase 2 experiments

TA2: Experimental Methods & Platforms

• Design experiments to test models with specific, preregistered predictions & analytic plans

• Cycle 1 (C1): Conduct experiments with 3 populations (appropriate sample sizes)

• Cycle 2 (C2): Replicate Cycle 1 experiment with 3 new populations (appropriate sample sizes)

• Develop methods to allow all performers to track team’s experimental progress, make predictions on its experiments

• Cycle 3 (C3): Replicate and extend Phase 1 experiments with three new populations (appropriate sample sizes)

• Cycle 4 (C4): Replicate C3 experiments with three new populations (appropriate sample sizes)End-to-

End

TA3: Interpretation & Reproducibility

• Incorporate and integrate multiple qualitative and quantitative data types

• Analyze results with appropriate visualization

• Data sharing with PII protection

• Score prediction accuracy

• Share data, code and analyses with other ETE performers to mutually ensure 100% reproducibility of C1 and C2 results

• Incorporate and integrate multiple qualitative and quantitative data types

• Analyze results with appropriate visualization

• Data sharing with PII protection

• Score prediction accuracy across all experiments

• Share data, code and analyses with other ETE performers to ensure 100% reproducibility of C3 and C4 results

Enablers

May address any TA topic(s), for example:

• Model development

• Prediction

• Experimental methods

• Platforms

• Analytic techniques

• Facilitation of reproducibility or replication

• Other

• Develop and test a novel research enabler addressing at least one goal in a TA

• Track and discuss development with ETE performers

• Plan integration of successful components into ETE research

Provide support (as appropriate) to integrate the enabling technology into ETE research

T&E All TAs (Evaluation)

• Manage registration and scoring of performers’ predictions and analytic plans; compare to other predictions

• Continue Phase 1 evaluation activities, modifying as appropriate

• Manage registration and scoring of performers’ and effect size

DARPA-BAA-16-32/NGS2 BAA 12

• Assist in evaluation of development, testing, and integration of Enablers predictions on all experiments;

compare to other predictions

All TAs (Support)

• Facilitate registered reports

• Consult on tools, methods, and best practices

• Support ETE and Enabler coordination as necessary

• Facilitate registered reports

• Consult on tools, methods, and best practices

• Support ETE and Enabler coordination as necessary

C. Other Program Objectives and Considerations

1. Data Management

An intended outcome of NGS2 is to enable ethical, rigorous, and reproducible experimental social science research at new scales for greater understanding of emergent properties of human social systems. DARPA anticipates that data collection, processing, curation, sharing, and preservation will be critical in achieving this outcome. Accordingly, all proposals must include a Data Management Plan (DMP). Proposals submitted without a DMP will be deemed non-conforming2 and will not be reviewed.

Proposed DMPs should support DARPA’s objectives for capturing, preserving and sharing digital data produced by scientific and engineering activities, including fulfillment of the objectives of the Federal Open Data initiative and DoD guidelines on research data sharing; advancing reproducibility and replicability in sponsored scientific and engineering projects; and, creating and promoting a foundation for new projects and programs that can exploit science and engineering data to accelerate scientific discovery and engineering innovation. As appropriate, the DMP should include, but not be limited to:

Plans for data sharing—to include extent and mechanisms during and after the NGS2 program

Description of the hosting environment(s) for sharing digital research data with user communities

Any data management standards, including meta-data standards, and/or community best practices that may apply

Plans for “data persistence” and preservation beyond the program Rough estimates of data kinds and assets; formats; sizes (e.g., KB, MB, GB, TB), etc.

Kinds of data might include:

o Data sets: experimental, test, and measurement data o Narratives: observational logs, journals, collaborations o Analyses o Decisions: alternatives, exploration branches, determinations o Experiments and simulations: setup, ingest, outputs o Codes (with build scripts), software (executables with source), algorithms, data consumed or produced by software o Models or simulations o Bibliographies and citations used by your research o Recordings of various physical phenomena (including images)

2 “Conforming” is defined as having been submitted in accordance with the requirements outlined herein.

DARPA-BAA-16-32/NGS2 BAA 13

Methods for addressing and protecting sensitive data, to include participant anonymity, privacy or data redaction

Anticipated current or future data quality issues Anticipated costs for digital data management How the DMP enhances validation and reproducibility of results How the DMP may support future innovation

2. Collaboration

Throughout the course of the NGS2 program, it is likely to be necessary for all performers— regardless of category—to share relevant information regarding their research and development to support the larger program goals. For example, ETE performers may need insight into the TA1, TA2, or TA3 approaches being tested by other ETE performers as well as access to data and code to reproduce their results. Similarly, ETE performers may need access to tools and methods being developed by Enablers, while Enabler performers may need insight into ETE performer approaches. The T&E team will need access to ETE and Enabler performer plans and methods.

DARPA expects all NGS2 performers to work collaboratively with one another to realize the program objectives outlined herein, so proposers should carefully review the goals for the entire program in order to fully understand the context of each program objective, performer category, and TA within the overall program structure. All proposals should describe plans for ensuring transparency of their processes to enable interactions with other NGS2 performers. Proposals that fail to include these plans may be deemed non-conforming and removed from consideration.

3. Intellectual Property

As discussed above, data sharing and collaboration are key aspects of this program. Therefore, intellectual property rights asserted by proposers are strongly encouraged to be aligned with open source regimes. See Section VI.B.1 for further information.

4. Human Subjects Research

It is anticipated that proposals submitted in response to this BAA may involve Human Subjects Research (HSR). All proposers planning to involve HSR should carefully review the HSR requirements in Section VI.B.2 of the BAA. Any required HSR information (including, but not limited to Institutional Review Board (IRB) draft application plans) must be included in the proposal. Furthermore, any task that involves HSR must be clearly identified in the proposed Statement of Work, schedule, and cost details. Proposed schedules must take into account the time required for all IRB approvals.

Proposals involving HSR that fail to supply evidence of or a plan for review by an IRB may be deemed non-conforming and, as such, will not be reviewed.

5. Validation

A key programmatic feature of NGS2 is the self-validation of each ETE performer’s models, methods, data, and analyses as well as cross-validation by other performer teams. Performers

DARPA-BAA-16-32/NGS2 BAA 14

should therefore consider carefully issues of data portability and protocol fidelity when selecting their research questions and experimental methods. During each research cycle, ETE performers will pre-register their predictions for their own experimental results, then conduct appropriate experiments with three separate representative populations. The T&E team is expected to assist performers with submitting “registered reports” as appropriate/feasible, as well as identifying and coordinating subject matter experts (SMEs) to provide baseline predictions against which performers’ accuracy can be further compared. These SMEs may include, but need not be limited to, researchers or recognized experts from different communities, organizations, or disciplines.

After each research cycle, ETE performers will extend their models to incorporate results and provide their data, analyses, and experimental methods to other teams to ensure reproducibility of their results and conclusions. Each ETE performer will attempt to reproduce all other ETE performers’ results using the data and tools provided to them. In the case of reproducibility difficulties, the T&E team may serve as a third-party evaluator. ETE performers will then develop new predictions and replicate their own experiment with three new populations. At this time, DARPA does not anticipate asking ETE teams to replicate another performer’s experimental protocol.

D. Schedule and Milestones

Table 2 describes the anticipated NGS2 schedule including major program milestones and anticipated deliverables. Due to the tight timeframes and coordination required by the program, proposals should include a detailed schedule that incorporates these deliverables/milestones and is consistent with the technical maturity and development status of the proposed approach.

All proposers should specify their research and development schedules for the full period of performance, including both the base Phase 1 (24 months) and the option Phase 2 (18 months).

Schedules will be synchronized across performers, as required, and monitored/revised as necessary throughout the program. Decisions on which performers (or whether the overall program) may continue into Phase 2 will be informed by demonstrable success in achieving the milestones in Figure 1 and Table 2. A target start date of October 2016 may be assumed for planning purposes.

Table 2: Detailed Schedule and Milestones Months after Award

Event Description

Phase 1 Research Cycle 1 (C1)

1 Program Kickoff ETE teams present technical approach and work plan Enablers present concept and work plan T&E team provides test and evaluation plan

3 Phase 1 IRB Performers with Human Subjects Research (HSR) must have submitted protocols for DoD Secondary Review

4 Site Visits ETE teams: Review progress towards experimental design, pre-registration Enablers: Preliminary Design Review T&E team: Prediction platforms and pre-registration review

DARPA-BAA-16-32/NGS2 BAA 15

Months after

Award Event Description

6 C1 Preregistration

ALL IRB APPROVALS OBTAINED

Approved IRB documentation provided to DARPA and T&E team Protocol manuscripts provided to T&E team for registered reports submissions to peer-reviewed journals ETE teams must have pre-registered study, initial predictions

9 Site Visits All teams: review technical progress

14 C1 Results Reproduced ETE performers evaluate reproducibility of other ETE performer results ETE teams provide reproducibility results to the T&E team for evaluation, adjudication

15 PI Meeting: C1 Review

ETE performers present review of C1 experimental results, updates to experimental design, processes, Enabler integration plan, and preliminary C2 preregistered hypotheses

T&E team presents results of C1 predictions and feedback on C2 plans Research Cycle 2 (C2)

16 C2 Design Update Initial C2 predictions registered Protocol manuscripts provided to T&E team for registered reports submissions to peer-reviewed journals

18 Site Visits and Enabler Demonstrations

Enablers demonstrate technology capabilities to the T&E team and

DARPA

23 C2 Results Reproduced ETE performers evaluate reproducibility of other ETE performer results ETE teams provide reproducibility results to the T&E team for evaluation, adjudication

24 PI Meeting: Phase 1 (C1 and C2) Review

ETE teams: present review of Phase 1 experimental results ETE teams: present updates to experimental design, processes, Enabler integration plan, and preliminary Phase 2 preregistered hypotheses Enablers: All Enabler tools integrated, present Phase 2 milestones and support as detailed and justified by performer in proposal T&E team: present results of C2 predictions and updated T&E plans for

Phase 2 Phase 2 (Option Period) Research Cycle 3 (C3)

28 Phase 2 IRB

ALL NEW IRB APPROVALS OBTAINED (if required) Approved IRB documentation provided to DARPA and the T&E team ETE performers submit protocols to the T&E team for registered reports, and to other ETE performers to allow them to make initial predictions for ALL experiments

Initial C3 predictions registered 30 Site Visits All teams: review technical progress

34 C3 Results Reproduced ETE performers evaluate reproducibility of other ETE performer results ETE teams provide reproducibility results to the T&E team for evaluation, adjudication

35 PI Meeting: C3 Review Performers (including T&E) present review of C3 experimental results to DARPA and transition partners

Research Cycle 4 (C4)

36 C4 Design Update ETE teams: present updates to experimental design, processes for other

ETE performers, and make predictions for ALL experiments T&E team: presents updates to T&E process

39 Site Visits All teams: review technical progress

DARPA-BAA-16-32/NGS2 BAA 16

Months after

Award Event Description

41 C4 Results Reproduced ETE performers evaluate reproducibility of other ETE performer results ETE teams provide reproducibility results to the T&E team for evaluation, adjudication

PI Meeting: Phase 2 Review and Program

Closeout

Performers (including T&E) present review of Cycle 4 experimental results o DARPA and transition partners

1. Meetings and Travel

Regular teleconference meetings will be scheduled with the Government team for progress reporting as well as problem identification and mitigation. Proposers should also anticipate at least two site visits per phase by the Government evaluation team, during which performers will have the opportunity to demonstrate progress toward agreed-upon milestones. In addition, include travel estimates for five two-day NGS2 review meetings over the course of 42 months: three meetings in the Washington, D.C. area and two meetings in the San Francisco, CA area.

2. Deliverables

Performers are expected to provide at a minimum the following deliverables:

Comprehensive quarterly technical reports due within ten days of the end of the given quarter, describing progress made on the specific milestones as laid out in the SOW.

A phase completion report submitted within 30 days of the end of each phase, summarizing the research done.

Other negotiated deliverables specific to the objectives of the individual efforts. These may include registered reports, experimental protocols, publications, intermediate and final versions of software libraries, code, and APIs, including documentation and user manuals, and/or a comprehensive assemblage of design documents, models, modeling data and results, and model validation data.

Reporting as outlined in Section VI.C.

II. Award Information A. Awards

Multiple awards are anticipated. The level of funding for individual awards made under this solicitation has not been predetermined and will depend on the quality of the proposals received and the availability of funds. Awards under this solicitation will be made to proposers whose proposals are determined to be the most advantageous and provide best value to the Government, all factors considered, including the potential contributions of the proposed work, overall funding strategy, and availability of funding. See Section V for further information.

The Government reserves the right to:

select for negotiation all, some, one, or none of the proposals received in response to this solicitation;

DARPA-BAA-16-32/NGS2 BAA 17

make awards without discussions with proposers;

conduct discussions with proposers if it is later determined to be necessary;

segregate portions of resulting awards into pre-priced options;

accept proposals in their entirety or to select only portions of proposals for award;

fund proposals in increments with options for continued work at the end of one or more phases;

request additional documentation once the award instrument has been determined (e.g., representations and certifications); and remove proposers from award consideration should the parties fail to reach agreement on award terms within a reasonable time or the proposer fails to provide requested additional information in a timely manner.

Proposals identified for negotiation may result in a procurement contract, grant, cooperative agreement, or other transaction (OT), depending upon the nature of the work proposed, the required degree of interaction between parties, and other factors.

In all cases, the Government contracting officer shall have sole discretion to select award instrument type and to negotiate all instrument terms and conditions with selectees. Proposers are advised that if they propose grants or cooperative agreements, DARPA may select other award instruments, as it deems appropriate. DARPA will apply publication or other restrictions, as necessary, if it determines that the research resulting from the proposed effort will present a high likelihood of disclosing performance characteristics of military systems or manufacturing technologies that are unique and critical to defense. Any award resulting from such a determination will include a requirement for DARPA permission before publishing any information or results on the program. For more information on publication restrictions, see the section below on Fundamental Research.

B. Fundamental Research It is DoD policy that the publication of products of fundamental research will remain unrestricted to the maximum extent possible. National Security Decision Directive (NSDD) 189 established the national policy for controlling the flow of scientific, technical, and engineering information produced in federally funded fundamental research at colleges, universities, and laboratories. The Directive defines fundamental research as follows:

'Fundamental research' means basic and applied research in science and engineering, the results of which ordinarily are published and shared broadly within the scientific community, as distinguished from proprietary research and from industrial development, design, production, and product utilization, the results of which ordinarily are restricted for proprietary or national security reasons.

As of the date of publication of this BAA, the Government expects that program goals as described herein may be met by proposers intending to perform fundamental research. The Government does not anticipate applying publication restrictions of any kind to individual awards for fundamental research that may result from this BAA. Notwithstanding this statement of expectation, the Government is not prohibited from considering and selecting research proposals

DARPA-BAA-16-32/NGS2 BAA 18

that, while perhaps not qualifying as fundamental research under the foregoing definition, still meet the BAA criteria for submissions. If proposals are selected for award that offer other than a fundamental research solution, the Government will either work with the proposer to modify the proposed statement of work to bring the research back into line with fundamental research or else the proposer will agree to restrictions in order to receive an award.

Proposers should indicate in their proposal whether they believe the scope of the research included in their proposal is fundamental or not. While proposers should clearly explain the intended results of their research, the Government shall have sole discretion to select award instrument type and to negotiate all instrument terms and conditions with selectees. Appropriate clauses will be included in resultant awards for non-fundamental research to prescribe publication requirements and other restrictions, as appropriate.

For certain research projects, it may be possible that although the research being performed by the prime contractor is restricted research, a subawardee may be conducting fundamental research. In those cases, it is the prime contractor’s responsibility to explain in its proposal why its subawardee’s effort is fundamental research.

The following statement or similar provision will be incorporated into any resultant non-fundamental research procurement contract or other transaction:

There shall be no dissemination or publication, except within and between the contractor and any subawardees, of information developed under this contract or contained in the reports to be furnished pursuant to this contract without prior written approval of DARPA’s Public Release Center (DARPA/PRC). All technical reports will be given proper review by appropriate authority to determine which Distribution Statement is to be applied prior to the initial distribution of these reports by the contractor. With regard to subawardee proposals for Fundamental Research, papers resulting from unclassified fundamental research are exempt from prepublication controls and this review requirement, pursuant to DoD Instruction 5230.27 dated October 6, 1987.

When submitting material for written approval for open publication, the contractor/awardee must submit a request for public release to the DARPA/PRC and include the following information: (1) Document Information: document title, document author, short plain-language description of technology discussed in the material (approx.

30 words), number of pages (or minutes of video) and document type (e.g., briefing, report, abstract, article, or paper); (2) Event Information: event type (conference, principal investigator meeting, article or paper), event date, desired date for DARPA's approval; (3) DARPA Sponsor: DARPA Program Manager, DARPA office, and contract number; and

(4) Contractor/Awardee's Information: POC name, email and phone. Allow four weeks for processing; due dates under four weeks require a justification. Unusual electronic file formats may require additional processing time. Requests may be sent either via email to public_release_center@darpa.mil or by mail at 675 North Randolph Street, Arlington VA 22203-2114, telephone (571) 218-4235. Refer to the following for link for information about DARPA’s public release process: http://www.darpa.mil/work-with-us/contract-management/public-release.”

mailto:public_release_center@darpa.mil

DARPA-BAA-16-32/NGS2 BAA 19

III.Eligibility Information A. Eligible Applicants

All responsible sources capable of satisfying the Government's needs may submit a proposal that shall be considered by DARPA.

1. Federally Funded Research and Development Centers (FFRDCs) and Government Entities

Federally Funded Research and Development Centers (FFRDCs) and Government entities (e.g., Government/National laboratories, military educational institutions, etc.) are subject to applicable direct competition limitations and cannot propose to this BAA in any capacity unless they meet the following conditions: (1) FFRDCs must clearly demonstrate that the proposed work is not otherwise available from the private sector; and (2) FFRDCs must provide a letter on official letterhead from their sponsoring organization citing the specific authority establishing their eligibility to propose to Government solicitations and compete with industry, and their compliance with the associated FFRDC sponsor agreement’s terms and conditions. This information is required for FFRDCs proposing to be prime contractors or subawardees. Government entities must clearly demonstrate that the work is not otherwise available from the private sector and provide written documentation citing the specific statutory authority and contractual authority, if relevant, establishing their ability to propose to Government solicitations. At the present time, DARPA does not consider 15 U.S.C. § 3710a to be sufficient legal authority to show eligibility.

While 10 U.S.C.§ 2539b may be the appropriate statutory starting point for some entities, specific supporting regulatory guidance, together with evidence of agency approval, will still be required to fully establish eligibility. DARPA will consider FFRDC and Government entity eligibility submissions on a case-by-case basis; however, the burden to prove eligibility for all team members rests solely with the proposer.

2. Foreign Participation Non-U.S. organizations and/or individuals may participate to the extent that such participants comply with any necessary nondisclosure agreements, security regulations, export control laws, and other governing statutes applicable under the circumstances.

B. Procurement Integrity, Standards of Conduct, Ethical Considerations, and Organizational Conflicts of Interest

Current federal employees are prohibited from participating in particular matters involving conflicting financial, employment, and representational interests (18 U.S.C. §§ 203, 205, and 208).

Once the proposals have been received, and prior to the start of proposal evaluations, the Government will assess potential conflicts of interest and will promptly notify the proposer if any appear to exist. The Government assessment does NOT affect, offset, or mitigate the proposer’s responsibility to give full notice and planned mitigation for all potential organizational conflicts, as discussed below.

DARPA-BAA-16-32/NGS2 BAA 20

Without prior approval or a waiver from the DARPA Director, in accordance with FAR 9.503, a contractor cannot simultaneously provide scientific, engineering, technical assistance (SETA) or similar support and also be a technical performer. As part of the proposal submission, all members of the proposed team (prime proposers, proposed subawardees, and consultants) must affirm whether they (their organizations and individual team members) are providing SETA or similar support to any DARPA technical office(s) through an active contract or subcontract. All affirmations must state which office(s) the proposer, subawardees, consultant, or individual supports and identify the prime contract number(s). All facts relevant to the existence or potential existence of organizational conflicts of interest (FAR 9.5) must be disclosed. The disclosure must include a description of the action the proposer has taken or proposes to take to avoid, neutralize, or mitigate such conflict. If in the sole opinion of the Government after full consideration of the circumstances, a proposal fails to fully disclose potential conflicts of interest and/or any identified conflict situation cannot be effectively mitigated, the proposal will be rejected without technical evaluation and withdrawn from further consideration for award.

If a prospective proposer believes a conflict of interest exists or may exist (whether organizational or otherwise) or has questions on what constitutes a conflict of interest, the proposer should send his/her contact information and a summary of the potential conflict via email to the BAA email address before time and effort are expended in preparing a proposal and mitigation plan.

C. Cost Sharing/Matching Cost sharing is not required; however, it will be carefully considered where there is an applicable statutory condition relating to the selected funding instrument (e.g., OTs under the authority of 10 U.S.C. § 2371).

D. Other Eligibility Requirements Proposers may submit multiple proposals in any of three categories described in Section I.B.1.

However, any proposer participating on a T&E proposal—whether as a prime, subcontractor, or in any other capacity from an organizational to individual level—may not propose to, or with a team that is proposing to, either of the two other categories (i.e., ETE team or Enabler). This is to avoid OCI situations between development and testing and to ensure objective evaluation results.

IV. Application and Submission Information Prior are strongly encouraged to submit an abstract in advance of a full proposal to minimize effort and reduce the potential expense of preparing an out of scope proposal. This process allows a proposer to ascertain whether the proposed concept is: (1) applicable to the NGS2 BAA and (2) currently of interest. For the purposes of this BAA, applicability is defined as follows:

The proposed concept is applicable to the technical areas described…