HR001120S0035.pdf

HR001120S0035 HABITUS 1

Broad Agency Announcement Habitus

Defense Sciences Office

HR001120S0035

February 17, 2020

HR001120S0035 HABITUS 2

Table of Contents I. Funding Opportunity Description

A. Introduction B. Background C. Program Description/Scope D. Program Structure E. Technical Area Descriptions F. Additional Parameters G. Schedule/Milestones H. Deliverables I. Other Program Objectives and Considerations

II. Award Information A. General Award Information B. Fundamental Research

III. Eligibility Information A. Eligible Applicants B. Organizational Conflicts of Interest C. Cost Sharing/Matching D. Other Eligibility Requirements

IV. Application and Submission Information A. Address to Request Application Package B. Content and Form of Application Submission C. Submission Dates and Times D. Funding Restrictions E. Other Submission Requirements

V. Application Review Information A. Evaluation Criteria B. Review and Selection Process C. Federal Awardee Performance and Integrity Information (FAPIIS)

VI. Award Administration Information A. Selection Notices B. Administrative and National Policy Requirements C. Reporting

VII. Agency Contacts VIII. Other Information

A. Frequently Asked Questions (FAQs) B. Proposers Day

BAA Attachments:

Attachment A: ABSTRACT SUMMARY SLIDE TEMPLATE Attachment B: ABSTRACT TEMPLATE Attachment C: PROPOSAL SUMMARY SLIDE TEMPLATE Attachment D: PROPOSAL TEMPLATE VOLUME 1 TECHNICAL & MANAGEMENT VOLUME Attachment E: PROPOSAL TEMPLATE VOLUME 2 COST VOLUME Attachment F: COST PROPOSAL TEMPLATE Attachment G: PROPOSAL TEMPLATE VOLUME 3 ADMINISTRATIVE & NATIONAL POLICY REQUIREMENTS VOLUME

HR001120S0035 HABITUS 3

PART I: OVERVIEW INFORMATION

Federal Agency Name: Defense Advanced Research Projects Agency (DARPA), Defense Sciences Office (DSO)

Funding Opportunity Title: Habitus

Announcement Type: Initial Announcement

Funding Opportunity Number: HR001120S0035

Catalog of Federal Domestic Assistance (CFDA) Number(s): 12.910 Research and Technology Development

Dates (All times listed herein are Eastern Time.)

o Posting Date: February 17, 2020 o Proposers Day: February 10, 2020. See Section VIII.B.

o TA1/ TA2 Abstract Due Date: February 28, 2020, 4:00 p.m.

o FAQ Submission Deadline for TA1/TA2: March 23, 2020, 4:00 p.m. See Section

VIII.A.

o FAQ Submission Deadline for CE: TBD o TA1/TA2 Full Proposal Due Date: April 23, 2020, 4:00 p.m.

o CE Full Proposal Due Date: TBD o BAA closing date: August 17, 2020

Anticipated Individual Awards: DARPA anticipates multiple awards.

Types of Instruments that May be Awarded: Procurement contracts, cooperative agreements, or other transactions

Agency contacts o Technical POC: Dr. Bart Russell, Program Manager, DARPA/DSO o BAA Email: Habitus@darpa.mil o BAA Mailing Address:

DARPA/DSO

ATTN: HR001120S0035

675 North Randolph Street Arlington, VA 22203-2114

DARPA/DSO Opportunities Website: http://www.darpa.mil/work-with-us/opportunities Frequently Asked Questions (FAQ): FAQs for this solicitation may be viewed on the

DARPA/DSO Opportunities Website. See Section VIII.A for further information.

mailto:Habitus@darpa.mil http://www.darpa.mil/work-with-us/opportunities?tFilter=&oFilter=2&sort=name

HR001120S0035 HABITUS 4

PART II: FULL TEXT OF ANNOUNCEMENT

I. Funding Opportunity Description

This Broad Agency Announcement (BAA) constitutes a public notice of a competitive funding opportunity as described in Federal Acquisition Regulation (FAR) 6.102(d)(2) and 35.016 as well as 2 C.F.R. § 200.203. Any resultant negotiations and/or awards will follow all laws and regulations applicable to the specific award instrument(s) available under this BAA, e.g., FAR

15.4 for procurement contracts.

A. Introduction The Defense Sciences Office (DSO) at the Defense Advanced Research Projects Agency (DARPA) is soliciting innovative research proposals to create self-sustaining, adaptive, generalizable, and scalable methods for generating causal system models based on local knowledge to aid operational decision making.

Understanding how to work with and influence local systems1 to support stability operations is critical for operational decision making and is most challenging in undergoverned regions in which the systems themselves are often in flux or illegible.2 Establishing stability in such regions requires we facilitate actions that are in line with local views, yet our current means for understanding local systems such as the political, socioeconomic, and/or those related to health and infrastructure are limited. Humans develop causal cognitive representations – or cognitive models – of systems of which they are a part. These models include factors (or variables), relationships among factors, and contexts that affect both. The knowledge behind these models is often hyper-localized, changing dramatically with regional and/or population dependent interactions of factors such as terrain, industries, population density (urban, rural), shared history, formal and informal power structures, religion, and ethnicity. These cognitive models, though often implicit, allow one to estimate which factors are most important for a given outcome and how those factors interact to anticipate future outcomes based on history, current events, and trends.

Unfortunately, the cognitive models locals have developed are largely inaccessible to outsiders and operational decision makers. This has led to operational challenges. Examples include:

The failure to recognize the interaction of local burial practices with Ebola’s virulence in West Africa led to a vast underprojection of the disease’s spread in 2014. Efforts to treat it as a health and education issue misunderstood the local importance of these rituals;

solutions that included local religious leadership were far more effective than education-

1 By “systems” for the purposes of this BAA we mean an interconnecting network of factors that affect or contribute to a real-world outcome. An agricultural system might include, for instance, natural factors (e.g., rainfall, temperatures) and human factors (e.g., irrigation methods, planting practices, and formal or informal regulations, and even perceptions of events) that affect the outcome of seasonal agricultural yields and/or crop prices.

2 We use “illegibility” here as James C. Scott has developed the concept to refer to ways in which states understand those they govern. Scott, J. C. (1998). Seeing like a state: How certain schemes to improve the human condition have failed. Yale University Press. Full text: https://libcom.org/files/Seeing%20Like%20a%20State%20- %20James%20C.%20Scott.pdf https://libcom.org/files/Seeing%20Like%20a%20State%20-%20James%20C.%20Scott.pdf https://libcom.org/files/Seeing%20Like%20a%20State%20-%20James%20C.%20Scott.pdf

HR001120S0035 HABITUS 5

only approaches.3 The US’s early Information Operations campaign in Iraq was largely ineffective because

Iraqis had learned to distrust traditional media outlets (radio, television) after years of listening to Ba’athist propaganda.4

Efforts to secure a small town from ISIS created local turmoil when US forces set up a perimeter that excluded the local glass factory and disrupted locals’ ability to get to work.

Such cultural factors, when studied and described in a vacuum, provide little quantitative or predictive power for understanding real-world outcomes. Instead, Habitus is focused on creating explicit computational models of complex, real-world local systems based on collective cognitive models of local populations. These computational models will include all relevant factors that contribute to predicting a system outcome, whether concrete (e.g., employment rates, rain totals) or traditionally unquantifiable (e.g., religious beliefs, historical experience). Systems of interest include those relevant to stability operations - whether related to security, socioeconomics, politics, and/or those involved with health and wellbeing.

The program will make the computational models available, accessible, and understandable to operators, providing them with an “insider” view to support operational decision making. The resulting capability will be specific enough to anticipate system-level effects in response to events that are generalizable across regions and populations, adaptive as societies change over time, and self-sustaining for maintainability and persistence.

B. Background Through lived experience, human cognition reduces the dimensionality of complex systems and derives causality among the most relevant factors, relationships among factors, and contexts that affect both to create an implicit cognitive model. These cognitive models form the basis of experience and “insider” knowledge. These cognitive models allow one to estimate which factors are most important for a given outcome and how those factors interact to anticipate future outcomes based on history, current events, and trends.

Unfortunately, the cognitive models locals have developed are mostly implicit and largely inaccessible to outsiders, and thus, cannot inform operational decision making. Most current approaches collect outputs of cognitive models among populations (e.g., polling, big data methods) but make very limited attempts to extract the causal reasoning by which locals derive those outputs. In many cases, existing tools were developed to support strategic forecasting and analysis rather than to provide the detailed understanding necessary to make projections and anticipate how a system and its factors will change over time. While Subject Matter Experts (SMEs) spend years developing similar cognitive models about a given region or domain topic,5 rarely are there enough SMEs available to guide operational decision making, and often their

3 Maxmen, A. (2015). How the fight against Ebola tested a culture’s traditions. National Geographic, 30.

4 McFate, M. (2005). The military utility of understanding adversary culture. Office of Naval Research, Arlington

VA.

5 By “domain topic” we mean a particular type of system that exists in multiple regions globally. Examples include:

agricultural practices, food insecurity, microfinance, infectious disease spread, informal governance structures, and others.

HR001120S0035 HABITUS 6

knowledge is not specific enough to understand the particularities of a given town or municipality. Likewise, big data methods often lack the ability to interpret data through a local lens and are not specific enough to a given Area of Operations (AO) to inform decision making.

Exceptions that are highly tuned to a single city do not generalize to other areas or populations.

Polling remains a common state of practice because the methods generalize widely; however, polling is not adaptive, and a poll must be completely rerun to account for changing views as a result of emerging events, which requires considerable labor.

Needed is a capability that is simultaneously:

Adaptive, so system representations and projections update to maintain accuracy as non-ergodic human systems change over time;

Scalable & generalizable, so that the methodology readily transfers to new regions, populations, and topics;

Self-sustaining, meaning it requires few resources to maintain over time; and Specific and detailed enough to provide causal insight for a given region, system, and/or population.

While any one perspective – and any one individual’s cognitive model – is unlikely to be complete and accurate for predicting regional system outcomes, recent advances in participatory science and modeling have demonstrated that building collective models based on the knowledge of populations who live, work, and interact with the system can provide more accurate estimations of future outcomes of that system than experts and other state of the art methods.6 Broader indications that we can develop such a capability come from trends in ecological research, where the accuracy and efficacy of assessments and interventions have improved by involving locals not just in outcome estimation but also in knowledge discovery (i.e., identifying relevant factors) and model construction (i.e., establishing relationships among factors and the effect of contexts).7,8 The level of agreement among respondents characterizes the collective uncertainty about the importance of each factor in the computational model, providing an estimate of error and community divergence. Though promising, these methods are limited in their ability to scale in a manner that makes them efficacious for operational needs.

Habitus will develop a method that, via engagement with locals, will construct computational models of the collective local view of political, economic, and/or social systems so decision makers can understand not only what factors are important but also why and how these systems work from an insider’s view (Figure 1). Developing such computational models will not only provide operators with the ability to anticipate how the system will change as events unfold, but also specific insight as to how to affect the system to achieve desired results.

6 Halbrendt, J., Gray, S. A., Crow, S., Radovich, T., Kimura, A. H., & Tamang, B. B. (2014). Differences in farmer and expert beliefs and the perceived impacts of conservation agriculture. Global Environmental Change, 28, 50-62.

7 Bélisle, A. C., Asselin, H., LeBlanc, P., & Gauthier, S. (2018). Local knowledge in ecological modeling. Ecology and Society, 23(2).

HR001120S0035 HABITUS 7

Developing an operational capability from these advances requires that we 1) develop automated methods for identifying factors and their relative importance, relationships among factors, and contexts that affect those relationships to create specific computational models and 2) create a self-sustaining engagement mechanism to elicit knowledge such that a model can adapt or maintain accuracy over time and as systems change. The goal is not to create a single computational model for all regions, but rather to develop a generalizable methodology that is scalable enough to create a customized computational model for each AO.

An example of Habitus’ potential impact is in support of a US Marine Air-Ground Task Force (MAGTF) deployment to East Asian regions with high levels of geographic, ethnic, and linguistic diversity. Arriving forces would deploy Habitus operationally in each new region to provide immediate (within a few days), coarse understanding of the most critical system factors, such as major players, industries, populations, and their locations and affiliations. Habitus would then develop a more robust, detailed, quantitative, and causal computational model of collective cognitive system model in a few weeks. The commander would query the computational model with potential actions and observe the predicted changes to the actions/attitudes of the populace.

C. Program Description/Scope The Habitus program will create a specific, generalizable/scalable, adaptive, and self-sustaining methodology for capturing and making local knowledge available to operational decision makers to support stability operations. This involves developing mechanisms of measurement and local engagement, identifying relevant factors, prioritizing and developing relationships among those factors, and testing and updating the model as systems change. Over the course of the program, performers will demonstrate, test, and validate the system by using the computational models generated by their systems to make predictions against real-world, externally verifiable outcomes.

Proposed research should investigate innovative approaches that enable revolutionary advances in science, methodologies, or systems to engage with and understand local populations to make their often implicit cognitive models of a local system explicit and available to operational

Figure 1. Habitus System Concept. Through engagement with local populations, Habitus creates an explicit computational model of the collective local cognitive model of a system in a (performer-specified) topic domain. A commander (decision maker) can view and query the computational model for assessments of system-level changes based on actual or hypothetical actions or events, providing the commander with an in-depth understanding of the system and its causal dynamics.

HR001120S0035 HABITUS 8

decision-makers. Specifically excluded is research that primarily results in evolutionary improvements to the existing state of practice.

D. Program Structure Habitus consists of two Technical Areas (TAs) and a Comparative Evaluation (CE) team in support of a government Testing and Evaluation (T&E) team.

TA1: Model Development TA2: Engagement Mechanism CE: Comparative Evaluation

This BAA comprises two proposal submission periods. The first submission period is for TAs 1 and 2. Because the engagement mechanisms will depend heavily on model development strategies, and vice versa, all proposals submitted during the first submission period must address both TA1 and TA2. The CE component is also being solicited under this BAA; however, additional technical detail regarding CE will be published in a subsequent BAA amendment. The amendment will also include requirements for submitting a proposal and due dates. The description of CE that follows is for informational purposes only in order to facilitate proposing to TA1/TA2. In order to ensure the impartiality of the evaluation of the TA1 and 2 technology, proposers selected for negotiation of an award for TA1/TA2 will not be eligible to propose to the CE BAA Amendment.

The program will consist of three Phases and a total of 42 months.

For TA1/TA2:

Phase 1 (Base period, 24 months) will focus on building the initial capabilities and demonstrating efficacy in one region. Performers will be expected to work directly (i.e., on site) with local populations to develop, test, and validate methods.

Phase 2 (Option 1, 12 months) will challenge performers to generalize the capabilities in two new regions/populations. Performers will demonstrate scalability of the method by sustaining performance in new regions with substantially less development time.

Phase 3 (Option 2, 6 months) will be an extended demonstration of the capability in a DARPA-specified region, which will be selected in collaboration with performers and interested transition partners. Discussions and decisions on DARPA specified region will occur early in Phase 2 to allow time for any necessary IRB modifications and subsequent HRPO approvals prior to the start of Phase 3.

TA1/TA2 Proposals should address all three Phases. For costing purposes, proposers should base the Phase 3 portion of their proposal on a location that is comparable (in terms of travel costs and infrastructure needs) to those proposed for Phases 1 and 2.

CE will begin roughly 12 months into Phase 1 and will run the duration of the rest of the program (30 months).

HR001120S0035 HABITUS 9

E. Technical Area Descriptions TA1: Model Development. TA1 will develop scalable and generalizable knowledge elicitation methods to semi-automatically and rapidly derive computational model factors and system level relationships (the system model). The methods should:

Be specific and quantifiable. The system model should quantify the relationships among factors and be able to generate quantifiable predictions based on changes in the system.

Capture implicit local knowledge. Elicitation strategies should account for the fact that people are often not explicitly aware of the sources or indicators of their knowledge. This implicit knowledge will need to be gathered if it is to be used in the model.

Capture unanticipated factors. The method should be able to elicit or discover factors the performer team or outside experts did not anticipate. This could involve identifying new terms or colloquialisms locals use to describe either the same factor outside experts use, or discovering new, previously unspecified factors. Performers are strongly encouraged to include methods that acknowledge and overcome challenges associated with education levels (e.g., literacy).

Incorporate local beliefs and/or practices. Local populations may have beliefs and/or practices that may not be proven by empirical scientific evidence but nevertheless influence behavior and system outcomes. Outsiders often miss these elements, which can lead to miscalculations and misunderstandings. Proposed methods should be able to discover and incorporate these beliefs and practices into computational models.

Represent dynamic systems. To achieve adaptivity, the computational model should be able to 1) update the relative importance of factors and their relationships to the predicted outcome and 2) be able to insert new factors, relationships, and contexts into the model as more information is gathered and as the system evolves.

Empower non-experts to anticipate real-world system outcomes. The computational model should allow the non-expert user (i.e., assume no special training in data science, behavioral or social science, or local languages) to adjust factors to explore the potential outcomes of given actions and provide understanding behind the projected outcome.

The proposed methods must exhibit generalizability across four subject populations. Proposers will demonstrate efficacy by testing their ability to predict the outcomes of real-world, local events. The government T&E team will evaluate efficacy against current practices by comparting TA1/TA2 results to those from the CE.

TA2: Engagement Mechanism. TA2 will create a self-sustaining capability that continually provides information necessary to build and update TA1’s computational models by providing value to the local users. The solution can be based on either passive observation or active engagement. Passive observations use natural behaviors that occur independent of Habitus, while active engagements are direct interactions with locals such as queries or probes. Each active engagement is considered a “touchpoint” with locals with respect to the Habitus metrics (Table 2). The solution should:

HR001120S0035 HABITUS 10

Be generalizable. Proposers should address how the capability will be effective in more than one region and why it will be appealing/useful to multiple populations.

Align incentives of Habitus with the local population to support accuracy. Methods that rely solely on monetary incentives encourage participation but not necessarily accuracy. Proposers should address how their engagement mechanism will provide value to locals in a way that aligns with existing motivations (e.g., intrinsic incentives) to encourage participation and accuracy in responses.

Identify and efficiently sample all relevant populations. Proposers should consider how to identify populations relevant to the domain topic they propose and how to engage with those populations (i.e., how they will avoid oversampling or harassing local populations).

The methods developed should identify the group to which the sampled people belong (including the possibility of multi-group membership). In addition, the methodology will need to address the challenges of obtaining representative samples from a small population.

Accommodate characteristics of the target population. Each local population will have characteristics that will need to be considered when creating an effective engagement mechanism. Examples of these characteristics are literacy rates, likely frequency of use, collection platform, communication equipment, infrastructure, and applications.9 The engagement mechanism must align with the local infrastructures of the subject populations.

Rather than constrain performers to specific technologies (e.g., dumb phones or Facebook), the methods developed should be able to access the subject populations both with their current infrastructure or platform and with those that will be available in the near future.

Demonstrate self-sustainment. In order to provide long-term transitionable value, the method will need to minimize impediments to self-sustainment. Such impediments include any recurring costs (e.g., labor) associated with creating computational models from collected data and ongoing payments for participation. Alternative incentives such as information exchange paradigms may be considered. To be self-sustaining, the mechanism should maximize the inherently limited opportunities for engagement. This is a particular problem for local populations where the total potential pool of respondents is limited, and the length of (intermittent) engagement is long. Proposers will need to describe their mechanisms for self-sustained use. We encourage performers to consider starting from existing points of departure (such as existing methods used by Non-Governmental Organizations (NGOs) or existing data collection platforms) to minimize development time and risk.

Passive engagement. If TA2 includes passive engagement methods, proposers must 1) address specifically how they will know/test that they are interpreting the data correctly within the local context, 2) specify if informed consent is required, and if so, 3) how they plan to obtain informed consent required for Independent Review Board (IRB) approval.

Active engagement. If TA2 includes active engagement methods, proposers must explain specifically how they will address interaction effects of the proposed engagement method.

9 For instance, currently dumb phones are far more prevalent than any kind of smartphone in Africa, however “dumb smartphones” are on the rise. Likewise, Facebook messenger and WhatsApp are far more popular than SMS in South America, but SMS is still the dominant messaging mechanism certain regions of Africa.

HR001120S0035 HABITUS 11

This may vary across and between cultures. An example of “interaction effects” are perspective based biases that can be introduced by the investigator. Obtaining informed consent in line with IRB and HRPO requirements will be required for all active engagement methods.

TA2 solutions can be specific to a topic domain (e.g. political, economic, health related, etc.)

although solutions that can be applied to multiple domains are preferred. Approaches to gathering data from populations lacking robust telecommunications infrastructure are of particular interest, but they will have to demonstrate that they can generate sufficient data necessary to support the needs of TA1.

Comparative Evaluation The Comparative Evaluation (CE) team will generate parallel predictions to those specified by TA1/TA2 using current methods in order to generate a question-for-question comparative data set that the government T&E team will use to evaluate performance. This is because DARPA expects TA1 and 2 teams to tackle domain topics that are important to stability and defense, rather than those that are easy to predict. Since the capabilities required of any CE team will depend on the populations, regions, and topics being investigated by the TA1 and 2 teams, proposals for CE teams are not being solicited at the present time. DARPA intends to release an amendment to this BAA with additional technical area details, potentially including information on the specific populations, regions, and topics from the TA1 and 2 proposals selected for negotiation of an award, abstract details (if applicable), and proposal requirements and due dates after TA1/TA2 proposers have been notified of DARPA’s selection decisions.

F. Additional Parameters Proposers to the Habitus program will need to develop computational models from, and demonstrate their engagement mechanisms among, populations and under circumstances that will demonstrate specificity, adaptivity, generalizability and self-sustainability. Therefore, proposers must describe how they plan to address the following challenges:

Real world event predictions. In order to enable validation, the proposed methods must generate predictions for externally verifiable real-world events. Proposers will need to specify the system that they will model and the events that will be predicted. The events targeted must have the following features:

o Ground truth must be externally verifiable (unemployment rates, agricultural yields, crop prices, infection rates, etc.);

o Events must occur with enough frequency so that there are sufficient instances of ground truth to evaluate progress on a quarterly basis.

Population size. In order to balance the desire to probe local behavior with the need for statistical power, the target populations must be between 50,000 and 500,000 people in at least three distinct populations in regions Outside the Continental United States (OCONUS). Performers should seek populations that have significant heterogeneity, especially in the latter phases of the program.

HR001120S0035 HABITUS 12

Work with non-WEIRD populations. In order to demonstrate the capability is widely generalizable, proposers should to work with populations that are not from Western Educated Industrialized Rich and Democratic (WEIRD)10 regions. Regions selected may exhibit one or two of the WEIRD features. Working with college undergraduate participants in the United States is out of scope. Initial work in largely unacculturated immigrant or isolated populations in the Continental United States (CONUS) is acceptable, but the capability must be demonstrated within populations and regions OCONUS over the course of the program.

Language. If the proposed methods will use linguistic data or language-based interactions, performers must do so in at least one language other than English. Proposers are encouraged to work in local dialects when possible. No new Natural Language Processing (NLP) libraries will be created under this program11 therefore performers must either work in languages for which translation capability exists or use non-linguistic methods.

OCONUS human subjects research. Performers are strongly encouraged to work with local partners, which may include universities or research institutions, to facilitate in-country human subjects research (HSR) approval and ensure research adheres to local HSR requirements.12 If performers choose not to partner locally, they must demonstrate experience obtaining HSR approval in the proposed region.

Submissions must propose a topic domain of interest and geographically distinct populations to be studied and demonstrate existing relationships to facilitate work in those regions. The topic domain that a performer team chooses may include any area related to stability operations, such as supporting economic stability, anticipating and offsetting food insecurity, improving disease prevention, or another defense-related issue. The specific populations and the topics addressed are up to the proposer to define, as long as they meet the parameters above.

Out of scope are solutions that rely solely on social media and/or Mechanical Turk.

G. Schedule/Milestones The Habitus performers will be evaluated using a number of milestones and metrics enumerated below. However, attainment of the milestones and metrics for a given phase does not guarantee transition into the next phase of the program. Individual efforts will also be judged on their expected ability to attain subsequent milestones. Options may be exercised, at the Government’s sole discretion, based on technical progress measured against the metrics and milestones defined in the BAA as well as funding availability.

10 Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world?. Behavioral and brain sciences, 33(2-3), 61-83.

11 It is acceptable to add new colloquialisms to extant libraries as encountered/discovered.

12 In addition to adhering to US and HRPO standards HSR, all protocols must adhere to the HSR requirements of the nation in which the work will occur. See the Health and Human Services website for guidance.

https://www.hhs.gov/ohrp/international/compilation-human-research-standards/index.html Partnering with a local research institution that is already familiar with their national requirements and approval processes will facilitate protocol adherence to local, US, and HRPO requirements.

HR001120S0035 HABITUS 13

Table 1: Habitus program milestone schedule

DARPA will assess TA1 and TA2 components separately in Phase 1. This means teams can develop TA1 using approximated methods for TA2 function (e.g., mimicking the nature and frequency of engagement) and can test TA2 methods for usability and traction among locals without the TA1 modeling capability being fully implemented. In Phases 2 and 3, DARPA will evaluate performance of the integrated TA1/TA2 capability against metrics.

Notable milestones:

HSR protocol and approval timeline. The schedule allows for 12 months to obtain IRB and Human Research Protection Office (HRPO) approvals. However, this includes time for delays, so teams are highly encouraged to complete their protocol, obtain IRB determination and approval, and submit the protocol to HRPO review by month 6 or earlier. Approval delays beyond 12 months will significantly harm the performer’s ability to move beyond Phase 1 in the program. Teams are encouraged to begin work with local populations as soon as they receive HRPO approval (i.e., prior to month 12).

Evidence TA2 can support TA1 needs at month 12. Performers will be required to demonstrate that the engagement methods employed in TA2 will provide enough information to support TA1 data needs. Likewise, TA1 must present evidence to quantify the amount of data they will need from TA2. Though a data-backed demonstration of this is not required until month 12, proposers should include projections or estimations of each in their proposals.

HR001120S0035 HABITUS 14

Predicting predictions by locals. Just as important as anticipating real-world outcomes is the ability to understand how the local population will respond to, or view, event outcomes. If the TA1 computational models are successful in providing an insider view, they should be able to anticipate and make predictions of what the local population will predict will happen as a result of a hypothetical event. Because this level of prediction does not rely on the event occurring and does not use external sources to provide ground truth, this evaluation can be performed earlier in the program and more frequently. Therefore, DARPA will first evaluate this prediction performance against the accuracy metric at month 18.

Table 2 Habitus program metrics

Brier Skill Score for Accuracy: To evaluate accuracy DARPA will use a variation of the Brier Skill Score (BSS), which compares the relative accuracy of the performer’s TA1 method against that of the State of the Art (SoA) methods the CE team will use. DARPA is using this comparative method rather than providing an absolute accuracy threshold to encourage performers to choose challenging topics and events for which there may be no current reliable methods for assessing the local view (the system outcome must still have externally verifiable ground truth, even if current methods are poor at predicting those outcomes). Note that a BSS of > 0 means TA1 is performing better than the CE team. Performers will need to establish this level of performance by the end of Phase 1 and demonstrate that they can maintain that performance in new regions and in shorter amounts of time across Phases 2 and 3.

Touchpoint Ratio for Efficiency: TA2 engagement methods will not be self-sustaining if they continually pester users/populations to provide information. DARPA will evaluate efficiency by comparing the number of “touchpoints” or number of times the engagement mechanism must request input from the participant – against the number of complete data points collected. All attempts to contact locals to participate (not just interactions with those who consent), and the process of obtaining consent as part of IRB compliance will count against the touchpoint ratio.

Other proposal properties

HR001120S0035 HABITUS 15

Proposers should provide a technical and programmatic strategy that conforms to the entire program schedule and presents an aggressive plan to fully address all program goals, metrics, milestones and deliverables.

The task structure must be consistent across the proposed schedule, Statement of Work, and cost volume.

A target start date of 1 October 2020 may be assumed for planning purposes.

Schedules will be synchronized across performers, as required, and monitored/revised as necessary throughout the program.

All proposals must include the following meetings and travel in the proposed schedule and costs:

o To continue integration and development between TAs, foster collaboration between teams, and disseminate program developments, a two-day Principal Investigator (PI) meeting will be held approximately every six months, with locations split between the East and West Coasts of the United States. For budgeting purposes, plan for eight two-day meetings over the course of 42 months: five meetings in the Washington, D.C. area and three meetings in the San Francisco, CA area.

o Regular teleconference meetings will be scheduled with the Government team for progress reporting as well as problems identification and mitigation.

Proposers should also anticipate at least one site visit per phase by the DARPA Program Manager during which they will have the opportunity to demonstrate progress towards agreed-upon milestones.

H. Deliverables Performers will be expected to provide at a minimum the following deliverables:

Comprehensive quarterly technical reports due within ten days of the end of the given quarter, describing progress made on the specific milestones as laid out in the SOW.

Items specified in the Program Milestone Schedule (Table 1).

Data and code necessary for the T&E team to evaluate performer progress against accuracy at least 90 days before the end of Phases 1 and 2.

A phase completion report submitted within 30 days of the end of each phase, summarizing the research done.

Other negotiated deliverables specific to the objectives of the individual efforts. These may include registered reports; experimental protocols; publications; data management plan; intermediate and final versions of software libraries, code, and Application Programming Interfaces (APIs), including documentation and user manuals and/or a comprehensive assemblage of design documents, models, modeling data and results; and model validation data.

Reporting as outlined in Section VI.C.

I. Other Program Objectives and Considerations

1. Collaboration

TA 1, 2 and the CE Team will be required to work with the government T&E team who will evaluate performance against the metrics outlined in Table 2. TA1/TA2 performers will – at a

HR001120S0035 HABITUS 16

minimum – be required to provide the government T&E team with their predictions prior to the specified event’s outcome. Collaboration may also include sharing all raw data, code for reproducing results, and output results.

II. Award Information A. General Award Information

DARPA anticipates multiple awards for combined TA1/TA2 efforts and a single award for the CE team (subject of a future amendment to this BAA).

The level of funding for individual awards made under this BAA will depend on the quality of the proposals received and the availability of funds. Proposals that are determined selectable will not necessarily receive awards. Awards will be made to proposers13 whose proposals are determined to be the most advantageous to the Government, all evaluation factors considered.

See Section V for further information.

The Government reserves the right to:

select for negotiation all, some, one, or none of the proposals received in response to this solicitation;

make awards without discussions with proposers;

conduct discussions with proposers if it is later determined to be necessary;

segregate portions of resulting awards into pre-priced options;

accept proposals in their entirety or to select only portions of proposals for award;

fund awards in increments with options for continued work at the end of one or more phases;

request additional documentation once the award instrument has been determined (e.g., representations and certifications); and remove proposers from award consideration should the parties fail to reach agreement on award terms within a reasonable time or the proposer fails to provide requested additional information in a timely manner.

Proposals identified for negotiation may result in a procurement contract, cooperative agreement, or other transaction (OT), depending upon the nature of the work proposed, the required degree of interaction between parties, and other factors.

Proposers looking for innovative, commercial-like contractual arrangements are encouraged to consider requesting Other Transactions. To understand the flexibility and options associated with

13 As used throughout this BAA, “proposer” refers to the lead organization on a submission to this BAA. The proposer is responsible for ensuring that all information required by a BAA--from all team members--is submitted in accordance with the BAA. “Awardee” refers to anyone who might receive a prime award from the Government, including recipients of procurement contracts, cooperative agreements, or Other Transactions. “Subawardee” refers to anyone who might receive a subaward from a prime awardee (e.g., subawardee, consultant, etc.).

HR001120S0035 HABITUS 17

Other Transactions, consult http://www.darpa.mil/work-with-us/contract-management#OtherTransactions.

In accordance with 10 U.S.C. § 2371b(f), the Government may award a follow-on production contract or Other Transaction (OT) for any OT awarded under this BAA if: (1) that participant in the OT, or a recognized successor in interest to the OT, successfully completed the entire prototype project provided for in the OT, as modified; and (2) the OT provides for the award of a follow-on production contract or OT to the participant, or a recognized successor in interest to the OT.

In all cases, the Government contracting officer shall have sole discretion to select award instrument type, regardless of instrument type proposed, and to negotiate all instrument terms and conditions with selectees. DARPA will apply publication or other restrictions, as necessary, if it determines that the research resulting from the proposed effort will present a high likelihood of disclosing performance characteristics of military systems or manufacturing technologies that are unique and critical to defense. Any award resulting from such a determination will include a requirement for DARPA permission before publishing any information or results on the program. For more information on publication restrictions, see the section below on Fundamental Research.

B. Fundamental Research It is DoD policy that the publication of products of fundamental research will remain unrestricted to the maximum extent possible. National Security Decision Directive (NSDD) 189 defines fundamental research as follows:

‘Fundamental research’ means basic and applied research in science and engineering, the results of which ordinarily are published and shared broadly within the scientific community, as distinguished from proprietary research and from industrial development, design, production, and product utilization, the results of which ordinarily are restricted for proprietary or national security reasons.

As of the date of publication of this BAA, the Government expects that TA1 program goals as described herein may be met by proposed efforts for fundamental research, while TA2 program goals may be met by non-fundamental research. Some proposed research may present a high likelihood of disclosing performance characteristics of military systems or manufacturing technologies that are unique and critical to defense. Based on the anticipated type of proposer (e.g., university or industry) and the nature of the solicited work, the Government expects that some awards will include restrictions on the resultant research that will require the awardee to seek DARPA permission before publishing any information or results relative to the program.

Proposers should indicate in their proposal whether they believe the scope of the research included in their proposal is fundamental or not. While proposers should clearly explain the intended results of their research, the Government shall have sole discretion to determine whether the proposed research shall be considered fundamental and to select the award instrument type. Appropriate language will be included in resultant awards for non-fundamental research to prescribe publication requirements and other restrictions, as appropriate. This language can be found at http://www.darpa.mil/work-with-us/additional-baa.

http://www.darpa.mil/work-with-us/contract-management#OtherTransactions http://www.darpa.mil/work-with-us/contract-management#OtherTransactions http://www.darpa.mil/work-with-us/additional-baa

HR001120S0035 HABITUS 18

For certain research projects, it may be possible that although the research to be performed by a potential awardee is non-fundamental research, its proposed subawardee’s effort may be fundamental research. It is also possible that the research performed by a potential awardee is fundamental research while its proposed subawardee’s effort may be non-fundamental research.

In all cases, it is the potential awardee’s responsibility to explain in its proposal which proposed efforts are fundamental research and why the proposed efforts should be considered fundamental research.

To ensure high research standards and support reproducibility of work performed under this program, all publications will have to pre-register their hypotheses with either a journal or pre-registration service prior to data collection.

III. Eligibility Information A. Eligible Applicants

All responsible sources capable of satisfying the Government's needs may submit a proposal DARPA’s consideration.

1. Federally Funded Research and Development Centers (FFRDCs) and Government Entities

a. FFRDCs FFRDCs are subject to applicable direct competition limitations and cannot propose to this BAA in any capacity unless they meet the following conditions. (1) FFRDCs must clearly demonstrate that the proposed work is not otherwise available from the private sector. (2) FFRDCs must provide a letter, on official letterhead from their sponsoring organization, that (a) cites the specific authority establishing their eligibility to propose to Government solicitations and compete with industry, and (b) certifies the FFRDC’s compliance with the associated FFRDC sponsor agreement’s terms and conditions. These conditions are a requirement for FFRDCs proposing to be awardees or subawardees.

b. Government Entities Government Entities (e.g., Government/National laboratories, military educational institutions, etc.) are subject to applicable direct competition limitations. Government entities must clearly demonstrate that the work is not otherwise available from the private sector and provide written documentation citing the specific statutory authority and contractual authority, if relevant, establishing their ability to propose to Government solicitations and compete with industry. This information is required for Government Entities proposing to be awardees or subawardees.

c. Authority and Eligibility At the present time, DARPA does not consider 15 U.S.C. § 3710a to be sufficient legal authority to show eligibility. While 10 U.S.C.§ 2539b may be the appropriate statutory starting point for some entities, specific supporting regulatory guidance, together with evidence of agency approval, will still be required to fully establish eligibility. DARPA will consider FFRDC and

HR001120S0035 HABITUS 19

Government Entity eligibility submissions on a case-by-case basis; however, the burden to prove eligibility for all team members rests solely with the proposer.

2. Foreign Participation Non-U.S. organizations and/or individuals may participate to the extent that such participants comply with any necessary nondisclosure agreements, security regulations, export control laws, and other governing statutes applicable under the circumstances. For classified submissions, this includes mitigating any Foreign Ownership Control and Influence (FOCI) issues prior to transmitting the submission to DARPA. Additional information on these subjects can be found at https://www.dcsa.mil/mc/ctp/foci/.

B. Organizational Conflicts of Interest

FAR 9.5 Requirements In accordance with FAR 9.5, proposers are required to identify and disclose all facts relevant to potential OCIs involving the proposer’s organization and any proposed team member (subawardee, consultant). Under this Section, the proposer is responsible for providing this disclosure with each proposal submitted to the BAA. The disclosure must include the proposer’s, and as applicable, proposed team member’s OCI mitigation plan. The OCI mitigation plan must include a description of the actions the proposer has taken, or intends to take, to prevent the existence of conflicting roles that might bias the proposer’s judgment and to prevent the proposer from having unfair competitive advantage. The OCI mitigation plan will specifically discuss the disclosed OCI in the context of each of the OCI limitations outlined in FAR 9.505-1 through

FAR 9.505-4.

Agency Supplemental OCI Policy In addition, DARPA has a supplemental OCI policy that prohibits contractors/performers from concurrently providing Scientific Engineering Technical Assistance (SETA), Advisory and Assistance Services (A&AS) or similar support services and being a technical performer.

Therefore, as part of the FAR 9.5 disclosure requirement above, a proposer must affirm whether the proposer or any proposed team member (subawardee, consultant) is providing SETA, A&AS, or similar support to any DARPA office(s) under: (a) a current award or subaward; or (b) a past award or subaward that ended within one calendar year prior to the proposal’s submission date.

If SETA, A&AS, or similar support is being or was provided to any DARPA office(s), the proposal must include:

The…