HR001120S0035-Amendment-02.pdf

HR001120S0035 HABITUS 1

Broad Agency Announcement Habitus: Comparative Evaluation (CE)

Defense Sciences Office

HR001120S0035

Amendment 2

November 06, 2020

HR001120S0035 HABITUS 2

Table of Contents I. Funding Opportunity Description

A. Introduction B. Background C. Program Description and CE Scope D. Program Structure E. Technical Area Descriptions F. Additional Parameters for CE Proposers G. Schedule/Milestones H. CE Deliverables I. Other Program Objectives and Considerations

II. Award Information A. General Award Information B. Fundamental Research

III. Eligibility Information A. Eligible Applicants

1. Organizational Conflicts of Interest C. Cost Sharing/Matching

IV. Application and Submission Information A. Address to Request Application Package B. Content and Form of Application Submission C. Submission Dates and Times D. Funding Restrictions E. Other Submission Requirements

V. Application Review Information A. CE Evaluation Criteria B. Review and Selection Process C. Federal Awardee Performance and Integrity Information (FAPIIS)

VI. Award Administration Information A. Selection Notices B. Administrative and National Policy Requirements C. Reporting

VII. Agency Contacts VIII. Other Information

A. Frequently Asked Questions (FAQs)

BAA Attachments:

Attachment H: PROPOSAL SUMMARY SLIDE TEMPLATE Attachment I: PROPOSAL TEMPLATE VOLUME 1: TECHNICAL & MANAGEMENT Attachment J: PROPOSAL TEMPLATE VOLUME 2: COST Attachment K: MS ExcelTM DARPA COST PROPOSAL SPREADSHEET Attachment L: PROPOSAL TEMPLATE VOLUME 3: ADMINISTRATIVE & NATIONAL POLICY

REQUIREMENTS

HR001120S0035 HABITUS 3

PART I: OVERVIEW INFORMATION

Federal Agency Name: Defense Advanced Research Projects Agency (DARPA), Defense Sciences Office (DSO)

Funding Opportunity Title: Habitus: Comparative Evaluation (CE)

Announcement Type: Amendment 2

Funding Opportunity Number: HR001120S0035

Catalog of Federal Domestic Assistance (CFDA) Number(s): 12.910 Research and Technology Development

Dates (All times listed herein are Eastern Time.)

o CE BAA Posting Date: November 06, 2020 o FAQ Submission Deadline for CE: January 13, 2021, 4:00 p.m. See Section VIII.A o CE Full Proposal Due Date: January 20, 2021, 4:00 p.m.

o BAA closing date: January 22, 2021

Anticipated Individual Awards: DARPA anticipates a single award for CE.

Types of Instruments that May be Awarded: Procurement contracts, cooperative agreements, or Other Transactions. Award instruments will be limited to procurement contracts and Other Transactions for Proposers whose proposed solution includes Controlled Unclassified Information (CUI).

Agency contacts o Technical POC: Dr. Bart Russell, Program Manager, DARPA/DSO o BAA Email: Habitus@darpa.mil o BAA Mailing Address:

DARPA/DSO

ATTN: HR001120S0035

675 North Randolph Street Arlington, VA 22203-2114 o DARPA/DSO Opportunities Website: http://www.darpa.mil/work-with-us/opportunities

Frequently Asked Questions (FAQ): FAQs for this solicitation may be viewed on the DARPA/DSO Opportunities Website. See Section VIII.A for further information.

mailto:Habitus@darpa.mil http://www.darpa.mil/work-with-us/opportunities?tFilter=&oFilter=2&sort=name http://www.darpa.mil/work-with-us/opportunities?tFilter=&oFilter=2&sort=name

HR001120S0035 HABITUS 4

PART II: FULL TEXT OF ANNOUNCEMENT

I. Funding Opportunity Description

This Broad Agency Announcement (BAA) constitutes a public notice of a competitive funding opportunity as described in Federal Acquisition Regulation (FAR) 6.102(d)(2) and 35.016 as well as 2 C.F.R. § 200.203. Any resultant negotiations and/or awards will follow all laws and regulations applicable to the specific award instrument(s) available under this BAA, e.g., FAR

15.4 for procurement contracts.

A. Introduction The Defense Sciences Office (DSO) at the Defense Advanced Research Projects Agency (DARPA) is soliciting innovative research proposals to use traditional methods of accessing local knowledge to aid operational decision making to enable a comparative evaluation (CE) of efficacy and efficiency against experimental methods developed under TA1/TA2 of the Habitus program as described below. Work solicited under this announcement is expressly for the Comparative Evaluation (CE) portion of the Habitus Program. DARPA is not accepting proposals for the TA1/TA2 portion of the program at this time. All information about TA1 and TA2 included in this amendment is for the purposes of informing CE proposers only.

In order to evaluate the efficacy of the Habitus experimental TA1/TA2 approaches to predict outcomes, DARPA will compare the performance of the computational cognitive models developed under TA1/TA2 against more traditional methods such as market research and polling applied by the Comparative Evaluation (CE) team (the subject of this announcement). In contrast to TA1/TA2, the CE team will apply traditional methods in innovative ways to provide comparison data for each question posed by the TA1/TA2 teams; the accuracy of TA1/TA2 and CE methods will both be compared to verifiable, real-world outcomes to determine whether experimental methods provide tangible advantages over traditional methods applied in innovative ways.

B. Background Understanding how to work with and influence local systems1 in support of stability operations is critical for operational decision making and is most challenging in undergoverned regions in which the systems themselves are often in flux or illegible.2 Establishing stability in such regions requires we facilitate actions that are in line with local views, yet our current means for understanding local systems such as the political, socioeconomic, and/or those related to health and infrastructure are limited. This has led to operational challenges. Examples include:

1 By “systems” for the purposes of this BAA we mean an interconnecting network of factors that affect or contribute to a real-world outcome. An agricultural system might include, for instance, natural factors (e.g., rainfall, temperatures) and human factors (e.g., irrigation methods, planting practices, and formal or informal regulations, and even perceptions of events) that affect the outcome of seasonal agricultural yields and/or crop prices.

2 We use “illegibility” here as James C. Scott has developed the concept to refer to ways in which states understand those they govern. Scott, J. C. (1998). Seeing like a state: How certain schemes to improve the human condition have failed. Yale University Press. Full text: https://libcom.org/files/Seeing%20Like%20a%20State%20- %20James%20C.%20Scott.pdf https://libcom.org/files/Seeing%20Like%20a%20State%20-%20James%20C.%20Scott.pdf https://libcom.org/files/Seeing%20Like%20a%20State%20-%20James%20C.%20Scott.pdf

HR001120S0035 HABITUS 5

The failure to recognize the interaction of local burial practices with Ebola’s virulence in West Africa led to a vast underprojection of the disease’s spread in 2014. Efforts to treat it as a health and education issue misunderstood the local importance of these rituals;

solutions that included local religious leadership were far more effective than education-only approaches.3

The US’s early Information Operations campaign in Iraq was largely ineffective because Iraqis had learned to distrust traditional media outlets (radio, television) after years of listening to Ba’athist propaganda.4

Efforts to secure a small town from ISIS created local turmoil when US forces set up a perimeter that excluded the local glass factory and disrupted locals’ ability to get to work.

Such cultural factors, when studied and described in a vacuum, provide little quantitative or predictive power for understanding real-world outcomes or for informing operational decisions.

Description of TA1/TA2

The TA1/TA2 portion of the Habitus program is exploring a potential way to overcome these limitations by creating a capability that makes implicit local understanding explicit and available to operational decision makers. Humans develop causal cognitive representations – or cognitive models – of systems of which they are a part. These models include factors (or variables), relationships among factors, and contexts that affect both. The knowledge behind these models is often hyper-localized, changing dramatically with regional and/or population dependent interactions of factors such as terrain, industries, population density (urban, rural), shared history, formal and informal power structures, religion, and ethnicity. These cognitive models, though often implicit, allow one to estimate which factors are most important for a given outcome and how those factors interact to anticipate future outcomes based on history, current events, and trends. If accurate, an explicit collective representation of these implicit, individual cognitive models would be able to accurately predict a system outcome, whether concrete (e.g., employment rates, rain totals) or traditionally unquantifiable (e.g., religious beliefs, historical experience). It should also allow one to estimate which factors are most important for a given outcome and how those factors interact to anticipate future outcomes based on history, current events, and trends.

An overarching objective of the program is to determine whether such explicit computational models of complex, real-world local systems based on collective cognitive models of local populations can better inform and improve operational decision making and lead to better outcomes (Figure 1). To improve upon current practices, the experimental TA1/TA2 capability must be specific enough to anticipate system-level effects in response to events that are generalizable across regions and populations, adaptive as societies change over time, and self-sustaining for maintainability and persistence. 5

3 Maxmen, A. (2015). How the fight against Ebola tested a culture’s traditions. National Geographic, 30.

4 McFate, M. (2005). The military utility of understanding adversary culture. Office of Naval Research, Arlington

VA.

5 Adaptive means system representations and projections update to maintain accuracy as non-ergodic human systems change over time. Scalable & generalizable means that the methodology readily transfers to new regions

HR001120S0035 HABITUS 6

Difference between TA1/TA2 and CE

Traditional methods such as surveys, polling, and market research remain common states of practice because the methods generalize widely. Even though polling and market research are not adaptive nor self-sustaining (a poll must be completely rerun to account for changing views as a result of emerging events) it is not clear if experimental approaches envisioned by TA1/TA2 will provide tangible cost or labor advantages over efficient applications of traditional means augmented by new forms of technology and access. Therefore, rather than evaluate TA1/TA2 against an arbitrary threshold for accuracy or efficiency, Habitus will compare TA1/TA2 model performance against CE using traditional methods to answer the same questions.

To assess the efficacy of TA1/TA2 solutions, the CE effort will generate comparative data against these metrics using innovative strategies for applying traditional methods. The key differences between TA1/TA2 and CE methods is that the CE team (1) will not be building causal models based on local understanding and (2) will use existing methods for engaging with populations rather than developing new engagement strategies.

It is important for both TA1/TA2 and CE performers to understand the application spaces Habitus aims to support. One example is a US Marine Air-Ground Task Force (MAGTF) deployment to East Asian regions with high levels of geographic, ethnic, and linguistic diversity.

Arriving forces would need immediate (within a few days), coarse understanding of the most critical system factors, such as major players, industries, populations, and their locations and affiliations. They will need a more robust, detailed, quantitative understanding in a few weeks.

Additionally, the analysis will need to be specific to the community or system of study; it is not sufficient to poll a sample from all Nigerians if the system in question only affects farmers in a and populations. Self-sustaining means it requires few resources to maintain over time. Specific means it is detailed enough to provide causal insight for a given region, system, and/or population.

Figure 1. Habitus TA1/TA2 System Concept. Through engagement with local populations, Habitus TA1/TA2 performers will create an explicit representation of the collective local cognitive model of a system in a (performer-specified) topic domain. A commander (decision maker) can view and query the computational model for assessments of system-level changes based on actual or hypothetical actions or events, providing the commander with an in-depth understanding of the system and its causal dynamics.

HR001120S0035 HABITUS 7

small region of the country. Therefore, one kind of innovation for traditional methods may involve more specific ways of identifying relevant populations based on behavior rather than on what may be an arbitrary geographic boundary.

C. Program Description and CE Scope The TA1/TA2 performers are working to create a specific, generalizable/scalable, adaptive, and self-sustaining methodology for capturing and making local knowledge available to operational decision makers to support stability operations. This involves developing mechanisms of measurement and local engagement, identifying relevant factors, prioritizing and developing relationships among those factors, and testing and updating the model as systems change. Over the course of the program, TA1/TA2 performers will demonstrate, test, and validate the system by using the computational models generated by their systems to make predictions against real-world, externally verifiable outcomes. CE will be using traditional methods to make predictions against the same questions. DARPA’s T&E team will compare both set of predictions against the same real-world, externally verifiable outcomes. The proposals for TA1/TA2 have already been received. This amendment is only soliciting proposals for the CE portion of the program.

Research proposed against this amendment should address how proposers will use traditional methods in innovative ways for understanding local populations and systems. CE proposers should provide specific information about the methods they are proposing, how they will access local populations, how often they will sample, and what kind of data the proposed method or methods will generate. CE proposers should include a detailed work and staffing plan demonstrating their capability and expertise in the proposed methods. If proposing in-country work, proposers are strongly encouraged to either work with local partners or demonstrate past successful performance working in these or similar regions.

DARPA anticipates making a single award for the CE component of the program. As such, CE proposers are expected to address (including costs) their plan for creating parallel data for all regions that TA1/TA2 performers will be operating in across all program phases (see Table 1 in Section E for a list of possible topics and regions by phase).

D. Program Structure Habitus consists of two Technical Areas (TAs), a Comparative Evaluation (CE) team, and a Government Test and Evaluation (T&E) team. TA1 Model Development is focused on developing explicit representations of implicit causal local knowledge. TA2 Engagement Mechanism is developing new methods for interacting with local populations in efficient and self-sustaining ways.

The CE team will be generating data using traditional methods for comparison against TA1/TA2 performance results. The T&E team will compare performance of all teams to assess accuracy against real-world event outcomes and relative performance against other teams (TA1/TA2 vs.

CE). In order to ensure the impartiality of the evaluation of the TA1 and TA2 technology, proposers who have been selected for negotiation (primes and subcontractors) of an award for TA1/TA2 are not eligible to propose to the CE portion solicited in this BAA Amendment. The

HR001120S0035 HABITUS 8

description of the TA1 and TA2 efforts that follows is for informational purposes only in order to assist CE proposers in preparing their proposals.

The larger Habitus program will consist of three phases and a total of 42 months. The period of performance for the CE portion will be during the final 30 months of the program.

For CE:

Because the CE portion focuses on innovative applications of existing methods rather than development of new technologies, the CE portion will begin roughly 12 months into TA1/TA2’s Phase 1 efforts and will run for the duration of the rest of the program (30 months). In all phases, the CE team must generate comparative data on a quarterly basis for each topic-region combination. A topic-region is the conjunction of a topic of study (e.g., food security) and a regional population (e.g., Southeast Asia). A list of topic-regions proposed by TA1/TA2 performers is listed in Table 1 below.

CE Phase 1 (Base period, 12 months) will include collecting comparative data for up to five (5) topic-region combinations.

CE Phase 2 (Option 1, 12 months) will include collecting comparative data for up to six (6) total topic-region combinations. The CE team will collect data in three (3) topic-region combinations in the first 6 months and three (3) in the second six months of the phase.

CE Phase 3 (Option 2, 6 months) will require collecting comparative data for one (1) additional location that may include those listed in Table 1 or another topic-region as determined in the later stages of the program.

TA1/TA2 performers in consultation with the T&E and CE teams will define the specific outcome questions and locations all performers will have to predict. The initial list will be provided to the CE team before the CE kickoff (approximately month 12 of the Habitus program). The topic/locations specified in Table 1 are subject to change due to programmatic and/or technical considerations. Therefore, CE proposers should use these as a guide but be flexible enough to adjust to a new location as the program requires.

E. Technical Area Descriptions Comparative Evaluation

The CE team will generate parallel outcome predictions to those specified by TA1/TA2 using innovative applications of current methods – traditional market research or survey/polling practices – in order to generate a question-for-question comparative data set that the T&E team will use to evaluate program performance. For CE, DARPA is seeking innovative ways to make these current methods more efficient. In each phase, specific questions relevant to each topic-region will be proposed by the TA1/TA2 teams and approved by the T&E team in consultation with the CE team. All of the outcome questions selected will be verifiable with real-world event outcomes. The T&E team will work with both TA1/TA2 and CE performers to resolve any ambiguity or inconsistencies among data sets and outcomes in order to provide the most appropriate comparisons of accuracy. The topics of study proposed include food security;

agricultural systems including supply chains; water, sanitation, and hygiene (WASH) practices

HR001120S0035 HABITUS 9

related to waterborne illness; maternal and child health (MC health) outcomes; and incidence of violence. More details on the types of outcome questions and the local population being probed can be found in Section F.

The single CE team must be able to provide comparative data on a quarterly basis for all topic-region combinations studied in each phase. All of the potential combinations of topic and location are listed in Table 1. For purposes of the proposal, it should be assumed that in Phase 1 the CE team will need to handle all five of the combinations listed. In Phase 2, the CE team will have to handle up to six of the combinations listed, specific combinations to be determined. In Phase 3, the CE team will have to handle one additional region to be determined in Phase 2. The CE team will be contracted to perform comparative assessments for a maximum of 5 topic-region combinations in Phase 1, a maximum of 6 topic-region combinations in Phase 2, and 1 topic-region combination in Phase 3.

Phase 1 Phase 2 Phase 3 Region Topic Region Topic Region Topic

Caribbean WASH central Africa WASH southeast Asia TBD west Africa agricultural supply chains west Africa agricultural supply chains South America

TBD

west Africa mc health west Africa agricultural supply chains north Africa TBD south Africa food security west Africa mc health south Africa violence west Africa mc health east Africa WASH South America violence south Asia food security southeast Asia food security

Table 1: Possible topics and regions by phase for the CE effort. A topic-region is the combination of a location with a domain topic. For instance, Phase 1 may include the study of violence in a region in southern Africa, and Phase 2 may include the study of food security outcomes in south Asia. Not all topic-regions in Phases 2 and 3 will be studied because of planned TA1/TA2 down-selections at the end of each phase.

CE solutions will be evaluated on accuracy of their prediction for the same questions posed by TA1/TA2 performers against real-world event outcomes, against TA1/TA2 performers’ relative accuracy rate, and on the CE team’s ability to generate the necessary data on at least a quarterly basis. The CE team will also record touchpoint6 rates for comparing efficiency.

NOTE - DARPA is not soliciting for TA1/TA2 at this time, but information about the TA1/TA2 efforts is included here to assist the CE proposers. Proposed CE solutions are not required to have each of the features described below for TA1 and TA2.

6 A Touchpoint is the number of times the CE team and the TA2 engagement mechanism must request input from the participant, against the number of complete data points collected. See Section I.G. for additional information.

HR001120S0035 HABITUS 10

TA1: Model Development. TA1 teams are developing scalable and generalizable knowledge elicitation methods to semi-automatically and rapidly derive computational model factors and system level relationships (the system model). The methods proposed were required to:

Be specific and quantifiable. The TA1 models quantify the relationships among factors and generate quantifiable predictions based on changes in the system.

Capture implicit local knowledge. Elicitation strategies account for the fact that people are often not explicitly aware of the sources or indicators of their knowledge. TA1 methods extract implicit knowledge and incorporate it into the TA1 models.

Capture unanticipated factors. The methods elicit or discover factors the performer team or outside experts did not anticipate. This involves identifying new terms or colloquialisms locals use to describe either the same factor outside experts use, and/or discovering new, previously unspecified factors. The methods acknowledge and address challenges associated with education levels (e.g., literacy).

Incorporate local beliefs and/or practices. Local populations may have beliefs and/or practices that may not be proven by empirical scientific evidence, but nevertheless influence behavior and system outcomes. Outsiders often miss these elements, which can lead to miscalculations and misunderstandings. Proposed methods discover and incorporate these beliefs and practices into their computational models.

Represent dynamic systems. To achieve adaptivity, the computational models (1) update the relative importance of factors and their relationships to the predicted outcome and (2) insert new factors, relationships, and contexts into the model as more information is gathered and as the system evolves.

Empower non-experts to anticipate real-world system outcomes. The computational models allow the non-expert user (i.e., with no special training in data science, behavioral or social science, or local languages) to adjust factors to explore the potential outcomes of given actions and provide understanding behind the projected outcome.

TA2: Engagement Mechanism. TA2 teams are creating self-sustaining capabilities that continually provide information necessary to build and update TA1’s computational models by providing value to the local users. The solutions include both passive observation and active engagement. Passive observations use natural behaviors that occur independent of Habitus, while active engagements are direct interactions with locals such as queries or probes. Each active engagement is considered a touchpoint with locals with respect to the Habitus metrics (Table 3).

The solutions:

Are Generalizable. Proposers are demonstrating the capability’s effectiveness and usefulness in more than one region and with multiple populations.

Are able to align incentives of Habitus with the local population to support accuracy.

Methods that rely solely on monetary incentives encourage participation but not necessarily accuracy. Proposers’ engagement mechanisms provide value to locals in a way that aligns with existing motivations (e.g., intrinsic incentives) to encourage participation and accuracy in responses.

HR001120S0035 HABITUS 11

Identify and efficiently sample all relevant populations. TA2 teams are identifying populations relevant to the domain topic they propose and are engaging without oversampling or harassing the local population. The methods under development identify the group(s) to which each sample belongs (including multi-group membership when applicable). In addition, the methodologies address the challenges of obtaining representative samples from a small population.

Accommodate characteristics of the studied population. Each local population has characteristics that were considered when creating the engagement mechanism. Examples of these characteristics include literacy rates, likely frequency of use, collection platform, communication equipment, infrastructure, and applications.7 The engagement mechanisms align with the local infrastructures of the subject populations. The methods under development consider current infrastructure or platform and those that will be available in the near future.

Demonstrate self-sustainment. In order to provide long-term transitionable value, the methods minimize impediments to self-sustainment; some include information exchange paradigms. The mechanisms maximize the inherently limited opportunities for engagement.

This is a particular problem for local populations where the total potential pool of respondents is limited and the length of (intermittent) engagement is long.

Address engagement effects. Performers are addressing the potential interaction effects of the engagement method. An example of interaction effects is a perspective based bias that can be introduced by the investigator. All engagement methods will obtain informed consent in line with Institutional Review Board (IRB) and Human Research Protection Office (HRPO) requirement.

F. Additional Parameters for CE Proposers

The Habitus program will develop computational models from, and demonstrate their engagement mechanisms among, populations and under circumstances that will exhibit specificity, adaptivity, generalizability, and self-sustainability. In addition to the TA1/TA2 performers’ predictions, the CE team will survey outcome questions subject to the following constraint:

Real world event predictions. TA1/TA2 teams are required to generate predictions for externally verifiable real word events. In order to validate those methods, the CE teams must propose to generate predictions for the same events on the same dates using their outcome questions. The TA1/TA2 and CE teams’ prediction accuracy will be determined by comparing predictions against real-world event outcomes. The events studied have the following features:

o Ground truth is externally verifiable (unemployment rates, agricultural yields, crop prices, infection rates, etc.).

7 For instance, currently dumb phones are far more prevalent than any kind of smartphone in Africa, however “dumb smartphones” are on the rise. Likewise, Facebook messenger and WhatsApp are far more popular than SMS in South America, but SMS is still the dominant messaging mechanism certain regions of Africa.

HR001120S0035 HABITUS 12

o Events occur with enough frequency so that there are sufficient instances of ground truth to evaluate progress on a quarterly basis.

Relative accuracy is determined by comparing TA1/TA2 accuracy rates against the CE team’s accuracy rates.

These predictions will be made by interacting with local populations with the following characteristics:

Population size. In order to balance the desire to probe local behavior with the need for statistical power, the populations studied by all teams are required to be between 50,000 and 500,000 people in regions Outside the Continental United States (OCONUS). The populations may have significant heterogeneity, especially in the latter phases of the program.

Language. The TA1/TA2 teams have selected populations whose primary language is not English and who might be using local dialects. The CE team will need to demonstrate their ability to work across languages if and when necessary for their approach.

Human Subjects Research (HSR) determination and OCONUS HSR. Traditional survey, polling, and market research methods are often not considered HSR.

o If seeking a non-HSR determination, CE must obtain the initial determination through their local or private IRB and submit the determination to the HRPO for concurrence and approval before any work can begin. Methods that do not collect personally identifiable information (PII) are more likely to be deemed non-HSR.

o If proposed CE methods are determined to be HSR by the proposer’s local IRB or if the Government HRPO does not concur with the local IRB’s non-HSR determination, CE teams must either amend their methods to make them non-HSR or add the additional step of obtaining in-country approval for HSR. If CE proposers know their methods qualify as HSR, they are strongly encouraged to work with local partners to facilitate in-country HSR approval and ensure research adheres to local HSR requirements.8 In this case they must also address how they intend to meet program schedule milestones. If CE proposers choose not to partner locally, they must demonstrate experience obtaining HSR approval in the proposed region.

All approvals must be finalized before any work can begin. Determinations and/or approvals must be obtained with enough time to provide comparative data six months after CE award (month 18 of Phase 1).

8 In addition to adhering to US and HRPO standards for HSR, all HSR protocols must adhere to the HSR requirements of the nation in which the work will occur. See the Health and Human Services website for guidance.

https://www.hhs.gov/ohrp/international/compilation-human-research-standards/index.html. Partnering with a local research institution that is already familiar with their national requirements and approval processes will facilitate protocol adherence to local, US, and HRPO requirements.

HR001120S0035 HABITUS 13

COVID-19 considerations. CE proposers are strongly encouraged to work with in-country partners to facilitate activities despite potential ongoing travel restrictions imposed by the COVID-19 pandemic.

Proposed CE solutions that rely solely on social media and/or Mechanical Turk are out of scope for this solicitation.

G. Schedule/Milestones Over the course of the program, the CE team will be evaluated based on their ability to capture data for comparison to the TA1/TA2 predictions and the timely transfer of the data to the T&E team for validation. The CE team will need to provide data for validation of predictions at a minimum on a quarterly basis starting six months after award for all specified topic-regions by phase. In addition, it is expected that the CE team will record the number of touchpoints required to collect their data so as to be able to compare efficiency with the methods developed by TA2 performers. If the selected CE team cannot generate the data necessary to make comparative predictions, the Government reserves the right to discontinue CE funding and seek other means to generate the necessary comparisons. The CE team will also be evaluated on its expected ability to perform in subsequent phases based on Phase 1 performance. Options may be exercised, at the Government’s sole discretion, based on technical progress measured against the metrics and milestones defined in this BAA amendment as well as funding availability.

CE Milestones: The CE team must provide comparative predictions and data to support those predictions on a quarterly basis starting six months after award (Month 18 of the Habitus program).

CE Metrics:

The CE team will be expected to meet and maintain performance metrics for (1) time to collect a sample and generate a new prediction in a new region and (2) touchpoint ratios to maintain efficiency at or above baseline as outlined in Table 2. The CE team is required to meet these metrics and encouraged to perform better than these metrics if and when possible.

CE Metrics Phase 1 Phases 2 & 3

Time to collect new sample in a new region < 6 weeks < 2 weeks

Efficiency Touchpoint Ratio 1:2000 Table 2. CE Metrics by Phase.

HR001120S0035 HABITUS 14

For reference, the TA1/TA2 metrics are outlined in Table 3 below. “Time to collect new sample in a new region” in CE metrics is analogous to “Time to establish in new region” in TA1/TA2 metrics.

Brier Skill Score for Accuracy: To evaluate the efficacy of TA1 predictions compared to traditional methods, DARPA will use a variation of the Brier Skill Score (BSS), which compares the relative accuracy of the TA1 performer’s method against the CE team’s prediction accuracy.

DARPA is using this comparative method rather than providing an absolute accuracy threshold to encourage TA1/TA2 performers to choose challenging topics and events for which there may be no current reliable methods for assessing the local view (the system outcome must still have externally verifiable ground truth, even if current methods are poor at predicting those outcomes). Note that in our notation a BSS of > 0 means TA1 is performing better than the CE team, and a BSS of <0 means CE is outperforming TA1. TA1 performers will need to establish this level of performance by the end of Phase 1 and demonstrate that they can maintain that performance in new regions and in shorter amounts of time across Phases 2 and 3. The CE team should strive to reach as accurate a prediction as possible against real world outcomes using its selected methods. Accuracy for CE will be recorded but the CE team’s continuation in the program will not be determined based on accuracy rates.

Touchpoint Ratio for Efficiency: DARPA will evaluate efficiency by comparing the number of touchpoints. All attempts to contact locals to participate (not just interactions with those who consent) and the process of obtaining consent as part of IRB (if applicable) and/or survey compliance will count against the touchpoint ratio for both CE and TA2.

Other proposal properties

CE proposers should provide a technical and programmatic strategy that conforms to the entire program schedule and presents a plan to fully address the CE metrics and milestones in this BAA Amendment.

Table 3. TA1/TA2 performance metrics by phase.

HR001120S0035 HABITUS 15

The proposed CE task structure must be consistent across the proposed schedule, Statement of Work, and cost volume.

A target start date of 1 October 2021 for the CE effort may be assumed for planning purposes.

The plan is for schedules to be synchronized across performers, as required, such that CE kickoff can coincide with month 12 of Phase 1. Schedules will be monitored/revised as necessary throughout the program.

All CE proposals must include the following meetings and travel in the proposed schedule and costs:

o To continue integration and development between TAs, foster collaboration between teams, and disseminate program developments, a two-day Principal Investigator (PI) meeting will be held approximately every six months, with locations split between the East and West Coasts of the United States. The CE team’s first PI meeting will coincide with TA1/TA2’s third PI meeting, tentatively scheduled for October of 2021. For budgeting purposes, plan for six two-day meetings over the course of 30 months: four meetings in the Washington, D.C. area and two meetings in the San Francisco, CA area. The first meeting for the CE effort will be the CE kickoff in month 1.

o Regular teleconference meetings will be scheduled with the Government team for progress reporting as well as problem identification and mitigation.

Proposers should also anticipate at least one site visit per phase by the DARPA Program Manager during which the CE Team will have the opportunity to demonstrate progress towards agreed-upon milestones.

H. CE Deliverables

CE performers will be expected to provide at a minimum the following deliverables:

Comprehensive quarterly technical reports due within ten days of the end of the given quarter, describing progress made on the specific milestones as laid out in the Statement of Work

Data necessary for the T&E team to evaluate performer progress against accuracy at least 90 days before the end of Phases 1 and 2A phase completion report submitted within 30 days of the end of each phase, summarizing the research done

Other negotiated deliverables specific to the objectives of the individual efforts. These may include items such as a data management plan.

Reporting as outlined in Section VI.C.

I. Other Program Objectives and Considerations

1. Collaboration

TA1/TA2 and the CE team will be required to work with the Government T&E team who will

HR001120S0035 HABITUS 16

evaluate performance against the metrics outlined in Tables 2 and 3, as applicable. All performers will – at a minimum – be required to provide the Government T&E team with their predictions prior to the specified event’s outcome so the T&E team can compare predictions against actual event outcomes and determine which method was most accurate. Collaboration may also include sharing all raw data, code for reproducing results, and output results.

II. Award Information A. General Award Information

DARPA anticipates a single award for the CE team.

The level of funding for individual awards made under this BAA will depend on the quality of the proposals received and the availability of funds. DARPA anticipates a single award will be made to the proposer9 whose proposal is determined to be the most advantageous to the Government, all evaluation factors considered. See Section V for further information.

The Government reserves the right to:

select for negotiation all, some, one, or none of the proposals received in response to this solicitation;

make awards without discussions with proposers;

conduct discussions with proposers if it is later determined to be necessary;

segregate portions of resulting awards into pre-priced options;

accept proposals in their entirety or to select only portions of proposals for award;

fund awards in increments with options for continued work at the end of one or more phases;

request additional documentation once the award instrument has been determined (e.g., representations and certifications); and remove proposers from award consideration should the parties fail to reach agreement on award terms within a reasonable time or the proposer fails to provide requested additional information in a timely manner.

Proposals identified for negotiation may result in a procurement contract, cooperative agreement, or Other Transaction (OT), depending upon the nature of the work proposed, the required degree of interaction between parties, and other factors.

9 As used throughout this BAA, “proposer” refers to the lead organization on a submission to this BAA. The proposer is responsible for ensuring that all information required by a BAA--from all team members--is submitted in accordance with the BAA. “Awardee” refers to anyone who might receive a prime award from the Government, including recipients of procurement contracts, cooperative agreements, or Other Transactions. “Subawardee” refers to anyone who might receive a subaward from a prime awardee (e.g., subawardee, consultant, etc.).

HR001120S0035 HABITUS 17

Proposers looking for innovative, commercial-like contractual arrangements are encouraged to consider requesting Other Transactions. To understand the flexibility and options associated with Other Transactions, consult http://www.darpa.mil/work-with-us/contract-management#OtherTransactions.

In accordance with 10 U.S.C. § 2371b(f), the Government may award a follow-on production contract or Other Transaction (OT) for any OT awarded under this BAA if: (1) that participant in the OT, or a recognized successor in interest to the OT, successfully completed the entire prototype project provided for in the OT, as modified; and (2) the OT provides for the award of a follow-on production contract or OT to the participant, or a recognized successor in interest to the OT.

In all cases, the Government contracting officer shall have sole discretion to select award instrument type, regardless of instrument type proposed, and to negotiate all instrument terms and conditions with selectees. DARPA will apply publication or other restrictions, as necessary, if it determines that the research resulting from the proposed effort will present a high likelihood of disclosing performance characteristics of military systems or manufacturing technologies that are unique and critical to defense. Any award resulting from such a determination will include a requirement for DARPA permission before publishing any information or results on the program. For more information on publication restrictions, see the section below on Fundamental Research.

B. Fundamental Research

It is DoD policy that the publication of products of fundamental research will remain unrestricted to the maximum extent possible. National Security Decision Directive (NSDD) 189 defines fundamental research as follows:

‘Fundamental research’ means basic and applied research in science and engineering, the results of which ordinarily are published and shared broadly within the scientific community, as distinguished from proprietary research and from industrial development, design, production, and product utilization, the results of which ordinarily are restricted for proprietary or national security reasons.

As of the date of publication of this BAA, the Government expects that the CE effort as described herein may be met by proposers intending to perform fundamental research and does not anticipate applying publication restrictions of any kind to individual awards for fundamental research that may result from this BAA. Notwithstanding this statement of expectation, the Government is not prohibited from considering and selecting research proposals that, while perhaps not qualifying as fundamental research under the foregoing definition, still meet the BAA criteria for submissions. If proposals are selected for award that offer other than a fundamental research solution, the Government will either work with the proposer to modify the proposed statement of work to bring the research back into line with fundamental research or else the proposer will agree to restrictions in order to receive an award.

http://www.darpa.mil/work-with-us/contract-management#OtherTransactions

HR001120S0035 HABITUS 18

Proposers should indicate in their proposal whether they believe the scope of the research included in their proposal is fundamental or not. While proposers should clearly explain the intended results of their research, the Government shall have sole discretion to determine whether the proposed research shall be considered fundamental and to select the award instrument type. Appropriate language will be included in resultant awards for non-fundamental research to prescribe publication requirements and other restrictions, as appropriate. This language can be found at http://www.darpa.mil/work-with-us/additional-baa.

For certain research projects, it may be possible that although the research to be performed by a potential awardee is non-fundamental research, its proposed subawardee’s effort may be fundamental research. It is also possible that the research performed by a potential awardee is fundamental research while its proposed subawardee’s effort may be non-fundamental research.

In all cases, it is the potential awardee’s responsibility to explain in its proposal which proposed efforts are fundamental research and why the proposed efforts should be considered fundamental research.

To ensure high research standards and support reproducibility of work performed under this program, all performers are strongly encouraged to pre-register their hypotheses with either a journal or pre-registration service prior to data collection for all planned publications.

III. Eligibility Information A. Eligible Applicants

All responsible sources capable of satisfying the Government's needs may submit a proposal DARPA’s consideration.

1. Federally Funded Research and Development Centers (FFRDCs) and Government Entities

a. FFRDCs

FFRDCs are subject to applicable direct competition limitations and cannot propose to this BAA in any capacity unless they meet the following conditions. (1) FFRDCs must clearly demonstrate that the proposed work is not otherwise available from the private sector. (2) FFRDCs must provide a letter, on official letterhead from their sponsoring organization, that (a) cites the specific authority establishing their eligibility to propose to Government solicitations and compete with industry, and (b) certifies the FFRDC’s compliance with the associated FFRDC sponsor agreement’s terms and conditions. These conditions are a requirement for FFRDCs proposing to be awardees or subawardees.

b. Government Entities

Government Entities (e.g., Government/National laboratories, military educational institutions, etc.) are subject to applicable direct competition limitations. Government Entities must clearly demonstrate that the work is not otherwise available from the private sector and provide written documentation citing the specific statutory authority and contractual authority, if relevant, http://www.darpa.mil/work-with-us/additional-baa

HR001120S0035 HABITUS 19

establishing their ability to propose to Government solicitations and compete with industry. This information is required for Government Entities proposing to be awardees or subawardees.

c. Authority and Eligibility

At the present time, DARPA does not consider 15 U.S.C. § 3710a to be sufficient legal authority to show eligibility. While 10 U.S.C.§ 2539b may be the appropriate statutory starting point for some entities, specific supporting regulatory guidance, together with evidence of agency approval, will still be required to fully establish eligibility. DARPA will consider FFRDC and Government Entity eligibility submissions on a case-by-case basis; however, the burden to prove eligibility for all team members rests solely with the proposer.

2. Foreign Participation Non-U.S. organizations and/or individuals may participate to the extent that such participants comply with any necessary nondisclosure agreements, security regulations, export control laws, and other governing statutes applicable under the circumstances. For classified submissions, this includes mitigating any Foreign Ownership Control and Influence (FOCI) issues prior to transmitting the submission to DARPA. Additional information on these subjects can be found at https://www.dcsa.mil/mc/ctp/foci/.

B. Other Eligibility Requirements

1. Organizational Conflicts of Interest

FAR 9.5 Requirements In accordance with FAR 9.5, proposers are required to identify and disclose all facts relevant to potential OCIs involving the proposer’s organization and any proposed team member (subawardee, consultant). Under this Section, the proposer is responsible for providing this disclosure with each proposal submitted to the BAA. The disclosure must include the proposer’s, and as applicable, proposed team member’s OCI mitigation plan. The OCI mitigation plan must include a description of the actions the proposer has taken, or intends to take, to prevent the existence of conflicting roles that might bias the proposer’s judgment and to prevent the proposer from having unfair competitive advantage. The OCI mitigation plan will specifically discuss the disclosed OCI in the context of each of the OCI limitations outlined in FAR 9.505-1 through

FAR 9.505-4.

Agency Supplemental OCI Policy In addition, DARPA has a supplemental OCI policy that prohibits contractors/performers from concurrently providing Scientific Engineering Technical Assistance (SETA), Advisory and Assistance Services (A&AS) or similar support services and being a technical performer.

Therefore, as part of the FAR 9.5 disclosure requirement above, a proposer must affirm whether the proposer or any proposed team member (subawardee, consultant) is providing SETA, A&AS, or similar support to any DARPA office(s) under: (a) a current award or subaward; or (b) a past award or subaward that ended within one calendar year prior to the proposal’s submission date.

https://www.dcsa.mil/mc/ctp/foci/

HR001120S0035 HABITUS 20

If SETA, A&AS, or similar support is being or was provided to any DARPA office(s), the proposal must include:

The name of the DARPA office receiving the support;

The prime contract number;

Identification of proposed team member (subawardee, consultant) providing the support; and An OCI mitigation plan in accordance with FAR 9.5.

Government Procedures In accordance with FAR 9.503, 9.504 and 9.506, the Government will evaluate OCI mitigation plans to avoid, neutralize or mitigate potential OCI issues before award and to determine whether…