HR001121S0008-Amendment-01.pdf

Broad Agency Announcement INfluence Campaign Awareness and Sensemaking (INCAS)

HR001121S0008

October 28, 2020

Amendment 1 November 3, 2020

Defense Advanced Research Projects Agency Information Innovation Office 675 North Randolph Street Arlington, VA 22203-2114

HR001121S0008 INCAS 2

Summary of Amendment 1 Changes:

The purpose of this amendment is to publish the Controlled Unclassified Information Guide.

HR001121S0008 INCAS 3

Table of Contents Part I: Overview Information……………………………………………………………………...………3

Part II: Full Text of Announcement………………………………………………………………………..5

I. Funding Opportunity Description

II. Award Information

A. Awards

B. Fundamental Research

C. Disclosure of Information and Compliance with Safeguarding Covered Defense Information Controls

III. Eligibility Information

A. Eligible Applicants

B. Organizational Conflicts of Interest

C. Cost Sharing/Matching

D. Other Eligibility Requirements

IV. Application and Submission Information

A. Address to Request Application Package

B. Content and Form of Application Submission

C. Submission Dates and Times

D. Funding Restrictions

E. Other Submission Requirements

V. Application Review Information

A. Evaluation Criteria

B. Review and Selection Process

VI. Award Administration Information

A. Selection Notices

B. Administrative and National Policy Requirements

C. Reporting

VII. Agency Contacts

VIII. Other Information

A. Frequently Asked Questions (FAQs)

B. Proposers Day

C. Submission Checklist

D. Associate Contractor Agreement (ACA)

HR001121S0008 INCAS 4

PART I: OVERVIEW INFORMATION

Federal Agency Name: Defense Advanced Research Projects Agency (DARPA), Information Innovation Office (I2O)

Funding Opportunity Title: INfluence Campaign Awareness and Sensemaking

(INCAS)

Announcement Type: Initial Announcement

Funding Opportunity Number: HR001121S0008

Catalog of Federal Domestic Assistance Numbers (CFDA): Not Applicable

Dates o Posting Date: October 27, 2020 o Abstract Due Date: November 17, 2020, 12:00 noon (ET) o Proposal Due Date: January 8, 2021, 12:00 noon (ET) o Proposers Day: October 30, 2020

Additional BAA Process/Timelines o Government Abstract Responses ~ November – December 2020 o Proposals are reviewed for BAA Compliance ~ January 2021 o Government conducts Scientific Review Process ~ January – March 2021 o Government sends out notification letters ~ March 2021 o Contracts negotiated & awarded ~ July 2021

Anticipated Individual Awards: DARPA anticipates multiple awards for Technical Areas (TAs) 1 and 2 and a single award for TAs 3 and 4 (No award for TA5, which is listed for information purposes only)

Types of Instruments that May be Awarded: Procurement contracts or Other Transactions

Agency Contacts o Technical POC: Dr. Brian Kettler, Program Manager, DARPA/I2O o BAA Email: INCAS@darpa.mil o BAA Mailing Address:

DARPA/I2O

ATTN: HR001121S0008

675 North Randolph Street Arlington, VA 22203-2114 o I2O Solicitation Website: https://www.darpa.mil/work-with-us/opportunities

HR001121S0008 INCAS 5

PART II: FULL TEXT OF ANNOUNCEMENT

I. Funding Opportunity Description

DARPA is soliciting innovative research proposals in the area of computational techniques and tools that aid analysts in detection and sensemaking of geopolitical online influence campaigns.

Proposed research should develop and demonstrate innovative approaches that enable revolutionary advances in capability. Specifically excluded is research that primarily results in evolutionary improvements to the existing state of practice.

This Broad Agency Announcement (BAA) is being issued, and any resultant selection will be made, using procedures under Federal Acquisition Regulation (FAR) 6.102(d)(2) and 35.016.

Any negotiations and/or awards will use procedures under FAR 15.4. Proposals received as a result of this BAA shall be evaluated in accordance with evaluation criteria specified herein through a scientific review process.

DARPA posts BAAs on the System for Award Management, Contract Opportunities (Beta.Sam.Gov) website (https://beta.sam.gov/) and, when applicable, the Grants.gov website (https://www.grants.gov/). (Grants will not be awarded under this program.)

The following information is for those wishing to respond to this BAA.

A. Introduction and Background

The INfluence Campaign Awareness and Sensemaking (INCAS) program will develop techniques and tools that enable analysts to detect, characterize, and track geopolitical influence campaigns with quantified confidence. The US is engaged with its adversaries in an asymmetric, continual, war of weaponized influence narratives. Adversaries exploit misinformation and true information delivered via influence messaging: blogs, tweets, and other online multimedia content. Analysts require effective tools for continual sensemaking of the vast, noisy, adaptive information environment to identify adversary influence campaigns.

Today, geopolitical influence campaign detection and sensemaking is largely manual and ad hoc.

Analysts use social listening tools to formulate complex keyword queries; track trending keywords, hashtags, and topics; and read hundreds to thousands of documents to identify influence themes. New or “low and slow” campaigns are hard to detect early as their message1 volume may be beneath platform “trending” thresholds and pertinent hashtags may be unknown.

With current tools, it is difficult to connect messages over time and across multiple platforms to track evolving campaigns and to assess confidence in analytic conclusions in a principled manner. Confidence assessment by analysts is ad hoc, manual, subjective, qualitative and susceptible to analyst cognitive biases (e.g., confirmation bias). Analyst reports often cover static time ranges, and static reports quickly become stale.

1 For this document, “message” is used to refer to any kind of online media post (blog post, tweet, news article, etc.).

https://www.grants.gov/

HR001121S0008 INCAS 6

Today, with current tools, analysts must manually sift through a high volume of messages to find those with relevant influence agendas then gauge which ones are gaining traction and with whom. Analysts track population response using digital marketing tools for analyzing audience demographics, interests, and personality. These tools lack explanatory and predictive power for deeper issues of geopolitical influence. Audience analysis is often done using static, demographic segmentation based on online and survey data. This lacks the flexibility, resolution, and timeliness needed for dynamic geopolitical influence campaign detection and sensemaking.

B. Program Description and Scope

INCAS tools will enable analyst-guided influence campaign analysis using automated influence detection (TA1). In contrast to current social media tools, INCAS tools will directly and automatically detect implicit and explicit indicators2 of geopolitical influence in multilingual online messaging to include author’s agenda, concerns, and emotion.

To explain and anticipate population response to influence messaging, INCAS tools will dynamically segment the responding population and identify psychographic3 attributes relevant to geopolitical influence (TA2). Psychographic attributes, such as worldviews, morals, and sacred values, are hypothesized to correlate more strongly with geopolitical influence response than the personality and demographic attributes used for marketing. A person's worldview is the way they see and understand the world, especially regarding issues such as politics, philosophy, and religion.4 Worldviews can include systems of moral and sacred values.5 Psychographic attributes will be extracted using analysis of text and online behavior, and attributes will be correlated with influence indicators in messaging to which the population segment is responsive.

INCAS tools will support analyst-guided linking of influence indicators and population response over time and across multiple platforms to capture dynamic, evolving campaign models (TA3).

Campaign models will combine machine-surfaced influence indicators and messaging and population response with analyst-provided campaign elements, including campaign tactics, objectives, actors, and events. Quantified confidence assessment will enable analysts to mitigate cognitive biases through INCAS automation that curates, elicits, combines, and organizes confidence intervals, evidence, alternative hypotheses, and supporting information. Integrated support for quantified confidence assessment represents another advance to assist analysts in evaluating the validity of conclusions.

INCAS will primarily use publicly available data sources including multilingual, multi-platform social media, online news sources, and online reference data sources. The program will focus primarily on text data. INCAS will not intentionally process text from US persons. If we inadvertently collect US person data, that will be deleted. See Section VI.B.3 for further

2 An influence indicator is a span of text or a classification of that span of text in a message that signifies potential attempts to persuade the reader. An influence indicator may also be the emotion or sentiment associated with specific text.

3 Psychographics describes the values, activities, interests, and opinions of populations or population segments.

American Psychological Association, "Psychographics," https://dictionary.apa.org/psychographics 4 https://www.collinsdictionary.com/us/dictionary/english/world-view 5 e.g., J. Haidt, The righteous mind: Why good people are divided by politics and religion, Vintage, 2012. J. Ginges and S.

Atran, "Sacred values and cultural conflict," Advances in culture and psychology, vol. 4, p. 273-301, 2013.

https://dictionary.apa.org/psychographics https://www.collinsdictionary.com/us/dictionary/english/world-view

HR001121S0008 INCAS 7

information.

INCAS is an applied research and development effort and is thus expected to result in portable, modular tools and technologies that operational users can assess. Towards this end, program performance will be rigorously evaluated on program-wide and technology-specific metrics for current and historical scenarios developed in consultation with operational stakeholders.

Scenarios will be built around specific combinations of non-U.S. populations, topics of geopolitical interest, and online media platforms and sources.

Operational relevance and utility of tools must be demonstrated. To enable stakeholder tool evaluation and, ultimately, transition, INCAS will develop an extensible testbed for evaluation of tools in the INCAS toolkit. Tools for influence messaging detection, population characterization, and analyst sensemaking will be interoperable through a testbed infrastructure and can be deployed individually or in combinations.

While misinformation and disinformation do play a role in influence campaigns, approaches that are focused exclusively on misinformation or disinformation detection (e.g., “fake news”) are not of interest, because influence campaigns can also build narratives around true events and facts.

Proposals focused solely on social network analysis, knowledge graph construction, cyber-forensic analysis (bot and deep fake detection) are considered out of scope. However, responsive proposals may include the integration of existing technologies listed in this paragraph, as long as their inclusion is well-justified.

For INCAS, it is expected that researchers will leverage, refine, extend, or combine state-of-the-art natural language processing (NLP) technologies for purposes of geopolitical influence campaign detection and characterization, and proposals that are mainly about NLP research for general applications are of limited interest. Finally, INCAS will focus on awareness and sensemaking for analysts, and thus tools for mitigating, preempting, or responding to influence operations (e.g., planning or executing messaging campaigns) are considered out of scope.

Sources such as Princeton’s Empirical Studies of Conflict Center document list nearly 100 foreign and domestic influence campaigns in the past 9 years.6 As another illustrative example, consider the ability of adversaries to use influence campaigns to undermine support for the 800 overseas US bases and the significant strategic implications of these campaigns. A specific campaign might target bases in particular country aimed at a target audience of host nation citizens. Messaging to undermine these bases might initially be based on true events such as crime or pollution associated with some bases. These might then be distorted or exaggerated by an adversary: i.e., by impugning sinister motives to the US or the host nation government. Other wedge issues exploited by a campaign might include economic opportunity costs of bases or potential dangers from base activities (e.g., helicopter crashes). Much of the discourse about bases may be organic debates among concerned citizens. But some of the discourse may be pushed by an adversary with the agenda of undermining local support for bases or the US in general.

6 This report uses only open source media reports and research articles. They identified 60 additional campaigns that only partially met their criteria. https://esoc.princeton.edu/publications/trends-online-influence-efforts https://esoc.princeton.edu/publications/trends-online-influence-efforts

HR001121S0008 INCAS 8

For illustrative purposes, the following is a notional workflow for a user of INCAS tools to address the analytic tasks and questions in Figure 1 for an influence campaign example, where an adversarial actor may be promoting narratives to undermine US bases in a country. Figure 1 depicts a notional workflow example, and proposers are encouraged to outline alternative approaches. INCAS tools should be able to support a variety of workflows with different interactions among the human analyst and INCAS automated capabilities. Figure 1 is meant to give proposers a sense of how INCAS components might work together with one another and with users and should not be construed as guidance or constraints for proposers. Proposers should, however, note that users are an essential part of the sensemaking process, and proposals should address how user interaction and feedback will be exploited.

Figure 1. Notional Analysts Task Activities (outer ring) and Campaign Model Fragment (inside ring).

Within the notional workflow, a user starts by opening the INCAS Human-Machine Interface (HMI) (TA3) to begin characterizing a new campaign. The user specifies a non-US target population for the host nation of a US base and specific entities of interest (e.g., the bases involved) or topic of interest (e.g., pollution, crime). TA1 algorithms identify related influence messaging in relevant social media and other data feeds by detecting influence indicators (e.g.

agenda, concerns, and emotion). For the anti-base example, agendas could include the downsizing, relocation, or removal of US bases. Identified concerns might include issues such as environmental pollution, crime, economic benefits (or costs), and security for the local population. Emotions might span the gamut from outrage associated with US bases to more positive emotions about the role these bases play in regional security. TA1 algorithms surface potentially relevant influence messaging to TA2.

TA2 algorithms then identify the population segments that are responding online. For example, TA2 algorithms might identify several host nation population segments responding to influence messaging about the bases. Using demographic and psychographic analysis of these messages and their metadata, several groups might be apparent. For example, one segment responding more favorably to US bases (via their messages and online actions) might be predominantly

HR001121S0008 INCAS 9

older citizens concerned about national security and generally pro-US. Another segment might be predominantly younger citizens who are responding negatively to US bases in much higher volume and much more negative sentiment or emotions. Upon demographic and psychographic analysis, this segment is found to be pro-environment and anti-US. These segments are surfaced to the user (along with confidence assessment information) to help a user understand which influence messaging is gaining traction with whom in the target population.

Taking advantage of previous analyses of influence indicators, TA2 algorithms begin to correlate influence indicators with the demographic and psychographic attributes of responding populations. These correlations can be exploited to explain or anticipate population segment response and answer questions such as “What else are these population segments responding to?” and “What might they respond to in the future?”. Using indicators and attributes surfaced by TA1 and TA2 algorithms along with their own hypotheses, users can populate models representing specific campaigns via the TA3 HMI. For example, a user may decide to link recent influence indicators and responding population segments to hypothesized influence campaign elements (e.g., campaign activities and their associated tactics and potential actors). Common influence indicators might be spotted across other anti-US base campaigns. TA3 analytics might begin to triage campaigns, ranking them by potential threat. As they flesh out campaign models building on machine-surfaced information, users can view confidence assessment scores for a campaign model, aggregated from confidence assessment information surfaced by the TA1-TA3 analytics.

In this notional workflow, newly hypothesized campaign elements from a user may trigger TA1 or TA2 to find additional relevant influence indicators and responding population segments.

Behind the scenes, TA4 provisions social media feeds and other data to all components and provides efficient access to campaign model data. As new data comes into INCAS through TA4, TA1-TA3 capabilities may be invoked to update their output and prior campaign models.

C. Program Structure

INCAS has five technical areas (TAs) as shown in Figure 2.

TA1, Influence Indicator Detection, will develop techniques to identify influence indicators in online messaging. Multiple awards are expected.

TA2, Population Response Characterization, will develop techniques to segment the responding population to a set of influence messages, characterize each segment using psychographic and demographic attributes, and identify correlations among these attributes, influence indicators, and response. Multiple awards are expected.

TA3, Influence Campaign Modeling, will develop techniques for analyst-machine sensemaking of influence campaigns including aiding analysts in assessing confidence in campaign models. A single award is expected.

TA4, Data and Testbed Development, will develop the infrastructure to provide social media messaging and other data feeds from online sources to all TAs. TA4 will collect and persist social media and other online data and implement low-level data analytics.

TA4 will also develop application programming interfaces (APIs) so performers in other TAs can access data and post the output of their algorithms in TA4’s infrastructure. In addition TA4 will develop the testbed infrastructure for program use. A single award is expected.

TA5, Program Evaluation, will design and conduct technology evaluations (including

HR001121S0008 INCAS 10

metrics and scenario definition), develop ground truth evaluation data for program scenarios, manage a Program Subject Matter Experts (SMEs) group, coordinate with an Operational Stakeholders Group, and coordinate PI meeting events. A single award is expected here for a University Affiliated Research Center (UARC) or Federally Funded Research and Development Center (FFRDC). TA5 will not be competed as part of this BAA, but is included here for informational purposes as all TAs are expected to interact with and support TA5 for program evaluation.

Figure 2. INCAS Technical Areas

INCAS is divided into three phases, totaling 48 months. The first two phases will be 18 months each, and the third phase will be 12 months.

Phase 1 (18 months) focuses on standing up initial capability for TA1 influence indicator detection, TA2 population segmentation and response analysis, and TA3 human-machine interfaces (HMI) paired with a data model for campaign modeling. A testbed infrastructure will be developed to include initial TA4 tools and services for data provisioning from multi-source data feeds and persistence of data for all TAs. This phase will exploit historical data for two related scenarios using English and an additional language to be determined by DARPA based on scenario selection. TA5 will work with all performers, DARPA, subject matter experts, and operational stakeholders to identify scenarios.

Phase 2 (18 months) focuses on standing up additional and improved capabilities for influence indicator detection; population segmentation and response analysis; and improvements to the HMI, the campaign data model, and campaign analytics. Additionally, INCAS tools will be hosted in a cloud-based testbed on the Secure Unclassified Network (SUNet) (or similar) to support operational partner tool evaluation. This phase will exploit contemporary data for two scenarios using English and an additional language for Phase 2.

Phase 3 (12 months) will focus on a capstone experiment with an operational partner, with a scenario, data, influence indicators, and psychographic attributes specifically tailored to the operational partner. This may include integration with extant operational partners’ tools.

DARPA is not anticipating structured down selects across the phases of the program, although performers may be discontinued if the Government deems the research approach is unlikely to

HR001121S0008 INCAS 11

yield adequate results.

D. Technical Areas Figure 3 shows INCAS technical areas and a high level data flow among them.

Figure 3. Data flow among INCAS technical area (TA) components

The arrows in Figure 3 show the functional data flow among INCAS TAs. External data feeds (social media, news) are ingested by TA4. TA4 will also persist campaign model data and data from social media feeds. Other TAs will access and store data in TA4 via its APIs. TA3 will provide the human machine interfaces (HMI) through which analysts will interact with INCAS tools.

Proposals may be submitted to all TAs, except TA5. Proposers interested in multiple TAs must submit separate proposals for each TA. Proposers addressing multiple TAs must describe expected research synergies and specific efficiencies and savings to the Government that would result if that proposer was selected for multiple TAs.

There are no inherent conflicts of interest between any of the competed TAs, and awards may be made to the same proposer under multiple TAs. If a proposer proposes to both TA4 and either TA1 or TA2, that proposer must address in the proposal how TA4, as the data provider and testbed developer, will ensure a “level playing field” to all other TA1 or TA2 performers.

Within a TA, teaming is encouraged but not required. Teams that can bring multidisciplinary insights and techniques to bear from computer science, artificial intelligence, social science, marketing, strategic communications, or other relevant disciplines are of particular interest.

There are multiple points of essential collaboration among TAs described in Section I.F, and DARPA expects all performers to collaborate effectively. Proposers should read the descriptions

HR001121S0008 INCAS 12

of all TAs and the information about program expectations and timelines in the Program Structure section to ensure a full understanding of the program context, structure, and anticipated relationships required among performers. To facilitate the open exchange of information, all program performers will have Associate Contractor Agreement (ACA) language included in their award. See Section VIII for further information. All performers should be aware of Human Subjects Research (HSR) requirements (see Section IV.B).

TA1: Influence Indicator Detection

TA1 performers will focus on the detection of influence indicators in online messaging (social media, news, etc.). Performers will be required to address the following types of influence indicators: agenda, concerns, and emotion and to propose two additional types of influence indicator7 in each of Phase 1 and Phase 2.

Agenda is what the text/author is trying to get an audience to believe or do. For the anti-base example above, the agendas in the messaging might include that the US is harming (“poisoning”) a host nation (via pollution) and that US bases should be removed. Agenda will help analysts identify influence messaging, understand its purpose, and determine its role in potential influence campaigns. Agendas may be explicit or implicit.

Concerns are deeper, often wedge issues (e.g., environmental pollution) or underlying moral or sacred values (e.g., purity/sanctity) that the message speaks to. Adversaries aim at wedge issues to widen social seams. Moral and sacred values play a large role in human behavior.8

Emotion and sentiment are expressed explicitly through emotional language or implicitly through tone. Emotion is frequently the major driver of peoples’ response to messages.

Strong emotion or sentiment in the text can also be an indicator of agenda and concerns as those tend to be “hot buttons”.9 Emotion here refers to that expressed by the author in the text.

While these examples are content-based indicators, other types of indicators may be possible that exploit metadata or other structural and temporal characteristics of messaging, or the messaging platform itself. Indicators must be meaningful to analysts to aid them in building models of campaigns.

For the base indicator types (agenda, concerns, and emotion) and additional indicator types, proposals must (1) define the indicator type and how it will be represented (e.g., text string, a canonical value from an ontology, etc.); (2) describe how it will be extracted from messaging or other data; (3) provide examples for the indicator type; (4) identify the theoretical or empirical basis for the relevance of the indicator type to geopolitical influence campaigns; and (5) describe

7 An indicator type is a category such as “agenda” or “emotion”. An indicator is a specific instance of an indicator type: e.g., the agenda that the US should remove its bases. For emotion, an indicator might be the presence of outrage or some other emotion.

8 e.g., J. Haidt, The righteous mind: Why good people are divided by politics and religion, Vintage, 2012. J. Ginges and S. Atran, "Sacred values and cultural conflict," Advances in culture and psychology, vol. 4, p. 273-301, 2013.

9 e.g., W. J. Brady, J. A. Wills, J. T. Jost, J. A. Tucker and J. J. V. Bavel, "Emotion shapes the diffusion of moralized content in social networks," Proceedings of the National Academy of Sciences, vol. 114, no. 28, pp. 7313-7318, 2017.

HR001121S0008 INCAS 13

the expected utility and generalizability of the indicator type and the extraction approach across topics of discourse, cultures, languages, and social media over time. Proposers should identify at least two languages, in addition to English, that their approach could feasibly address.

Influence indicator detection and extraction for content-based indicator types is expected to exploit multiple extant and emerging natural language processing (NLP) techniques individually and in novel combinations. Proposals must describe the applicability of specific techniques to specific influence indicator types, the current state-of-the-art (including quantitative performance), expected research to extend and adapt these techniques for detecting indicators of geopolitical influence, and the basis for confidence in the approach. Proposals should describe how these techniques will be used in novel combinations; their scalability with regard to message length and high message volume; and their application to multilingual and multicultural data.

Proposals that include supervised learning approaches must describe the required type and volume of training data (e.g., labeled corpora in multiple languages) and its availability.

Proposals are expected to identify and provision quality training data for their algorithms to supplement the scenario-specific training data provided by TA4.

Texts in social media may be fragmentary, making influence indicator detection difficult.

Additionally, influence indicators may be implicit, which may necessitate the need to augment messaging with contextual text and other information: e.g., surrounding messages in a conversation, prior messaging by the same author, or vetted news stories on the same topic.

Proposals must address how they expect to effectively and efficiently identify, access, and exploit such contextual information (or obviate the need for it) and identify specific publicly available sources where possible.

TA1 algorithms must be able to exploit campaign model elements (e.g., entities of interest) and user feedback from TA3 (provided via TA4) as context for indicator detection. Feedback includes, but is not limited, to user correction of extracted indicators and identification and labeling of missed indicators in influence messaging. Early in Phase 1, TA1 will work with TA3 on specifying the kinds of feedback to be captured from analysts and the format of how this feedback will be provided to TA1.

TA1 algorithms are expected to provide confidence assessment information along with indicator output. Such information may include confidence intervals, alternative ranked hypotheses, and supporting evidence. Proposals must describe what information they intend to produce and its format, reliability, and expected utility in support of campaign model confidence assessment

(TA3).

In Phase 1, performers will address the core indicator types of agenda, concerns, and emotion and two additional indicator types. They will also address the challenge of fragmentary texts. In Phase 2, performers will refine Phase 1 indicator detection algorithms to improve accuracy and add two other indicator types. In Phase 3, performers will refine all indicator types to improve accuracy and generalizability. Phase 1 will focus on English and a second language. Phase 2 will add an additional language. The scenarios and languages for each phase will be identified by DARPA at each phase kickoff. See Section I.E for schedule details.

TA1 proposals must identify how they will achieve the TA1 program metrics listed in Section I.F

HR001121S0008 INCAS 14

(Table 1). TA1 program metrics include accuracy of detection of influence messages, and, given such a message, accuracy of extraction of influence indicator detection. TA1 performers must propose additional technical metrics appropriate to their algorithms that will be used to demonstrate regular technical progress at a more detailed level.

TA1 performers are expected to work closely with performers in the other TAs, as described above and in Section I.F. Required TA1 deliverables are listed in Section I.F.

TA2: Population Response Characterization

Technology developed under TA2 will (1) segment the responding population at sufficient granularity based on response to influence messaging (e.g., responder sentiment or emotion10, actions, volume); (2) extract demographic and psychographic attributes and behavioral patterns from online data; and (3) correlate influence indicators with population attributes to explain and anticipate a population segment’s response to influence messaging. TA2 proposals must include technical approaches to meet these three objectives.

Proposers must extract two (2) demographic and two (2) psychographic attributes in Phase 1. In Phase 2 they must extract two (2) additional demographic and two (2) additional psychographic attributes. TA2 technology will ingest content, metadata and labeled influence indicators for influence messaging detected by TA1 technology and provided via TA4’s APIs. Behavioral data may include type and volume of actions (likes, shares, etc.). TA2 can use additional publicly available data to aid in characterizing (sub)populations. Technical approaches for population segmentation must be data driven. TA4 will provide data relevant to the target population that TA2 can use for segmentation and psychographic attribute extraction. Proposals must identify required data, expected granularity, and the scalability of their segmentation approach.

Over the past 10 years, growth in users’ online messaging has enabled audience demographic and psychographic attributes to be computed more easily for marketing purposes. Attributes include personality, interests, values, attitudes, etc. INCAS will go beyond the commercial focus on demographics and personality by developing new methods to extract psychographic attributes of population segments that are relevant to geopolitical influence campaigns, such as worldviews. Examples of worldviews might include whether someone (or some group) is globalist or nationalist, individualist or conformist, or has a hierarchical or egalitarian perspective.11 For example, hierarchical and individualist groups may be more likely to view immigration as a risk and hence more receptive to influence campaigns that use immigration concerns to sway an audience (e.g., Brexit). Worldviews also include moral and sacred values, which have begun to be addressed in the research literature.12

TA2 proposals must identify two demographic attributes and two psychographic attributes each for Phase 1 and Phase 2. For each attribute, proposals should describe its (1) relevance to geopolitical influence campaigns; (2) theoretical support from scientific/marketing literature; (3)

10 Emotion here is expressed by an individual in response to messaging. TA1 focuses on characterizing emotion expressed by the author of influence messaging.

11 e.g., C. Graves and J. Puleston, "The Real Why and the Hidden Who," in ESOMAR, 2018.

12 e.g., K. Kalimeri, M. G. Beiró, M. Delfino, R. Raleigh and C. Cattuto, "Predicting demographics, moral foundations, and human values from digital behaviours," Computers in human behavior, vol. 92, pp. 428-445, 2019.

HR001121S0008 INCAS 15

expected utility; (4) generality across different populations, cultures, languages and over time;

and (5) approach and feasibility of extraction from publicly available online data. Proposers should identify at least two languages, in addition to English, that their approach could feasibly address. If approaches require labeled training data, the proposal must make clear whether that data is already available. If the data is not already available, proposals must describe the resources required to obtain it, and the impact on timeline. For approaches using survey data to develop training data, proposals must also describe how potential sampling, self-report, and other cognitive biases will be mitigated.

As an example of an alternative approach to explicit and direct extraction of psychographic attributes, online activity data could be used to discover patterns that are directly predictive of behavior. This approach is now being used in e-commerce by product recommender systems, though such predictive patterns are harder for analysts to interpret. Proposals which include such techniques must address how to make output more interpretable by analysts.

TA2 technology must correlate a growing collection of influence indicators with population responses. For example, influence messaging that mentions immigration (a concern) might correlate positively or negatively with response by a subpopulation with a more globalist worldview. Proposals must include specific techniques to determine such correlations and update them over time as needed.

TA2 algorithms must be able to use campaign model elements (e.g., entities of interest) and user feedback from TA3 (provided via TA4) as context for segmentation and attribute extraction. This feedback includes, but is not limited to, user correction of segmentation and attributes and identification and labeling of missed attributes for population segments. Early in Phase 1, TA2 will work with TA3 on specifying the kinds of feedback to be captured from analysts and the format of how this feedback will be provided to TA2.

TA2 algorithms are expected to provide confidence assessment information along with population segmentation, attributes, and correlations. This confidence information might include, but is not limited to, confidence intervals, alternative ranked hypotheses, and supporting evidence. Proposals must describe what information they intend to produce and its format, reliability, and expected utility in support of campaign model confidence assessment (TA3).

In Phase 1, performers will extract two demographic and two psychographic attributes, and will focus on English and one additional language. Initial population response segmentation capabilities will be developed in Phase 1 as well. In Phase 2, performers will refine Phase 1 attributes extraction algorithms to improve accuracy and add two additional demographic and two additional psychographic attributes. Phase 2 will also focus on refining population segmentation capabilities developed in Phase 1, correlating population response with influence indicators, and introducing another language. In Phase 3, performers will refine all attributes to improve accuracy for the operational scenario. The scenarios and languages for each phase will be identified by DARPA at each phase kickoff. See Section I.E for schedule details.

TA2 proposals must identify how they will achieve the TA2 program metrics listed in Section I.F (Table 1). TA2 program metrics include the extraction accuracy for the demographic and psychographic attributes, and the accuracy of the correlations among influence indicators and population segment attributes. TA2 performers must propose additional technical metrics

HR001121S0008 INCAS 16

appropriate to their algorithms that will demonstrate regular technical progress at a more detailed level.

TA2 performers are expected to work closely with performers in the other TAs, as described above and in Section I.F. Required TA2 deliverables are listed in Section I.F.

TA3: Influence Campaign Modeling

A single TA3 performer will develop analyst-guided influence campaign modeling tools that accelerate analysts’ ability to detect and link influence indicators and messaging with population response over time. Section I.B presents an illustrative example of one potential analytic workflow that can be used to help analysts answer questions such as those shown in Figure 1.

TA3 proposals must specifically address how analysts will work collaboratively with one another and with automated capabilities to build and understand campaign models.

TA3 has four major tasks: (1) design a campaign data model; (2) provide rich human machine interfaces (HMI) for analysts and operators; (3) provide analytics to populate, analyze, and compare campaign models; and (4) provide techniques (and HMI) that aid users in confidence assessment of a campaign model.

The campaign data model is envisioned as a data schema and associated taxonomies or ontologies that can be used to represent campaign elements including, but not limited to, influence indicators; population segments characterized by demographic and psychographic attributes; actors orchestrating a campaign and their objectives; target populations; tactics; and events. The data model must also support metadata capturing, data pedigree, data provenance, multiple hypotheses, supporting evidence, and other confidence assessment information from TA1 and TA2. Analysts and TA1-TA3 analytics will jointly populate campaign models: e.g., the notional workflow in Section 1.B. TA4 is responsible for implementing the campaign model and thus will work closely with TA3 on its design.

TA3 will develop a scalable, web-based HMI that helps visualize, explore, edit, search, and compare campaign models, represented using the campaign data model. The HMI must support drilldown to underlying social media and other data, and provide mechanisms for more free-form exploration of this data. The HMI must also capture feedback from analysts on output of TA1- TA3 analytics. The HMI will access data persisted by TA4 through TA4’s APIs. The HMI must support multiple concurrent users and user-tailored views.

The TA3 performer will work closely with subject matter experts (SMEs) and operational users to iteratively refine the HMI. TA5 will coordinate access to some SMEs and operational users, though proposers may wish to supplement this with SME participation on their teams. Proposals must address how TA3 plans to work with these users to collect and incorporate their feedback.

Thus TA3 performers must include some essential technical personnel cleared at least to the Secret level. Proposers should identify cleared personnel in their proposals.

TA3 will provide supporting analytics to aid users in deriving insight across one or more campaign models. For example, these might include campaign model clustering and triage by estimated threat or other criteria. Additional analytics could help filter out benign influence messaging such as product advertising. TA3 proposals must describe innovative analytics that

HR001121S0008 INCAS 17

will support analysts, and how these analytics will specifically aid in campaign understanding, how they will be validated using ground truth data; and how they will scale to large campaign models and a growing collection of campaign models.

TA3 will prototype integrated validation tools for analysts to assess confidence in campaign models in consistent, principled ways. TA3 will analyze, aggregate, and surface information to analysts to enable them to make more principled confidence assessment of evolving campaign models. Minimally, TA3 could exploit and present available confidence intervals, multiple sources for validation, and scores associated with alternative hypotheses from TA1 and TA2 analytics. Beyond these notional examples, proposals must describe innovative approaches to capturing, eliciting, aggregating, and presenting confidence assessment information and make clear any expectations for information provided by TA1 and TA2. Confidence assessment inputs and outputs will need to be persisted (by TA4) via the campaign data model.

In Phase 1, the TA3 performer will develop the initial HMI capabilities and supporting analytics for the campaign data model and confidence assessment design. The TA3 performer is expected to deliver a description of the campaign data model at the Preliminary and Critical Design Reviews (see Section I.E) and complete its initial design by the first Principal Investigator (PI) meeting (6 months after kickoff). The campaign data model will be refined roughly quarterly as needed. In Phase 2, performers will refine the HMI and analytics based on user feedback and implement confidence assessment mechanisms. In Phase 3, the performer will continue to refine the HMI and confidence assessment methodologies in the context of the planned operational scenario. See Section I.E for schedule details.

TA3 proposals must identify how they will achieve the program metrics (see Table 1 in Section I.F for details). TA3 proposals must list additional technical metrics appropriate to their algorithms that will be used to demonstrate regular technical progress at a more detailed level.

TA3 performers are expected to work closely with performers in the other TAs, as described above and in Section I.F. Required TA3 deliverables are listed in Section I.F.

TA4: Data Provisioning and Testbed Development

TA4 will provision data for development and evaluation and develop a testbed infrastructure that enables TA1-TA3 tools to be easily assimilated by operational users.

TA4 is responsible for ingest, storage, and provisioning of data from multimedia, multilingual, multi-platform feeds (social media messaging, news stories, etc.) for other TAs for the INCAS evaluation scenarios. TA4 will also persist and provide access to data from TA1-TA3, to include influence indicators, population response characterization, and campaign model elements/feedback provided by TA3 via analysts or analytics.

In addition to ingesting and storing data, TA4 will be responsible for data enrichment to include metadata analysis; incorporation of cyber forensic indicators such as bot detection (from extant detector algorithms); and potentially other enrichments to be determined. All these enrichments should leverage primarily extant tools (open source or Commercial Off The Shelf (COTS)/Government Off The Shelf (GOTS)).

HR001121S0008 INCAS 18

Although text media is the focus, in Phases 2 and 3 INCAS will investigate static images such as memes and ads and convert these to textual representations leveraging image recognition, meme libraries, optical character recognition (OCR) via extant open source or COTS/GOTS tools.

Image analysis for video is considered to be out of scope.

TA4 proposals must describe this data ingest and analytic pipeline in detail and how well it will scale to higher data volumes and velocities. TA4 proposals must describe techniques for ingest of batch data for historical and current scenarios in Phase 1 and streaming data in Phase 2. As new data comes into the system through TA4, TA1-TA3 capabilities may be invoked via various workflows to update output and prior campaign models based on this new data. TA4 proposals should describe candidate workflows.

Proposals must describe how TA4 will work closely with all other TA performers to design APIs for data storage/access and identify common, low-level analytics (e.g., named entity recognition, etc.) and with TA3 to design the campaign data model. TA4 must also work with other TA performers to identify data relevant to INCAS evaluation scenarios and ingest those sources.

TA1 and TA2 performers are expected to provide their own training data for supervised learning algorithms apart from that provided for the INCAS scenarios by TA5.

A key challenge for TA4 is to address the scalability of mining and processing high volume, high velocity social media and other data feeds. TA4 proposals must address techniques for filtering data feeds to the most relevant data for downstream analysis such as messaging involving members of the target population of interest. TA4 should assume that most messages are not geotagged.

The data provisioning approach must be flexible in order to meet the demands of changing information requirements from analysts in TA3 algorithm development in TA1 and TA2 (see Figure 3) and multiple INCAS evaluation scenarios. TA4 is expected to procure access to publicly available data sources that support the INCAS evaluation scenarios. This includes social media platforms including, but not limited to, YouTube, Twitter, and regional feeds and sources.

TA4 proposals must describe two realistic influence campaign examples and use the examples to motivate the technical approach. Examples may be derived from real world influence campaigns or hypothetical extrapolations of real world events. The descriptions must include estimates of the amount of data required and the likely data sources. Data collection strategies for each data source must be explained. The purpose of these descriptions is to demonstrate the ability of the TA4 team to think through the variable data collection requirements in light of the structure of the program and its goals. The descriptions do not need to be a complete picture of an engineered data pipeline, but must include enough information to provide a coherent rationale.

TA4 proposals should use these influence campaigns to estimate cost for data requirements. For scoping purposes, these should be on the scale of detecting a campaign directed against a specific target non-U.S. population on a set of related issues (e.g., a campaign targeted at a host nation to promote anti-base sentiment with the goal of reducing or eliminating US bases). Multilingual text data collection must be addressed.

TA4 is expected to maintain appropriate controls with respect to collection of data on US persons, privacy, personally identifiable information (PII), etc. Proposers must detail how they

HR001121S0008 INCAS 19

plan to implement these controls, with specific attention to identifying and removing PII when necessary. TA4 performers will be responsible for providing relevant data policies and privacy training to all performers. Please see Section VI.B.3 for more information.

TA4 is expected to develop a cloud-based testbed infrastructure that will host INCAS tools and components from TA1, TA2, and TA3 and TA4 services for data ingest and persistence. This must support access by INCAS performers, government, and other stakeholders. TA4 is responsible for integrating algorithms and tools into this infrastructure, working with performers on other TAs. The infrastructure must support flexible workflows, handle dynamic updates from data feeds, instrumentation for performance analysis, and security. TA4 proposals must describe the proposed testbed architecture.

Initially, TA4 will host this testbed using its own infrastructure or a government cloud. In Phase 1, TA1-TA3 performers will primarily host their own algorithmic capabilities, and the TA4 testbed will host only TA4 capabilities (e.g., data ingest, persistence, and low-level data analytics). TA4 will host scenario data and the output of TA1-TA3. In Phases 2-3, the TA4 performer will support integration of TA1-TA3 capabilities into the testbed via containers and related technologies. TA4 may also support insertion of operational stakeholder tools into the testbed.

In Phase 2, it is expected that a demonstration and experimentation testbed for access by operational stakeholders will be stood up on an unclassified network such as the government’s Secure Unclassified Network (SUNet) or similar.