HR001117S0026.pdf

Broad Agency Announcement Active Interpretation of Disparate Alternatives (AIDA)

HR001117S0026

April 3, 2017

Defense Advanced Research Projects Agency Information Innovation Office 675 North Randolph Street Arlington, VA 22203-2114

HR001117S0026 ACTIVE INTERPRETATION OF DISPARATE ALTERNATIVES (AIDA) 2

Table of Contents

I. Funding Opportunity Description

A. Program Overview

B. Concept of Operations

C. Program Structure

D. Program Evaluation

E. Schedule/Milestones

F. Deliverables

G. Government-furnished Property/Equipment/Information

H. Intellectual Property

I. Security Clearance Requirements

II. Award Information

A. Awards

B. Fundamental Research

III. Eligibility Information

A. Eligible Applicants

B. Organizational Conflicts of Interest

C. Cost Sharing/Matching

D. Other Eligibility Requirements

IV. Application and Submission Information

A. Address to Request Application Package

B. Content and Form of Application Submission

C. Submission Date and Time

D. Funding Restrictions

E. Other Submission Requirements

V. Application Review Information

A. Evaluation Criteria

B. Review and Selection Process

VI. Award Administration Information

A. Selection Notices

B. Administrative and National Policy Requirements

C. Reporting

VII. Agency Contacts

VIII. Other Information

HR001117S0026 ACTIVE INTERPRETATION OF DISPARATE ALTERNATIVES (AIDA) 3

A. Frequently Asked Questions (FAQs)

B. Proposers Day

C. Submission Checklist

HR001117S0026 ACTIVE INTERPRETATION OF DISPARATE ALTERNATIVES (AIDA) 4

PART I: OVERVIEW INFORMATION

Federal Agency Name: Defense Advanced Research Projects Agency (DARPA), Information Innovation Office (I2O)

Funding Opportunity Title: Active Interpretation of Disparate Alternatives (AIDA)

Announcement Type: Initial Announcement

Funding Opportunity Number: HR001117S0026

Catalog of Federal Domestic Assistance Numbers (CFDA): 12.910 Research and Technology Development

Dates o Posting Date: April 3, 2017 o Proposers Day: April 7, 2017 o Abstract Due Date: April 21, 2017, 12:00 noon (ET) o Proposal Due Date: June 6, 2017, 12:00 noon (ET) o BAA Closing Date: June 6, 2017, 12:00 noon (ET)

Anticipated Individual Awards per Technical Area (TA): Multiple awards are anticipated for TA1, TA2 and TA3. A single award is anticipated for Technical Area 4, and a single award is anticipated for TA5.

Technical Areas (TAs): There are five TAs under this BAA. Proposers should note that a proposal may address more than one TA.

Types of Instruments that May be Awarded: Procurement contracts, cooperative agreements or Other Transactions

Agency Contacts o Technical POC: Boyan Onyshkevych, Program Manager, DARPA/I2O o BAA Email: AIDA@darpa.mil o BAA Mailing Address:

DARPA/I2O

ATTN: HR001117S0026

675 North Randolph Street Arlington, VA 22203-2114 o I2O Solicitation Website: http://www.darpa.mil/work-with-us/opportunities mailto:AIDA@darpa.mil http://www.darpa.mil/work-with-us/opportunities

HR001117S0026 ACTIVE INTERPRETATION OF DISPARATE ALTERNATIVES (AIDA) 5

PART II: FULL TEXT OF ANNOUNCEMENT

I. Funding Opportunity Description

DARPA is soliciting innovative research proposals in the area of creating a multi-hypothesis semantic engine that generates explicit alternative interpretations of events, situations, or trends from a variety of unstructured sources, by mapping structured knowledge into a common semantic space, for use in an environment where there are noisy, conflicting, and potentially deceptive information. Proposed research should investigate innovative approaches that enable revolutionary advances in science, devices, or systems. Specifically excluded is research that primarily results in evolutionary improvements to the existing state of practice.

This Broad Agency Announcement (BAA) is being issued, and any resultant selection will be made, using procedures under Federal Acquisition Regulation (FAR) 6.102(d)(2) and 35.016.

Any negotiations and/or awards will use procedures under FAR 15.4 (or 32 CFR § 200.203 for grants and cooperative agreements). Proposals received as a result of this BAA shall be evaluated in accordance with evaluation criteria specified herein through a scientific review process.

DARPA BAAs are posted on the Federal Business Opportunities (FBO) website (https://www.fbo.gov/) and the Grants.gov website (http://www.grants.gov/).

The following information is for those wishing to respond to this BAA.

A. Program Overview

The United States Government has an interest in developing and maintaining a strategic understanding of events, situations, and trends around the world, in a variety of domains. The information used in developing this understanding comes from many disparate sources, in a variety of genres, and data types, and as a mixture of structured and unstructured data.

Unstructured data can include text or speech in English and a variety of other languages, as well as images, videos, and other sensor information. Even structured sources can vary in the expressiveness, semantics, and specificity of their representations. Moreover, analysis is complicated by the need to overcome the noisy, conflicting, and potentially intentionally deceptive nature of the data.

It is a challenge for those who strive to achieve and maintain an understanding of these events, situations, and trends that information from each medium is often analyzed independently, without the context provided by information from other media. Often, each independent analysis results in only one interpretation, with alternatives being eliminated due to lack of evidence, even in the absence of contradictory evidence. When these independent, impoverished analyses are combined, generally late in the analysis process, the result can be a single apparent consensus view that does not reflect a true consensus.

The goal of Active Interpretation of Disparate Alternatives (AIDA) is to develop a multi-hypothesis semantic engine that generates explicit alternative interpretations of events, situations, and trends from a variety of unstructured sources, for use in noisy, conflicting, and potentially deceptive information environments. This engine must be capable of mapping knowledge https://www.fbo.gov/ http://www.grants.gov/

HR001117S0026 ACTIVE INTERPRETATION OF DISPARATE ALTERNATIVES (AIDA) 6

elements automatically derived from multiple media sources into a common semantic representation, aggregating information derived from those sources, and generating and exploring multiple hypotheses about the events, situations, and trends of interest. This engine must establish confidence measures for the derived knowledge and hypotheses, based on the accuracy of the analysis and the coherence of the semantic representation of each hypothesis.

This engine must also be capable of utilizing knowledge in the common semantic representation and the generated hypotheses as alternate contexts for the media analysis algorithms by altering their models or prior probabilities to enhance accuracy and resolve ambiguities in line with expectations from the context. In addition, the engine must be able to communicate with its user to reveal the generated hypotheses and to allow the user to alter the hypotheses or to suggest new ones.

B. Concept of Operations

A user who needs to understand an event, situation, or trend will use AIDA to analyze relevant multimedia information streams. These information streams are expected to be partially filtered, that is, to be fairly rich with items relevant to the event, situation, or trend of interest but with a significant number of irrelevant items in the stream.

The AIDA prototype will be expected to operate in a constantly streaming data environment, where incoming information items (e.g., individual documents or images) will be handled one at a time, and will not be available to be revisited in their raw form, only in featurized form.

Each incoming information item will be analyzed, resulting in a set of structured representations about events, sub-events or actions, entities, relations, locations, time, and sentiments (referred to in this BAA as “knowledge elements”) that are observable in that information item. These knowledge elements will be expressed in the AIDA common semantic representation formalism.

Next, the knowledge elements derived from each information item will be linked, co-referenced, and disambiguated relative to the aggregate of knowledge elements from all previous information items. The aggregated knowledge elements will be referred to in this BAA as a knowledge base. As knowledge elements are accumulated into the knowledge base, duplicate knowledge elements will be combined, while distinct knowledge elements will be preserved with all of their properties. In the resulting knowledge base, only one knowledge element will reference each real-world entity, event, etc. (Alternate interpretations of a real-world entity, event, etc. will be represented in the knowledge base in a manner to be determined by the relevant performers.)

Either continuously or at set intervals, the AIDA prototype will generate all plausibly relevant hypotheses about the real-world situation, event, or trend that are supported by some interpretation of the input information items. These hypotheses must be differentiated by one or more significant knowledge elements, but may share subsets of knowledge elements.

At any time during AIDA processing, both the whole knowledge base and the individual hypotheses will be available to the user for inspection and editing. The user will be able to adjust or downgrade the confidence of hypotheses postulated by the AIDA prototype or even add completely new hypotheses. In addition, the knowledge base and the hypotheses will be made available to the media analysis algorithms as alternate contexts, to be used as feedback to help disambiguate or refine their analysis in line with expectations from the context. Based on AIDA

HR001117S0026 ACTIVE INTERPRETATION OF DISPARATE ALTERNATIVES (AIDA) 7

output, the user will be able to produce reports or move resolved knowledge into a persistent enterprise knowledge base.

As a result of the processing carried out by the AIDA system, the user will have a complete, comprehensive, and nuanced view of an event, situation, or trend of interest. Knowledge derived from multiple media sources will be accessible to both users and automated analysis tools in a form that will enable them to find patterns, conflicts, or deception attempts that were previously imperceptible. This new access to and organization of knowledge will enable improved downstream analysis and contingency planning.

C. Program Structure

It is anticipated that the AIDA program will consist of three 18-month phases. The structure of the AIDA program will include the following technical areas (TAs), which are described in detail below.

TA1 - Semantic Mapping and Feedback TA2 - Common Semantic Representation TA3 - Multiple Hypotheses TA4 - Integration and Multiple Interpretations TA5 - Data

Figure 1: Arrangement of AIDA technical areas

A proposal may address any single technical area or combination of TA1, TA2, and TA3.

Proposals addressing TA4 or TA5 should not be combined with any other TA. Performers selected for TA1, TA2, TA3, and/or TA4 may not perform in TA5, and vice versa. Any proposer submitting to more than one technical area (TA1-TA3) should submit a single proposal for all the TA’s proposed, but they must ensure that the budget is well delineated so that partial awards are possible. Proposers may also submit a proposal addressing only part of TA1, TA2, or TA3

HR001117S0026 ACTIVE INTERPRETATION OF DISPARATE ALTERNATIVES (AIDA) 8

instead of addressing the whole TA, but it is subject to limitations listed in the descriptions of each TA below. In such cases where a proposer is addressing only part of a TA (TA1, TA2 and TA3), the proposal must state clearly how the proposed work will contribute to the entire TA or the entire program. Performers addressing only part of a TA will need to show value to the program in order to be considered for continued funding after Phase 1.

The program evaluations will be conducted by the National Institute of Standards and Technology (NIST) and this BAA is not soliciting proposals for this effort. The evaluation piece will be performed on the scenarios developed by the TA5 performer in consultation with the Government. The program evaluations will take place one month before the end of each phase.

There will be one additional program evaluation within the first year of the program. This will serve as a dry-run for the subsequent program evaluations, as well as determine a baseline for the AIDA technologies. The “ground truth” for the corpus will be produced by the TA5 performer.

TA1: Semantic Mapping and Feedback

Today, there are numerous technologies and platforms to analyze various media inputs, such as text, speech, images, videos, metadata, and other sensor inputs. However, these analytics provide disparate outputs with semantics that are not easily combined. Using only one-best output may result in errors due to ambiguities in the input or inaccuracies in the analysis process. Also, it is often unclear which derived knowledge elements should be considered for hypotheses and which can be treated as background information.

Proposers to TA1 should address the core research challenge of extracting knowledge elements from all available media using existing analysis algorithms provided by the TA1 performer and identified in the proposal, as well as mapping this information into the common semantic representation language defined by the TA2 performer(s).

The primary input to TA1 will be streams of text, speech, images, video, and their associated metadata, from a variety of genres, both formal (e.g., news) and informal (e.g., social media, blogs). As previously mentioned, these information streams are expected to be partially filtered, with items relevant to the event, situation, or trend of interest, but also with a significant number of irrelevant items. TA5 will be providing approximately 100k discrete information items (documents, audio snippets, etc.) that will have been selected on the basis of metadata alone to be from the relevant time period and region for the situation in question. It is up to the TA1 performer(s) to filter the information further, as they see fit, and to exclude irrelevant items.

TA1 algorithms must analyze each incoming information item and produce a set of structured representations about events, sub-events or actions, entities, relations, locations, time, and sentiments (referred to in this BAA as knowledge elements) that are observable in that information item. These knowledge elements, expressed in the AIDA common semantic representation formalism, will constitute the output of the TA1 algorithms.

TA1 proposals must not concentrate on improving the accuracy of analysis of individual media.

They must use existing algorithms and modify them to work within the AIDA framework. While TA1 algorithms should be capable of resolving ambiguities and discovering conflicts within an information item, this resolution and discovery is not expected to be absolute; in fact, it is preferable that there be multiple interpretations if any uncertainty remains.

HR001117S0026 ACTIVE INTERPRETATION OF DISPARATE ALTERNATIVES (AIDA) 9

Each knowledge element of the representation is expected to have a confidence measure associated with it, in addition to a provenance record. Mutually-exclusive knowledge elements should be marked as such. For example, a single object in a particular image might be either a dog or a cat, but cannot be both.

In addition to being able to perform analysis of a specific input information item in a generic background context, TA1 algorithms must also be capable of accepting alternate contexts and producing alternate analyses for each context. Proposers to TA1 must offer a means of using the common semantic representation of each hypothesis as context to modify priors, provide priming, produce feedback, or adjust models. For example, the analysis of a certain image produces knowledge elements representing a tank on a road. However, knowledge elements in one or more hypotheses or user input suggest that this is a river rather than a road. The analysis algorithm should use this information for additional analysis of the image with priors favoring a boat. The result of this feedback-context analysis should not eliminate the initial background-context analysis. Rather, the feedback-context analysis should produce alternate analyses with adjusted confidences and/or new knowledge elements to be submitted to the TA2 knowledge base. The common semantic representation language will combine knowledge elements of a context uniformly, regardless of the provenance of the element, whether human input, video, foreign language text, etc.

TA1 algorithms must utilize the application programming interface (API) layer in the form specified by the TA4 performer to input the information stream and output knowledge elements and must input and output partial hypotheses expressed in the common semantic representation data structure specified by the TA2 performer(s). Since the common semantic representation to be used in AIDA is to be defined by TA2 performer(s) early in the program, it has not been defined at the time of this BAA. For the purposes of the proposals in response to this BAA only, TA1 proposers may assume that knowledge will be communicated to the TA1 algorithms using the common semantic representation below (The actual knowledge base may be in some different representation, this description is just defining the representation conveyed through the API for the purposes of proposal writing only). The syntactic representation will be the Terse RDF Triple Language (Turtle) syntax for RDF triples, and the proposer may specify schemata, ontologies, or vocabularies of their choosing.

The first algorithm delivery will be due in time for integration in advance of the baseline evaluation (see schedule section below) and other algorithm deliveries will be due in time for integration for each end-of-phase evaluation.

Proposals partially addressing TA1 (as opposed to addressing all of TA1) must meet the following minimum requirements; they must address two or more media types (text, speech, image, video, or structured data suggested by proposers) and they must address the issues of mapping each media analysis output into the common semantic representation and using the hypotheses as feedback context.

TA2: Common Semantic Representation

Proposals for TA2 should be focused on research and development of a novel common semantic representation capable of representing structured information derived from the various TA1 analytics for the multiple media of interest in AIDA, and related algorithms. This structured information, referred to in this BAA as knowledge elements, shall be aggregated in a knowledge

HR001117S0026 ACTIVE INTERPRETATION OF DISPARATE ALTERNATIVES (AIDA) 10

base. Although there may be need for some natural language, image thumbnails, featurized media, etc. in the knowledge base for reference, registration, or matching purposes, it is expected that most of the assertions in the knowledge base will be expressible in the structured representation, with elements derived from an ontology. Alternatively, they may be mapped into an ontology-based structured representation, if the actual common semantic representation being proposed is a vector space or other non-symbolic representation.

Although this BAA uses the term “knowledge base,” it is not expected that the AIDA knowledge base will necessarily conform to standard definitions of a “knowledge base” or even a “probabilistic knowledge base,” but will be developed as needed to address AIDA program requirements.

The primary input to TA2 will be structured knowledge elements, with confidence measures, expressed in the common semantic representation language, that have been derived from one information item (e.g. a document) at a time. Additional inputs to TA2 will come from TA4 in the form of user edits, modifications, or additions to the knowledge base. Like the primary input, these additional inputs will be structured knowledge elements, with confidence measures, expressed in the common semantic representation language.

The knowledge base and individual knowledge elements in the common semantic representation must have a human-readable presentation form, in addition to the machine-tractable form to be used by TA3. Proposers to TA2 may propose any internal representation for the knowledge base, whether triple-store, multi-graph, vector space/embedding, etc., so long as it meets the technical requirements and has the presentation forms specified above. Additional requirements for the representations (and the presentation forms) include assertion-level confidence measures and assertion-level provenance(s).

TA2 proposals must address knowledge element reference determination (“linking”) and aggregation. For example, they must be able to determine whether the knowledge elements derived from a new document refer to an entity already existing in the knowledge base, merge in any new information into the record, and update confidences and provenances accordingly.

In addition, TA2 proposers should address the challenge of introduction of new ontological concepts that will enable the analytics to handle the various scenarios involved in the training and evaluation of the AIDA integrated prototype. It is expected that new ontological elements will be primarily specific concepts or leaf nodes, as opposed to upper-model concepts. Fully automated solutions to this challenge are preferred, but are not required, and are not a primary focus of the AIDA program.

It is expected that the TA2 performer(s) will share initial draft specifications for the common semantic representation language approximately 4 months after the beginning of Phase 1, so that TA1 and TA3 performers are able to start adapting their API layers accordingly.

Initial experimental versions of algorithms will be due in time for integration in advance of the baseline evaluation (see schedule section below) and interim and full algorithm deliveries will be due in time for integration for each subsequent evaluation.

Proposals partially addressing TA2 (as opposed to fully addressing TA2) may propose solutions for ontological concept learning, entity disambiguation, common semantic space design, etc. In

HR001117S0026 ACTIVE INTERPRETATION OF DISPARATE ALTERNATIVES (AIDA) 11

order to be selected, however, a proposal must demonstrate significant potential utility to the overall program.

TA3: Multiple Hypotheses

Proposers to TA3 should address the core research challenges of assembling information as expressed in the common semantic representation in the knowledge base to form one or more internally-consistent hypotheses. Each hypothesis should capture an internally-consistent model of the world as conveyed by an interpretation of some subset of the input data and expressed in the knowledge base. The goal is not to describe epistemic truth, but to construct hypotheses from the TA2 knowledge base to provide users and automated analytics with a representation of events, situations, or trends as expressed in the data stream. It is expected that, in most cases where the knowledge base is constructed from unstructured sources, there will be more than one hypothesis that could be extracted from the knowledge base. Furthermore, any contradictory or deceptive input data is likely to also result in additional hypotheses being generated. If there is a consistent disinformation narrative in the data, it should result in an identifiable hypothesis or hypotheses that capture the narrative that the disinformation campaign is attempting to convey.

Input information can flow from two sources to the TA3 algorithms:

The first input to TA3 will be the knowledge base produced in TA2. Since the common semantic representation to be used in AIDA is to be defined by TA2 performer(s) early in the program, it has not been defined at the time of this BAA. For the purposes of the proposal in response to this BAA only, TA3 proposers may assume that knowledge will be communicated to the TA3 algorithms using the common semantic representation below (the actual knowledge base may be in some different representation, this description is just defining the representation conveyed through the API, for the purposes of proposal writing only). The syntactic representation will be the Turtle syntax for RDF triples, and the proposer may specify schemata, ontologies, or vocabularies of their choosing.

The second input to TA3 will be from the user interface to be constructed by the TA4 performer. The user interface will allow the user to inspect individual hypotheses produced by TA3 (in some suitable presentation form) and potentially prune them or adjust the overall individual confidences for the hypotheses. Additionally, TA4’s user interface will allow the user to create new hypotheses (either from scratch, or by editing existing hypotheses), which then need to be passed back to the TA3 process, in addition to having elements, confidences, or relations injected back into the TA2 knowledge base.

For the purposes of proposing to TA3, proposers may assume that new or modified hypotheses created in the TA4 interface would be communicated to the TA3 algorithms using the same syntax as above (actual syntax and semantics of APIs would be defined early in the AIDA program).

TA3 algorithms must find all relevant hypotheses that are substantially different from all other hypotheses. Substantially different means that at least one of the knowledge elements in the hypothesis differentiates that hypothesis from all others. TA3 algorithms will be expected to produce a confidence measure for each hypothesis, building on the confidences of the component assertions in the knowledge base that contribute to the hypothesis.

HR001117S0026 ACTIVE INTERPRETATION OF DISPARATE ALTERNATIVES (AIDA) 12

Hypotheses produced by TA3 algorithms must be internally semantically consistent (logically, ontologically, and pragmatically). TA3 algorithms may discard hypotheses generated through errors caused by the individual analytics and/or wrong interpretation of ambiguous unstructured data, where possible. However, hypotheses should not be discarded just because their probability is low.

Proposals to TA3 must address research and development of novel techniques for estimating the confidence of the hypotheses, based on the elements associated with the hypotheses and on other factors as proposed. The confidence measures should take into account the confidences produced by the analysis processes, the ability to resolve ambiguities, knowledge element aggregation successes, etc.

In addition, every TA3 proposal should include a proposed semantic coherence metric (see section D. Program Evaluation).

The first algorithm delivery will be required to be in time for integration in advance of the baseline evaluation (see schedule section below), and other algorithm deliveries will be due in time for integration for each end-of-phase evaluation.

Proposals addressing only part of TA3 may focus on combination of evidence, semantic consistency/coherence metrics, or generation of a hypothesis as a subset of a larger knowledge base or semantic representation.

TA4: Integration and Multiple Interpretations

Proposals to TA4 should be focused on development of a prototype that is capable of accepting various data streams, utilizing the algorithms developed by the TA1-3 performers. The TA4 prototype must also be capable of managing user interaction by probing the knowledge base, displaying the hypotheses, submitting a query to the system, editing the knowledge elements in the knowledge base or the hypotheses, and injecting additional “what-if” hypotheses to any stage of the computation. Access to the knowledge base must be flexible, there must be a variety of means to visualize the information, confidences must be provided, and the sources (provenances) of all elements of the hypotheses or the semantic space must be maintained. The table below summarizes inputs and outputs to components in the integrated prototype, facilitated by APIs and infrastructure produced by the TA4 performer.

TA Input From Output To raw media media

TA1

hypotheses TA4

TA3

knowledge elements TA2 knowledge elements TA1 TA2 knowledge elements and hypotheses TA4 knowledge elements TA3

TA4

TA3 knowledge elements TA2 hypotheses TA1

TA2

TA4

HR001117S0026 ACTIVE INTERPRETATION OF DISPARATE ALTERNATIVES (AIDA) 13

knowledge elements and hypotheses TA4 knowledge elements TA2 hypotheses TA3TA4 hypotheses user knowledge elements and hypotheses

User

TA1

TA2

TA3

It is anticipated that significant TA4 work will be required in Phase 1, with definition of APIs, system building, and integration. In later phases, the TA4 team will continue to improve the integrated prototype, add features as needed, and coordinate with transition partners. The TA4 performer will be responsible for developing the APIs that TA1, TA2, and TA3 algorithms will use to accept input and output.

Operational testing of AIDA technology will be a crucial aspect of the program. Potential transition partners include a variety of Department of Defense, military, and intelligence community organizations, and proposers to TA4 must be prepared to travel to both CONUS and OCONUS transition partner sites.

Proposers to TA4 must have the capability to handle classified data. At the time of proposal submission, all proposers to TA4 must have personnel with Top Secret clearances who are eligible for SCI; access to facilities to store and process SCI material and hold SCI discussions;

and the ability to conduct experiments with AIDA prototypes on classified data in government facilities.

TA5: Data

Proposals to TA5 should address the development of novel techniques for effective creation, collection, and annotation of the data necessary for AIDA research, development, and evaluation.

It is expected that the program will explore four different scenarios: one scenario for training and three for evaluation. The scenarios will involve events such as international conflicts, natural disasters, violence at international events, or protests and demonstrations. DARPA will provide data for the training scenario with limited, basic annotation at the program’s kick-off. Proposers to TA5 should prepare to re-annotate the training scenario data according to the more extensive AIDA annotation standard to be developed in collaboration with DARPA, NIST and the other performers, and collect and annotate data for three evaluation scenarios, one for each end-of-phase evaluation. The re-annotation of the training scenario must be completed by 13 months after the beginning of Phase 1. The data for the evaluation scenario in Phase 1 must be completed by 16 months after the beginning of Phase 1, the data for the evaluation scenario in Phase 2 must be completed 13 months after the beginning of Phase 2, and the data for the evaluation scenario for Phase 3 must be completed 13 months after the beginning of Phase 3. These corpora will include all media of interest to the program (text, speech, images, video, and the associated metadata). They will contain both data that is relevant to the scenario as well as irrelevant data, but relevant data is expected to comprise between five and 20 percent of all data.

HR001117S0026 ACTIVE INTERPRETATION OF DISPARATE ALTERNATIVES (AIDA) 14

DARPA will work with the TA5 performer(s) to select real-world events, situations, or trends for the scenarios. A scenario may involve one major event, an event with multiple sub-events, or multiple events. The corpus for each scenario should consist of at least 100,000 information items (e.g., documents, audio snippets, etc.) with a balance between the different genres and media.

The languages of the input for each scenario will be English plus a local language relevant to the scenario. For every non-English language chosen for any scenario, the Government will provide linguistic resources and tools of a quality and composition to be determined, but consisting at least of the type and size found in a LORELEI Related Language Pack (LRLP)1. To see a sample LRLP, please visit the DARPA Open Catalog and refer to the Uzbek language pack (https://opencatalog.darpa.mil/BOLT.html).

Any proposal to TA5 should include a proposed annotation scheme that covers all aspects of data annotation for AIDA research, development, and evaluation. The annotation plan should be described in sufficient detail to show its applicability and feasibility, but may include open questions to be resolved in coordination with DARPA, NIST, and the other performers during the program.

D. Program Evaluation

As stated above, NIST will perform the program evaluation for AIDA. This section describes AIDA evaluation as it is currently planned, but NIST will make adjustments and changes as necessary during the course of the program.

It is anticipated that the assessment will consist of four parts, as summarized in the table below.

What will be evaluated Performance criteria

End-to-End Comprehensive, ranked analytic alternate interpretations F-measure

TA1 Events, actions, entities, relations, locations, time, and sentiment F-measure

TA2 Differentiated and disambiguated elements in semantic representation F-measure

TA3 Semantic coherence of hypotheses or interpretations semantic coherence measure

End-to-end program evaluation will determine the ability of an integrated AIDA system to discover and generate the different hypotheses. System results will be compared against the “ground truth” that will be assembled by combining all hypotheses generated by annotators and augmented by all the system-generated hypotheses that were adjudicated to be correct. Two adjudicators will perform the adjudication for each hypothesis and, in the case of discrepancies, a third adjudicator will be used to make a final decision. An F-measure will determine the accuracy of the system counting all off-target, inaccurate, or insufficiently differentiated hypotheses as false positives.

1 http://www.lrec-conf.org/proceedings/lrec2016/pdf/1138_Paper.pdf https://opencatalog.darpa.mil/BOLT.html http://www.lrec-conf.org/proceedings/lrec2016/pdf/1138_Paper.pdf

HR001117S0026 ACTIVE INTERPRETATION OF DISPARATE ALTERNATIVES (AIDA) 15

TA1 evaluation will consist of evaluating the accuracy of the knowledge elements produced by the TA1 algorithms. The “ground truth” will consist of all knowledge elements relevant to the various hypotheses. An F-measure will determine the accuracy of the algorithms counting all erroneous knowledge elements as false alarms. The specific means of determining relevance will be determined by the evaluation team during the program.

TA2 evaluation will involve the assessment of the knowledge elements entered into the common semantic representation and the aggregation algorithms, especially the disambiguation function.

This will also be an assessment of completeness and accuracy and will be evaluated by the F-measure.

TA3 evaluation will assess the semantic coherence of the generated hypotheses. This assessment will evaluate each hypothesis to ensure that the knowledge elements and other assertions are mutually logically, ontologically, and pragmatically consistent and not contradictory. Program participants and NIST will develop this metric during Phase 1. Every TA3 proposal should include a proposed semantic coherence metric.

E. Schedule/Milestones

PI meetings will be held every nine months, in the middle and end of each phase. In addition, there will be a kick-off meeting at the start of the program. Because AIDA PI meetings are expected to include working sessions on specific engineering, standards, and interoperability issues, performers are expected to include appropriate technical personnel (software developers, grad students, etc.) in PI meetings as needed to address meeting agendas, in addition to the PI.

Proposers should estimate travel costs accordingly.

The program manager and other US Government stakeholders will visit the different sites in each phase of the program.

Evaluations will occur on the seventeenth month of each phase. In addition, there will be a baseline evaluation nine months after the start of the program. The first phase training scenario will be used for the baseline evaluation, to ensure that the end-of-phase evaluations run properly.

F. Deliverables

TA1, TA2, and TA3 performers are required to deliver software compatible with the APIs for integration three months before the end of each phase so that the TA4 team can integrate their software into the TA4 integrated prototypes for evaluation and six months after the kickoff so that their software will be integrated into the prototypes for the baseline evaluation.

The TA4 team must integrate all available software in time to begin the baseline and end-of-phase evaluations.

The TA5 team must deliver: the re-annotated training scenario corpus by 13 months after the beginning of Phase 1; the Phase 1 evaluation scenario corpus by 16 months after the beginning of Phase 1; the Phase 2 evaluation scenario corpus by 13 months after the beginning of Phase 2; and the Phase 3 evaluation scenario corpus by 13 months after the beginning of Phase 3.

HR001117S0026 ACTIVE INTERPRETATION OF DISPARATE ALTERNATIVES (AIDA) 16

The evaluation team will deliver the evaluation results 4 weeks before the end of the phase.

G. Government-furnished Property/Equipment/Information

For every non-English language chosen for any scenario, the Government will provide linguistic resources and tools of a quality and composition to be determined, but consisting at least of the type and size found in an LRLP2.

H. Intellectual Property

The program will emphasize creating and leveraging open source technology and architecture.

Intellectual property rights asserted by proposers are strongly encouraged to be aligned with open source regimes. See Section VI.B.2 for more details on intellectual property.

A key goal of the program is to establish an open, standards-based, multi-source, plug-and-play architecture that allows for interoperability and integration. This includes the ability to easily add, remove, substitute, and modify software and hardware components. This will facilitate rapid innovation by providing a base for future users or developers of program technologies and deliverables. Therefore, it is desired that all noncommercial software (including source code), software documentation, hardware designs and documentation, and technical data generated by the program be provided as deliverables to the Government, with a minimum of Government Purpose Rights (GPR), as lesser rights may adversely impact the lifecycle costs of affected items, components, or processes. See Section VI.B.2 for more details on intellectual property.

I. Security Clearance Requirements

Technical Area 4 performers must have cleared personnel to the TS/SCI level during the period of performance in order to communicate with transition partners and possibly evaluate the technology on their corpora.

2 http://www.lrec-conf.org/proceedings/lrec2016/pdf/1138_Paper.pdf http://www.lrec-conf.org/proceedings/lrec2016/pdf/1138_Paper.pdf

HR001117S0026 ACTIVE INTERPRETATION OF DISPARATE ALTERNATIVES (AIDA) 17

II. Award Information

A. Awards

Multiple awards are anticipated. The level of funding for individual awards made under this solicitation has not been predetermined and will depend on the quality of the proposals received and the availability of funds. Awards will be made to proposers whose proposals are determined to be the most advantageous and provide the best value to the Government, all factors considered, including the potential contributions of the proposed work, overall funding strategy, and availability of funding. See Section V for further information.

The Government reserves the right to:

select for negotiation all, some, one, or none of the proposals received in response to this solicitation;

make awards without discussions with proposers;

conduct discussions with proposers if it is later determined to be necessary;

segregate portions of resulting awards into pre-priced options;

accept proposals in their entirety or to select only portions of proposals for award;

fund proposals in increments and/or with options for continued work at the end of one or more phases;

request additional documentation once the award instrument has been determined (e.g., representations and certifications); and remove proposers from award consideration should the parties fail to reach agreement on award terms within a reasonable time or the proposer fails to provide requested additional information in a timely manner.

Proposals selected for award negotiation may result in a procurement contract, cooperative agreement, or Other Transaction (OT) depending upon the nature of the work proposed, the required degree of interaction between parties, and other factors.

Proposers looking for innovative, commercial-like contractual arrangements are encouraged to consider requesting Other Transactions. To understand the flexibility and options associated with Other Transactions, consult www.darpa.mil/work-with-us/contract-management#OtherTransactions.

In all cases, the Government contracting officer shall have sole discretion to select award instrument type, regardless of instrument type proposed, and to negotiate all instrument terms and conditions with selectees. DARPA will apply publication or other restrictions, as necessary, if it determines that the research resulting from the proposed effort will present a high likelihood of disclosing performance characteristics of military systems or manufacturing technologies that are unique and critical to defense. Any award resulting from such a determination will include a requirement for DARPA permission before publishing any information or results on the program. For more information on publication restrictions, see the section below on Fundamental Research.

http://www.darpa.mil/work-with-us/contract-management#OtherTransactions http://www.darpa.mil/work-with-us/contract-management#OtherTransactions

HR001117S0026 ACTIVE INTERPRETATION OF DISPARATE ALTERNATIVES (AIDA) 18

B. Fundamental Research

It is DoD policy that the publication of products of fundamental research will remain unrestricted to the maximum extent possible. National Security Decision Directive (NSDD) 189 defines fundamental research as follows:

‘Fundamental research’ means basic and applied research in science and engineering, the results of which ordinarily are published and shared broadly within the scientific community, as distinguished from proprietary research and from industrial development, design, production, and product utilization, the results of which ordinarily are restricted for proprietary or national security reasons.

As of the date of publication of this BAA, the Government expects that program goals as described herein may be met by proposers intending to perform fundamental research and does not anticipate applying publication restrictions of any kind to individual awards for fundamental research that may result from this BAA. Notwithstanding this statement of expectation, the Government is not prohibited from considering and selecting research proposals that, while perhaps not qualifying as fundamental research under the foregoing definition, still meet the BAA criteria for submissions. If proposals are selected for award that offer other than a fundamental research solution, the Government will either work with the proposer to modify the proposed statement of work to bring the research back into line with fundamental research or else the proposer will agree to restrictions in order to receive an award.

Proposers should indicate in their proposal whether they believe the scope of the research included in their proposal is fundamental or not. While proposers should clearly explain the intended results of their research, the Government shall have sole discretion to select award instrument type and to negotiate all instrument terms and conditions with selectees. Appropriate clauses will be included in resultant awards for non-fundamental research to prescribe publication requirements and other restrictions, as appropriate. This clause can be found at www.darpa.mil/work-with-us/additional-baa.

For certain research projects, it may be possible that although the research being performed by the awardee is restricted research, a subawardee may be conducting fundamental research. In those cases, it is the awardee’s responsibility to explain in their proposal why its subawardee’s effort is fundamental research http://www.darpa.mil/work-with-us/additional-baa

HR001117S0026 ACTIVE INTERPRETATION OF DISPARATE ALTERNATIVES (AIDA) 19

III. Eligibility Information

A. Eligible Applicants

DARPA welcomes engagement from all responsible sources capable of satisfying the Government's needs, including academia (colleges and universities); businesses (large, small, small disadvantaged, etc.); other organizations (including non-profit); entities (foreign, domestic, and government); FFRDCs; minority institutions; and others.

DARPA welcomes engagement from non-traditional sources in addition to current DARPA performers.

1. Federally Funded Research and Development Centers (FFRDCs) and Government Entities

FFRDCs FFRDCs are subject to applicable direct competition limitations and cannot propose to this BAA in any capacity unless they meet the following conditions: (1) FFRDCs must clearly demonstrate that the proposed work is not otherwise available from the private sector. (2) FFRDCs must provide a letter on official letterhead from their sponsoring organization citing the specific authority establishing their eligibility to propose to Government solicitations and compete with industry, and their compliance with the associated FFRDC sponsor agreement’s terms and conditions. This information is required for FFRDCs proposing to be awardees or subawardees.

Government Entities Government Entities (e.g., Government/National laboratories, military educational institutions, etc.) are subject to applicable direct competition limitations. Government entities must clearly demonstrate that the work is not otherwise available from the private sector and provide written documentation citing the specific statutory authority and contractual authority, if relevant, establishing their ability to propose to Government solicitations.

Authority and Eligibility At the present time, DARPA does not consider 15 U.S.C. § 3710a to be sufficient legal authority to show eligibility. While 10 U.S.C.§ 2539b may be the appropriate statutory starting point for some entities, specific supporting regulatory guidance, together with evidence of agency approval, will still be required to fully establish eligibility. DARPA will consider FFRDC and Government entity eligibility submissions on a case-by-case basis; however, the burden to prove eligibility for all team members rests solely with the proposer.

2. Foreign Participation

Non-U.S. organizations and/or individuals may participate to the extent that such participants comply with any necessary nondisclosure agreements, security regulations, export control laws, and other governing statutes applicable under the circumstances.

HR001117S0026 ACTIVE INTERPRETATION OF DISPARATE ALTERNATIVES (AIDA) 20

B. Organizational Conflicts of Interest

FAR 9.5 Requirements In accordance with FAR 9.5, proposers are required to identify and disclose all facts relevant to potential OCIs involving the proposer’s organization and any proposed team member (subawardee, consultant). Under this Section, the proposer is responsible for providing this disclosure with each proposal submitted to the BAA. The disclosure must include the proposer’s, and as applicable, proposed team member’s OCI mitigation plan. The OCI mitigation plan must include a description of the actions the proposer has taken, or intends to take, to prevent the existence of conflicting roles that might bias the proposer’s judgment and to prevent the proposer from having unfair competitive advantage. The OCI mitigation plan will specifically discuss the disclosed OCI in the context of each of the OCI limitations outlined in FAR 9.505-1 through FAR 9.505-4.

Agency Supplemental OCI Policy In addition, DARPA has a supplemental OCI policy that prohibits contractors/performers from concurrently providing Scientific Engineering Technical Assistance (SETA), Advisory and Assistance Services (A&AS) or similar support services and being a technical performer.

Therefore, as part of the FAR 9.5 disclosure requirement above, a proposer must affirm whether the proposer or any proposed team member (subawardee, consultant) is providing SETA, A&AS, or similar support to any DARPA office(s) under: (a) a current award or subaward; or (b) a past award or subaward that ended within one calendar year prior to the proposal’s submission date.

If SETA, A&AS, or similar support is being or was provided to any DARPA office(s), the proposal must include:

The name of the DARPA office receiving the support;

The prime contract number;

Identification of proposed team member (subawardee, consultant) providing the support; and An OCI mitigation plan in accordance with FAR 9.5.

Government Procedures In accordance with FAR 9.503, 9.504 and 9.506, the Government will evaluate OCI mitigation plans to avoid, neutralize or mitigate potential OCI issues before award and to determine whether it is in the Government’s interest to grant a waiver. The Government will only evaluate OCI mitigation plans for proposals that are determined selectable under the BAA evaluation criteria and funding availability.

The Government may require proposers to provide additional information to assist the Government in evaluating the…