KAIROS_BAA_final_AMENDED_2018-12-27_v2.pdf

Broad Agency Announcement Amendment Knowledge-directed Artificial Intelligence Reasoning Over Schemas (KAIROS)

HR001119S0014

December 27, 2018

As amended 27 December to update Proposers Day registration, page 50

Defense Advanced Research Projects Agency Information Innovation Office 675 North Randolph Street Arlington, VA 22203-2114

HR001119S0014 KAIROS 2

Table of Contents

I. Funding Opportunity Description

A. Introduction/Background

B. Program Description/Scope

C. Program Structure

D. Technical Areas

E. Program Evaluation

F. Schedule/Milestones

G. Deliverables

H. Government-furnished Property/Equipment/Information

I. Intellectual Property

II. Award Information

A. Awards

B. Fundamental Research

C. Disclosure of Information and Compliance with Safeguarding Covered Defense Information Controls

III. Eligibility Information

A. Eligible Applicants

B. Organizational Conflicts of Interest

C. Cost Sharing/Matching

D. Other Eligibility Requirements

IV. Application and Submission Information

A. Address to Request Application Package

B. Content and Form of Application Submission

C. Submission Dates and Times

D. Funding Restrictions

E. Other Submission Requirements

V. Application Review Information

A. Evaluation Criteria

B. Review and Selection Process

VI. Award Administration Information

A. Selection Notices

B. Administrative and National Policy Requirements

C. Reporting

HR001119S0014 KAIROS 3

VII. Agency Contacts

VIII. Other Information

A. Frequently Asked Questions (FAQs)

B. Proposers Day

C. Submission Checklist

D. Associate Contractor Agreement Clause (ACA)

HR001119S0014 KAIROS 4

PART I: OVERVIEW INFORMATION

• Federal Agency Name: Defense Advanced Research Projects Agency (DARPA), Information Innovation Office (I2O)

• Funding Opportunity Title: Knowledge-directed Artificial Intelligence Reasoning Over Schemas (KAIROS)

• Announcement Type: Initial Announcement

• Funding Opportunity Number: HR001119S0014

• Catalog of Federal Domestic Assistance Numbers (CFDA): 12.910 Research and Technology Development

• Dates o Posting Date: December 20, 2018 o Proposers Day: January 9, 2019 o Abstract Due Date: January 23, 2019, 12:00 noon (ET) o Proposal Due Date: February 27, 2019, 12:00 noon (ET) o BAA Closing Date: February 27, 2018, 12:00 noon (ET)

• Anticipated Individual Awards: DARPA anticipates multiple awards for Technical

Areas 1 and 2 and single awards for Technical Areas 3 and 4

• Types of Instruments that May be Awarded: Procurement contracts, cooperative agreements, or Other Transactions. Grants will not be awarded.

• Agency Contacts o Technical POC: Dr. Boyan Onyshkevych, Program Manager, DARPA/I2O o BAA Email: KAIROS@darpa.mil o BAA Mailing Address:

DARPA/I2O

ATTN: HR001119S0014

675 North Randolph Street Arlington, VA 22203-2114 o I2O Solicitation Website: http://www.darpa.mil/work-with-us/opportunities http://www.darpa.mil/work-with-us/opportunities

HR001119S0014 KAIROS 5

PART II: FULL TEXT OF ANNOUNCEMENT

I. Funding Opportunity Description

DARPA is soliciting innovative research proposals for the creation of a schema-based artificial intelligence capability to enable contextual and temporal reasoning about complex real-world events, in order to generate actionable understanding of these events and predict how they will unfold. Proposed research should investigate innovative approaches that enable revolutionary advances in science, devices, or systems. Specifically excluded is research that primarily results in evolutionary improvements to the existing state of practice.

This Broad Agency Announcement (BAA) is being issued, and any resultant selection will be made, using procedures under Federal Acquisition Regulation (FAR) 6.102(d)(2) and 35.016.

Any negotiations and/or awards will use procedures under FAR 15.4 (or 32 CFR § 200.203 for cooperative agreements). Proposals received as a result of this BAA shall be evaluated in accordance with evaluation criteria specified herein through a scientific review process.

DARPA BAAs are posted on the Federal Business Opportunities (FBO) website (https://www.fbo.gov/) and the Grants.gov website (https://www.grants.gov/).

The following information is for those wishing to respond to this BAA. Proposers are strongly encouraged to read the entirety of this document, as information on interactions among technical areas (TAs) and information on evaluation, schedule, and deliverables is provided in sections other than those describing the particular TAs.

A. Introduction/Background

Rapid comprehension of world events is essential for informing U.S. policy, diplomacy, and national security, a task that becomes more difficult as the amount of unstructured, multimedia information grows exponentially. Vital early indicators are often overlooked in the sheer amount of information available. Many important events are not simple occurrences, but complex phenomena that are composed of numerous subsidiary elements, some of which happen simultaneously, while others are sequential and dependent on each other. The KAIROS program will develop mixed-initiative systems that can identify complex events and bring them to the attention of users.

KAIROS will explore how to understand complex events described in multi-media input by developing a semi-automated system that identifies, links, and temporally sequences their subsidiary elements, identifying the participants of the complex events and the subsidiary elements, and identifying the complex event type. An event is a recognizable and significant change in either the natural world or human society. Events of interest either create changes that have significant impact on national security or participate in causal chains that produce such impacts.

Humans make sense of events by organizing them into narrative structures that occur frequently.

These structures are abstracted into schemas, which are organized units of knowledge that represent a pattern of memory used in human cognition. The schemas now used in Artificial Intelligence (AI) research are based on schemas defined by the cognitive scientist Jean Piaget in https://www.fbo.gov/ https://www.grants.gov/

HR001119S0014 KAIROS 6

1923, as a way in which people process and store memories about event sequences. First-wave (i.e., rule-based, symbolic reasoning) AI systems1 that incorporated hand-crafted schemas were unable to scale when matching those schemas to real-world data. Second-wave (i.e., machine learning) AI systems require far too many manually-produced, annotated examples as training data for supervised machine learning methods to be practical.

B. Program Description/Scope

DARPA is seeking revolutionary ideas that use schema-based AI to comprehend events, their components, and the participants involved. The KAIROS program will seek to overcome the scaling limitations of prior approaches in two stages. The first stage entails learning schemas from big data, and the second stage applies these schemas to multi-media/multilingual information to discover and extract complex events of interest to KAIROS users. Although it is expected that there will be some overlap between the technical approaches required for the two stages, each will likely require a different type of expertise. Figure 1 depicts the two stages of KAIROS processing. DARPA seeks approaches that automate these operations to the greatest extent possible but recognizes that human curation, at points in the processing that would not impede scalability, could enable achievement of a substantially more capable system.

The first stage, schema learning, will construct a library of schemas through generalization, composition, and specialization operations, as discussed in the paragraphs below. In order to construct schemas of specific relevance to US Government users, the KAIROS approach envisions identifying and constructing many general schemas which serve as building blocks for generalization, composition, and/or specialization processes to produce the relevant schemas.

Generalization. This will entail identifying, from a large corpus of openly-available news and other public data, schemas describing primitive and complex events. For example, we observe multiple instances of people buying parts for motor vehicles. From those instances, we generalize to produce a schema for buying a tire. Similarly, we could build schemas for buying books, motor oil, lumber, etc. for other event instances observed in the data set. From these schemas, we can generalize to produce a generic purchase transaction schema (which has a buyer, a seller, and an exchange of payment for goods or services). This purchase transaction schema will then be available for further hierarchal generalization, for example, to create a general financial transaction schema. It is expected that generalization will be primarily automated, possibly with some manual curation in which a user can triage putative generalized schemas.

Composition. The purchase transaction schema will also be available for composition to form complex event schemas like adapt vehicle (describing the process of adapting a general-purpose vehicle for some special purpose such as a self-driving car). This adapt vehicle schema would be composed of a sequence of multiple schemas, including a purchase transaction schema with the vehicle as the goods purchased, multiple additional purchase transactions of materials or equipment by the buyer of the vehicle, and an install structure schema in which the buyer of the vehicle in the first purchase is likely to be the installer of a structure built from the new materials and equipment. Composition, therefore, requires the use of role constraints (e.g., the buyer of the vehicle is likely be identical to the buyer of the materials and the installer of the new structure) and temporal constraints (the buying of the vehicle will likely take place before the

1 See https://www.darpa.mil/attachments/AIFull.pdf for discussion of First Wave, Second Wave, and Third Wave AI

HR001119S0014 KAIROS 7

acquisition of the materials and equipment for the new structure, which in turn must precede the installation of the new structure). It is expected that composition will involve both manual curation for knowledge augmentation and automated suggestion of schemas to be composed.

Specialization. The purchase transaction schema can be specialized to include useful domain-specific knowledge about domains not observed in the original training data set. For example, if we have never observed vehicle purchase transactions in the training data set, a user can add information about the domain-specific ways in which a vehicle purchase transaction differs from other types of purchase transactions, such as adding the roles of auto loan officer, title clerk, and inspection mechanic. It is expected that specialization will be primarily accomplished through user curation, possibly with some assistance from automated suggestions for added constraints.

In addition to addressing event ordering, the temporal constraints will also address temporal proximity (e.g., frequently people will install a tire soon after purchasing it). Similarly, attributes like is located at and has job have periods for which they are likely to be valid, in the absence of specific contradicting information. All of these temporal constraints may require probabilistic representation.

Figure 1: The two stages of KAIROS processing

The second stage will be applied at run-time, to analyze event instances mentioned in multi-media data (e.g., multilingual and/or multi-media news, video, and other data sources). It will involve matching the schemas generated by the first stage to the run-time input data in order to detect events. The roles and temporal constraints of the complex schemas will enable making contextually-sensitive inferences about the logical and temporal structures of events from new input data. This stage will require identifying events and entities, as well as relationships among them, in streaming or batch data, in order to construct and extend a knowledge base. This stage will also require the full or partial relative temporal ordering of events and subsidiary elements to achieve levels of accuracy not possible with only timestamp information.

HR001119S0014 KAIROS 8

The schemas that best cover the run-time data will then become available for users as a basis for prediction of possible future events and event elements (such as event participants or locations).

KAIROS will not produce specific predictions (e.g., who will win an election), but rather predictions about generic event sequences (such as predicting that an election will probably have a winner after the voting is completed). Additionally, the user should be able to (further) specialize schemas at run time to incorporate domain-specific knowledge, with suggestions of elements to dynamically expand the schema by the automated system.

The KAIROS platform should supply both human-readable output for users and computer-readable output for further downstream processing by analytics.

C. Program Structure

The program will consist of four technical areas (TAs):

• TA1 – Generation of Schemas for Events

• TA2 – Representation and Use of Temporal Knowledge and Schemas

• TA3 – System Integration and User Interface

• TA4 – Data Creation for Development and Evaluation

Figure 2: The structure of KAIROS showing the different Technical Areas

The program will have three phases. The first phase (Phase 1) will be 18 months long, the second phase (Phase 2) will be 24 months long, and the third phase (Phase 3) will be 12 months long, for a total of 54 months.

HR001119S0014 KAIROS 9

Figure 2 depicts the structure of KAIROS including primarily automated and primarily manual processing steps. A proposal may address any single technical area, a combination of TA1 and TA2, or a combination of TA3 and TA4. Any other combination of technical areas addressed in one proposal is not permitted. Any proposer submitting a solution to more than one technical area may submit a combined proposal for both technical areas but must ensure that the budget and statement of work are well delineated so that partial awards are possible.

Evaluations will be performed on datasets developed by the TA4 performer in consultation with DARPA. There will be a total of four evaluations: approximately one month prior to the end of each of the three phases, and approximately one month before the halfway point of Phase 2.

There will also be a dry-run exercise approximately halfway through Phase 1 that will serve as a pilot for the subsequent evaluations and will help validate the evaluation methodology, as well as exercise the evaluation mechanisms. The TA4 performer will produce the “ground truth” for the corpus. A metric for each technical area is discussed below. Performers not demonstrating sufficient progress during each phase may not be funded for subsequent phases.

Classified information will not be used in the research and development activities conducted by TA1, TA2, TA3, and TA4 performers. It is possible that transition activities carried out by the TA3 performer will require evaluation on classified data to be provided by transition partners and evaluated on transition partner systems.

D. Technical Areas

Technical Area 1 – Generation of Schemas for Events

Figure 3: The structure of KAIROS highlighting TA1

Proposals to TA1 should concentrate on generation of schemas from large data sets. The resultant schemas should be the representations of the structure of events and their subsidiary elements, how the events evolve, and what the typical durations and orderings of the subsidiary event elements are. The input to TA1 will be the large multi-media/multilingual corpora of events to be provided by TA4 (see the section addressing TA4 below for details of the input data), in batch mode. Although KAIROS is primarily focused on developing automated technologies, especially for the purpose of scaling, there are aspects of the work that may be best done manually in advance (when the task size is manageable, the required analytic accuracy is high, and the manual work would not impede scalability of a deployed system). Figure 3 shows the KAIROS tasks involved in TA1.

HR001119S0014 KAIROS 10

Strong proposals to TA1 will extend and adapt previously-developed entity, relationship, and event extraction (detection, classification, and representation) technology for all the relevant media and data conditions, and should concentrate on augmenting that technology with temporal and event sequencing capabilities.

It expected that TA1 proposals will define methods to construct a library of schemas through hierarchical application of generalization, composition, and specialization. However, proposers may define different approaches, so long as the output schema library is as described in this BAA, and the level of user interaction is manageable. The format of the schemas in the library should be defined by TA3 in consultation with TA1. The paragraphs below describe what the generalization, composition, and specialization processes would accomplish. In order to construct schemas of specific relevance to US Government users, the KAIROS approach envisions identifying and constructing many general schemas which serve as building blocks for generalization, composition, and/or specialization processes to produce the relevant schemas.

Proposals incorporating manual curation should discuss how the resulting system would scale.

Generalization. This will entail identifying, from a large corpus of openly-available news and other public data, schemas describing primitive and complex events. For example, the system should be capable of detecting multiple instances of people buying parts for motor vehicles and generalizing from them to produce a schema for buying a tire. Similarly, it should be capable of creating schemas for buying books, motor oil, lumber, etc. for other event instances observed in the data set, and from these generalizing to produce a generic purchase transaction schema (which has a buyer, a seller, and an exchange of payment for goods or services). This purchase transaction schema will then be available for further hierarchal generalization, for example, to create a general financial transaction schema. It is expected that generalization will be primarily automated, possibly with some manual curation in which a user can triage putative generalized schemas.

Composition. The purchase transaction schema will also be available for composition to form complex event schemas like adapt vehicle (describing the process of adapting a general-purpose vehicle for some special purpose such as a self-driving car). This adapt vehicle schema would be composed of a sequence of multiple schemas, including a purchase transaction schema with the vehicle as the goods purchased, multiple additional purchase transactions of materials or equipment by the buyer of the vehicle, and an install structure schema in which the buyer of the vehicle in the first purchase is likely to be the installer of a structure composed of the new materials and equipment. Composition therefore requires the use of role constraints (e.g., the buyer of the vehicle is likely be identical to the buyer of the materials and the installer of the new structure) and temporal constraints (the purchase of the vehicle will likely take place before the acquisition of the materials and equipment for the new structure, which in turn must precede the installation of the new structure). It is expected that composition will involve both manual curation for knowledge augmentation and automated suggestion of schemas to be composed.

Specialization. The system should be capable of supporting the specialization of schemas to include useful domain-specific knowledge about domains not observed in the original training data set. For example, if we have never observed vehicle purchase transactions in the training data set, a user can add information about the domain-specific ways in which a vehicle purchase transaction differs from other types of purchase transactions, such as adding the roles of auto loan officer, title clerk, and inspection mechanic. It is expected that specialization will be

HR001119S0014 KAIROS 11

primarily accomplished through user curation, possibly with some assistance from automated suggestions for added constraints.

In addition to addressing event ordering, the temporal constraints will also address temporal proximity (e.g., frequently people will install a tire soon after purchasing it). Similarly, attributes like is located at and has job have periods for which they are likely to be valid, in the absence of specific contradicting information. All of these temporal constraints may require probabilistic representation.

Figure 3 illustrates one possible flow for the process of learning the schemas and user triage of these schemas. The resultant schemas would be stored in a schema library that would be accessible to the TA2 and TA3 software through a defined application programming interface

(API).

Figure 4: Schema of adaptive reuse showing the schema for each instance, their generalization, and the hierarchical structure resulting from composition.

Figure 4 gives an example of hierarchical schema construction using generalization and composition. The example addresses the adaptive reuse of cars and trucks for other purposes.

Starting with videos and textual material that describe examples of this process, each adaptive reuse instance in the multi-media training data set is analyzed in terms of the event elements that together make up the instance. Each instance undergoes media-specific analysis (information extraction, video event detection, etc.) resulting in a structure indicating the order of the event elements and their durations, when possible. These instance-level truck reuse structures are generalized to establish a schema for adaptive reuse, using composition to sequence multiple elements or events as needed. Some of the event elements could be basic (primitive or atomic) events, while others could be complex events themselves. Some of the composed event schemas, such as purchase and acquisition, would have already been derived from a larger set of event descriptions, because these would not be unique to the adaptive reuse schema. In Figure 4, the schemas generated earlier (purchase and acquisition) are used in the composed event schemas as shown at the right of the figure. Because some of the event elements appear in different orders, the system must infer that the order of those particular elements is not crucial to the process.

HR001119S0014 KAIROS 12

TA1 algorithms should also be able to derive and represent expected durations for event schemas and their components. The schemas should include representations of how long the component events of complex events typically last. TA2 algorithms will be expected to use the duration information in making a determination of whether or not a new potential event instance can be a match or not. Similarly, TA1 algorithms should be able to derive estimates for the expected validity period of relational information once it is observed, in order to assist the matching of event instances and their arguments to schemas. For example, the expected validity period of an observed birth date for a person is indefinite (people’s birth dates never change), but the validity period of a person’s observed location has a very limited time span (because people travel).

The TA1 algorithms must be able to communicate their results to a user in a human-readable format and accept user curation in the form of mixed-initiative interaction, by means of a TA1-produced user interface that allows users to examine, name, and validate the automatically-derived schemas. TA1 algorithms should be capable of accepting user input at multiple points in their analysis, such as during the initial input analysis and sequencing of events for a specific instance in the input data, but primarily during the production of event schemas through generalization, composition, and specialization. The curation process needs to allow the user, for example, to triage schemas to identify overgeneralizations, eliminate extraneous elements of composed schemas that could unnecessarily restrict their application, in addition to adding constraints or enrichment in the form of additional events or roles to specialized schemas. The user interface to the TA1 platform and user staffing will be the responsibility of the TA1 performers.

To ingest the input data corpora and to produce the content of the schema libraries in both human and machine readable forms, all TA1 algorithms must utilize the KAIROS APIs that the TA3 performer will provide.

The first delivery of TA1 algorithms will be due in time for integration in advance of the pilot evaluation (see schedule section below) and subsequent algorithm deliveries will be due in time for integration by TA3 for each evaluation. TA1 and TA2 performers will be expected to provide their KAIROS software to TA3 in a Docker container or similar form so that the TA3 performer can run evaluation data through the TA1 and TA2 software for the National Institute of Standards and Technology (NIST) and transition partner evaluations.

(Technical area descriptions continued on next page)

HR001119S0014 KAIROS 13

Technical Area 2 – Representation and Use of Temporal Knowledge and Schemas

Figure 5: The structure of KAIROS highlighting TA2

Proposals to TA2 should address the run-time analysis of event instances mentioned in multi-media data (e.g., multilingual and/or multi-media news, video, and other data sources), as well as associated entities and relations. TA2 systems should match the schemas generated by TA1 to the input data (batch or streaming) in order to detect events and characterize them using schemas and TA1-provided duration and validity constraints, as shown in Figure 5. The roles and temporal constraints of the schemas will enable contextually-sensitive inferences about the logical and temporal structures of events from new input data. TA2 technology will also determine the full or partial relative temporal ordering of events and subsidiary elements to achieve levels of accuracy not possible with only timestamp information.

TA2 algorithms will be required to assess the temporal relationships among the incoming event instances and the schemas stored in the schema library at all stages of the analysis. Specifically, each event instance identified in the input data should be represented in the temporal knowledge base. Those event instances should be matched by the TA2 algorithms not only against all the schemas in the schema library, but they should also be matched against all the partially-instantiated schemas already in the knowledge base to check whether the new event instance should be added to one or more of those existing partially instantiated schemas. Matching event instances to new or partially instantiated schemas involves ensuring that roles match as appropriate and temporal ordering relative to previously-observed component events make sense. The enriched partially (or fully) instantiated schemas should be represented in the knowledge base in a format defined by TA3 in consultation with TA2. Therefore, the temporal knowledge base will contain individual event instance representations, partially instantiated schemas, and fully instantiated schemas.

An event instance may be associated with more than one schema from the schema library and/or more than one partially-instantiated schema. At later stages of the analysis, it is expected that some of the partially-instantiated schemas will be eliminated due to lack of further matching

HR001119S0014 KAIROS 14

event elements.

Although this BAA uses the term “knowledge base,” it is not expected that the KAIROS knowledge base will necessarily conform to any specific standard definition of a “knowledge base” or even a “probabilistic knowledge base,” but will be developed as needed to address program requirements.

TA2 algorithms must be able to predict possible subsequent or missing (unmatched) component events from a partially instantiated schema. This not only will involve outputting an expected component event to the user but also may involve predicting which roles or event arguments are likely, based on the event participants of other events already observed in that instantiated schema. TA2 algorithms should also be able to predict an expected time frame for the subsequent event. These predictions may be utilized by a user to search for evidence that the component event has occurred, but may also be used to establish alerts in case the event happens in the future.

TA2 algorithms should also be able to make use of the expected durations for schemas produced by TA1. When matching event instances to a schema, TA2 algorithms should pay attention to the observed durations of the candidate component event instances and the sequencing as part of the determination of whether or not that potential event instance can be a match to the schema or not, or the determination of the probability of a match.

Additionally, the user should be able to (further) specialize schemas at run time to incorporate domain-specific knowledge, with suggestions of elements to dynamically expand the schema being made by the automated TA2 system, and the enhancements to the schema instance being propagated back to the curated schema library when appropriate.

Strong proposed approaches to TA2 will extend and adapt previously-developed entity, relationship, and event extraction (detection, classification, and representation) technology for all the relevant media and data conditions (see the section addressing TA4 below for details of the input data), and will concentrate on augmenting that technology with temporal and event sequencing capabilities, as well as enabling streaming processing for those technologies.

To ingest the multi-media, multi-lingual inputs, and the TA1’s schema library’s machine-readable output, all TA2 algorithms must utilize the KAIROS APIs that the TA3 performer will provide. The knowledge base will be the repository of event instances found in the run-time data, as well as being the repository for partially-instantiated and fully-instantiated schemas that are constructed from those event instances. There may be many partially-instantiated schemas, with only some of them populated enough to make them worth showing to a user.

The first delivery of TA2 algorithms will be due in time for integration in advance of the pilot evaluation (see schedule section below), and subsequent algorithm deliveries will be due in time for integration for each evaluation. TA1 and TA2 performers are expected to provide their KAIROS software to TA3 in a Docker container, or similar form, so that the TA3 performer can run evaluation data through the TA1 and TA2 software for NIST and transition partner evaluations.

HR001119S0014 KAIROS 15

Technical Area 3 – System Integration and User Interface

Figure 6: The structure of KAIROS highlighting TA3

Proposers to TA3 should present a plan for developing an integrated platform that hosts TA1 and TA2 algorithms, enables access to the curated schema library and the temporal knowledge base by system elements that need to access them, maintains overall system flow at runtime, and includes the run-time user interface. The TA3 performer will be responsible for designing a platform that accepts multi-media input in streaming mode or in batch from a corpus and passes that data to both TA1 and TA2 algorithms. Data should be passed to TA1 in batch mode only, and to TA2 in either batch or streaming modes. Figure 6 shows the KAIROS tasks that will be involved in TA3.

The TA3 platform will be responsible for creating the user interface for the automated processes within TA2 software. This user interface must enable users to read, edit, curate, and/or visualize the schemas partially or fully instantiated within TA2, the events predicted by the TA2 analysis, their sub-events in the appropriate order, confidence measures, and all participants involved.

The TA3 platform will enable joint human-computer predictive analysis by allowing utilization of event schema instances, the schema library, and temporal knowledge. The output of the user interface must also include temporal information (relative or absolute, as appropriate).

It is anticipated that significant TA3 work will be required in Phase 1, to include definition of APIs, system building, and integration. TA3 should expect to be responsible for defining the format for the schemas in the schema library in consultation with TA1 and defining the knowledge base format in consultation with TA2. In later phases, the TA3 team should continue to improve the integrated prototype, add features as needed, and coordinate with transition partners. The TA3 performer will be responsible for developing the APIs that TA1 and TA2 algorithms will use to accept input data and produce output. TA1 and TA2 performers will be required to provide their KAIROS software to TA3 in a Docker container, or similar form for integration, so that the TA3 performer can run evaluation data through the TA1 and TA2 software for NIST and transition partner evaluations.

HR001119S0014 KAIROS 16

Operational testing of KAIROS technology will be a crucial aspect of the program. Potential transition partners are expected to include a variety of Department of Defense, military, and intelligence community organizations, and proposers to TA3 must be prepared to travel to both continental United States (CONUS) and outside continental United States (OCONUS) transition partner sites.

Proposers to TA3 must have the capability to handle classified data. At the time of proposal submission, all proposers to TA3 should have personnel with Top Secret clearances who are eligible for sensitive compartmented information (SCI), and access to facilities to store and process SCI material and hold SCI discussions, as well as the ability to conduct experiments with KAIROS prototypes on classified data in U.S. Government facilities.

Technical Area 4 – Data Creation for Development and Evaluation

Proposals to TA4 should address the development of novel techniques for effective creation, collection, and annotation of the data necessary for KAIROS research, development, and evaluation. Proposers to TA4 will need to collect two types of data, one for the learning of schemas and the other for the run-time development and evaluation.

Data for the schema learning process will consist of large volumes of open source material, including news and public posts, videos, etc. This corpus will include material in English and one additional language for which reasonable natural language processing and speech processing training data sets are available. This corpus will consist of roughly 1,000,000 documents containing on the order of one hundred different types of events complex enough to give rise to multi-level schemas. There should be at least five (5) different instances of each type of event, with multiple sources for each event instance when possible. The initial delivery of the schema-learning corpus covering approximately forty (40) complex event types must be delivered by the sixth month of the program. Delivery of the remainder of the schema-learning corpus, covering approximately sixty (60) more event types, must be delivered by the twelfth month of the program. Additional smaller corpora will be required for each evaluation to ensure that the event types relevant to the evaluations are covered. Annotations will consist of the labeling of event schema and event instances. Proposers to TA4 should propose annotation schemes that provide at least brief narrative descriptions of the schemas and labeling of the event instances in the relevant sources.

Run-time data will consist of resources selected to contain events and schema to support training and evaluation. This data will be in English for every evaluation and one (1) other language per evaluation. It is expected that the program will explore five (5) different scenarios: one scenario for development and four for evaluation. The scenarios will involve complex events like international conflicts, natural disasters, violence at international events, or protests and demonstrations. The annotation for the evaluation scenarios should consist of the labeling of complex events and schemas of interest, the participants in each event, temporal information (absolute or relative), and all properties essential for comprehension of each event and its relation to the schema(s).

The data for each evaluation scenario must be delivered approximately two (2) months prior to that evaluation. The evaluation corpora will include all media of interest to the program (text, speech, images, video, and the associated metadata). The evaluation corpora will contain both data that is relevant to the scenario as well as irrelevant data, with relevant data expected to

HR001119S0014 KAIROS 17

comprise between five (5) and twenty (20) percent of all data.

For every non-English language chosen by DARPA for any scenario, the Government will provide linguistic resources and tools of a quality and composition to be determined, but consisting at least of the type and size found in standard automatic speech recognition data sets and in a LORELEI Related Language Pack (LRLP)2. To see a sample LRLP, please visit the DARPA Opportunities page (under the KAIROS solicitation) and refer to the Uzbek language pack (http://www.darpa.mil/work-with-us/opportunities). For the purposes of proposal preparation, proposers may show examples of data in any non-English language of their choosing, but the government will select the languages to be investigated during the program.

Any proposal to TA4 should include a proposed annotation scheme that covers all aspects of data creation and annotation for KAIROS research, development, and evaluation. The annotation plan should be described in sufficient detail to show its applicability and feasibility but may include open questions to be resolved in coordination with DARPA, NIST, and the other performers during the program.

E. Program Evaluation

NIST is expected to perform the program evaluation for KAIROS. This section describes KAIROS evaluation as it is currently planned, but NIST will make adjustments and changes as necessary during the course of the program.

For unclassified evaluations, the input data will be corpora of news and other publicly available material, and the evaluation schemas will be manually selected by DARPA to relate to topics of general interest, such as humanitarian assistance and disaster response, public corporate activities, or law enforcement. For transition partner evaluations of TA3, both the input data and evaluation schemas will be classified at the level designated by the data and system owner.

NIST will perform evaluations using program data. For the purpose of proposing to TA1, proposers should expect that the schema-generation module (TA1) will be evaluated for accuracy, consistency, and completeness. For each schema, as well as the generalized schemas, the subsidiary elements, associated actors, and other roles and event order will be verified. All correct results will be counted for a recall score, while all errors will count as false alarms for a precision score. The final score will be a composite of the F-value (harmonic mean of recall and precision).

For the purpose of proposing to TA2, proposers should expect that the new temporal knowledge base (TA2) will be evaluated for accuracy, consistency, and completeness of knowledge elements and their temporal information, whether relative or absolute. As discussed for TA1, F-value will be calculated from the precision and recall values of the knowledge base elements. To be correct, each knowledge base element and its assigned time value will be evaluated against the “ground truth.” For both TA1 and TA2, the metric will be stated in terms of error defined as one minus F-value. The second aspect of TA2 that will be explored is the ability to tie the input data to one or more schemas in the schema library and ability to enable a user to predict future events.

2 http://www.lrec-conf.org/proceedings/lrec2016/pdf/1138_Paper.pdf http://www.darpa.mil/work-with-us/opportunities

HR001119S0014 KAIROS 18

End-to-end evaluations with the human-machine partnership, including validation and curation of schemas, will be conducted by transition partners.

A dry-run evaluation will be conducted nine (9) months after kickoff to establish the validity of the evaluation paradigm. During the Phase 1, TA2 will use a manually constructed schemas, while for the latter phases, TA2 will use automatically-generated schemas. The evaluation at the end of Phase 1 will serve as a baseline. Following Phase 1, the program will conduct three (3) additional evaluations, one (1) in the middle of Phase 2, and one (1) each at the end of Phase 2 and Phase 3. Because the technology is new and has never been tested before, it is impossible to assign a baseline at this time. After the baselines have been established, the metric will consist of an improvement of 40% relative error reduction at each phase.

F. Schedule/Milestones

The schedule for KAIROS is shown in Figure 7. Principal Investigator (PI) meetings will be held every six (6) months, with three (3) meetings in Phase 1, four (4) meetings in Phase 2, and two (2) in Phase 3. In addition, there will be a kick-off meeting at the start of the program.

Because KAIROS PI meetings are expected to include working sessions on specific engineering, standards, and interoperability issues, performers are expected to include, in addition to the PI, appropriate technical personnel (software developers, grad students, etc.) in PI meetings as needed to address meeting agendas. The location of PI meetings will vary and may include locations throughout the continental U.S. For the purpose of estimating travel costs, proposers should assume that five (5) PI meetings will occur in the Washington, DC area and five (5) will occur in Los Angeles, and each PI meeting will require three (3) full days.

Evaluations are anticipated to occur on month 17, month 29, month 41, and month 53 of the program. Evaluations will be conducted remotely and travel to the evaluation site will not be required. In addition, there will be a pilot evaluation nine (9) months after the start of the program, which will help validate the evaluation methodology and exercise the evaluation mechanisms.

HR001119S0014 KAIROS 19

Figure 7: KAIROS Schedule

The program manager and other U.S. Government stakeholders will visit the different sites approximately once per year.

G. Deliverables

Performers may be required to provide, at a minimum, the deliverables described below.

• All performers must deliver any technical papers derived from work funded by KAIROS.

• TA1 and TA2 performers are required to deliver software source code compatible with

TA3-developed KAIROS APIs for integration one month before the start of each evaluation (as well as six months after the kickoff) so that the TA3 team can integrate their software into the KAIROS prototype system.

• The TA3 team must integrate all available software in time to begin each evaluation.

• TA1, TA2, and TA3 performers are required to make a final delivery of software source code compatible with TA3-developed KAIROS APIs, as well as all supporting language models, knowledge bases, and schema libraries, to the Government at the end of the overall period of performance.

• TA4 must provide data deliverables as specified in the TA4 section above.

• Annotated slide presentations must be submitted within one week after the program kickoff meeting and after each program event (site visits, PI meetings, etc.).

• Quarterly technical status reports detailing progress made, tasks accomplished, major risks, planned activities, trip summaries, changes to key personnel, any potential issues or problem areas that require the attention of the Government team, research, development, evaluation, and transition highlights from the relevant period must be provided within 10 calendar days of the end of each calendar quarter.

HR001119S0014 KAIROS 20

• Monthly financial status reports must be provided within 10 calendar days of the end of each calendar month.

• Each performer must deliver a final report at the end of the overall period of performance that summarizes their project.

H. Government-furnished Property/Equipment/Information

For every non-English language chosen for any scenario, the Government will provide linguistic resources and tools of a quality and composition to be determined, but consisting at least of the type and size found in an LRLP3.

I. Intellectual Property

The program will emphasize creating and leveraging open source technology and architecture.

Intellectual property rights asserted by proposers are strongly encouraged to be aligned with open source regimes. See Section VI.B.1 for more details on intellectual property.

A key goal of the program is to establish an open, standards-based, multi-source, plug-and-play architecture that allows for interoperability and integration. This includes the ability to easily add, remove, substitute, and modify software and hardware components. This will facilitate rapid innovation by providing a base for future users or developers of program technologies and deliverables. Therefore, it is desired that all noncommercial software (including source code), software documentation, hardware designs and documentation, and technical data generated by the program be provided as deliverables to the Government, with a minimum of Government Purpose Rights (GPR), as lesser rights may adversely impact the lifecycle costs of affected items, components, or processes.

3 http://www.lrec-conf.org/proceedings/lrec2016/pdf/1138_Paper.pdf http://www.lrec-conf.org/proceedings/lrec2016/pdf/1138_Paper.pdf

HR001119S0014 KAIROS 21

II. Award Information

A. Awards

DARPA anticipates multiple awards for TA1 and TA2, as well as single awards for TA3 and TA4. The level of funding for individual awards made under this solicitation has not been predetermined and will depend on the quality of the proposals received and the availability of funds. Awards will be made to proposers whose proposals are determined to be the most advantageous to the Government, all factors considered, including the potential contributions of the proposed work, overall funding strategy, and availability of funding. See Section V for further information.

The Government reserves the right to:

− select for negotiation all, some, one, or none of the proposals received in response to this solicitation;

− make awards without discussions with proposers;

− conduct discussions with proposers if it is later determined to be necessary;

− segregate portions of resulting awards into pre-priced options;

− accept proposals in their entirety or to select only portions of proposals for award;

− fund proposals in increments and/or with options for continued work at the end of one or more phases;

− request additional documentation once the award instrument has been determined (e.g., representations and certifications); and − remove proposers from award consideration should the parties fail to reach agreement on award terms within a reasonable time or the proposer fails to provide requested additional information in a timely manner.

Proposals selected for award negotiation may result in a procurement contract, cooperative agreement, or Other Transaction (OT) depending upon the nature of the work proposed, the required degree of interaction between parties, and other factors. Grants will NOT be awarded under this program.

Proposers looking for innovative, commercial-like contractual arrangements are encouraged to consider requesting Other Transactions. To understand the flexibility and options associated with Other Transactions, consult http://www.darpa.mil/work-with-us/contract-management#OtherTransactions.

In accordance with 10 U.S.C. § 2371b(f), the Government may award a follow-on production contract or Other Transaction (OT) for any OT awarded under this BAA if: (1) that participant in the OT, or a recognized successor in interest to the OT, successfully completed the entire prototype project provided for in the OT, as modified; and (2) the OT provides for the award of a follow-on production contract or OT to the participant, or a recognized successor in interest to the OT.

In all cases, the Government contracting officer shall have sole discretion to select award instrument type, regardless of instrument type proposed, and to negotiate all instrument terms and conditions with selectees. DARPA will apply publication or other restrictions, as necessary, if it determines that the research resulting from the proposed effort will present a high likelihood http://www.darpa.mil/work-with-us/contract-management#OtherTransactions http://www.darpa.mil/work-with-us/contract-management#OtherTransactions

HR001119S0014 KAIROS 22

of disclosing performance characteristics of military systems or manufacturing technologies that are unique and critical to defense. Any award resulting from such a determination will include a requirement for DARPA permission before publishing any information or results on the program. For more information on publication restrictions, see the section below on Fundamental Research.

B. Fundamental Research

It is DoD policy that the publication of products of fundamental research will remain unrestricted to the maximum extent possible. National Security Decision Directive (NSDD) 189 defines fundamental research as follows:

‘Fundamental research’ means basic and applied research in science and engineering, the results of which ordinarily are published and shared broadly within the scientific community, as distinguished from proprietary research…