HR001117S0035.pdf

Broad Agency Announcement Cyber-Hunting at Scale (CHASE)

HR001117S0035

May 31, 2017

Defense Advanced Research Projects Agency Information Innovation Office 675 North Randolph Street Arlington, VA 22203-2114

HR001117S0035 CYBER-HUNTING AT SCALE (CHASE) 2

Table of Contents

PART I: OVERVIEW INFORMATION

PART II: FULL TEXT OF ANNOUNCEMENT

I. Funding Opportunity Description

A. Introduction and Background

B. Problem

C. Program Scope

D. Program Structure

E. Technical Areas

F. Evaluation and Schedule

G. Metrics

H. Integration and Evaluation Workshops

I. Intellectual Property

J. Deliverables

K. Government-Furnished Data and Equipment

L. Security Clearance Requirements

M. Facilities

N. Abstract and Proposal Structure

II. Award Information

A. Awards

B. Fundamental Research

C. Disclosure of Information and Compliance with Safeguarding Covered Defense Information Controls

III. Eligibility Information

A. Eligible Applicants

B. Organizational Conflicts of Interest

C. Cost Sharing/Matching

D. Other Eligibility Requirements

IV. Application and Submission Information

A. Address to Request Application Package

B. Content and Form of Application Submission

C. Submission Dates and Times

D. Funding Restrictions

HR001117S0035 CYBER-HUNTING AT SCALE (CHASE) 3

E. Other Submission Requirements

V. Application Review Information

A. Evaluation Criteria

B. Review and Selection Process

VI. Award Administration Information

A. Selection Notices

B. Administrative and National Policy Requirements

C. Reporting

VII. Agency Contacts

VIII. Other Information

A. Frequently Asked Questions (FAQs)

B. Collaborative Efforts/Teaming

C. Proposers Day

D. Submission Checklist

E. Associate Contractor Agreement Clause (ACA)

HR001117S0035 CYBER-HUNTING AT SCALE (CHASE) 4

PART I: OVERVIEW INFORMATION

Federal Agency Name: Defense Advanced Research Projects Agency (DARPA), Information Innovation Office (I2O)

Funding Opportunity Title: Cyber-Hunting at Scale (CHASE)

Announcement Type: Initial Announcement

Funding Opportunity Number: HR001117S0035

Catalog of Federal Domestic Assistance Numbers (CFDA): Not Applicable

Dates o Posting Date: May 31, 2017 o Proposers Day: June 1, 2017 o Abstract Due Date: June 14, 2017, 12:00 noon (ET) o Proposal Due Date: August 22, 2017, 12:00 noon (ET) o BAA Closing Date: August 22, 2017, 12:00 noon (ET)

Anticipated Individual Awards: Multiple awards anticipated for Technical Areas 1-4, one award for Technical Area 5.

Types of Instruments that May be Awarded: Procurement contracts or Other Transactions (OTs)

Agency Contacts o Technical POC: Dr. Jennifer Roberts, Program Manager, DARPA/I2O o BAA Email: CHASE@darpa.mil o BAA Mailing Address:

DARPA/I2O

ATTN: HR001117S0035

675 North Randolph Street Arlington, VA 22203-2114 o I2O Solicitation Website: http://www.darpa.mil/work-with-us/opportunities mailto:CHASE@darpa.mil http://www.darpa.mil/work-with-us/opportunities

HR001117S0035 CYBER-HUNTING AT SCALE (CHASE) 5

PART II: FULL TEXT OF ANNOUNCEMENT

I. Funding Opportunity Description

DARPA is soliciting innovative research proposals in the area of data-driven cyber-hunting tools for real-time cyber threat detection, characterization, and protection within enterprise-scale networks. Proposed research should investigate innovative approaches that enable revolutionary advances in science, devices, or systems. Specifically excluded is research that primarily results in evolutionary improvements to the existing state of practice.

This Broad Agency Announcement (BAA) is being issued, and any resultant selection will be made, using procedures under Federal Acquisition Regulation (FAR) 6.102(d)(2) and 35.016.

Any negotiations and/or awards will use procedures under FAR 15.4. Proposals received as a result of this BAA shall be evaluated in accordance with evaluation criteria specified herein through a scientific review process.

DARPA BAAs are posted on the Federal Business Opportunities (FBO) website (https://www.fbo.gov/).

The following information is for those wishing to respond to this BAA.

A. Introduction and Background

The DARPA Information Innovation Office (I2O) is soliciting innovative research proposals for the development of data-driven, cyber-hunting tools that support real-time cyber threat detection, characterization, and protection to bolster defenses across multiple enterprise networks. The Cyber-Hunting at Scale (CHASE) program will develop, demonstrate, and evaluate these tools for use within enterprise networks, across enterprise networks, and at internet exchange points.

The CHASE program will address the challenges of protecting dozens of enterprise networks in a coordinated manner. Each month, users within enterprise networks generate an order of magnitude more cyber-relevant data than the total storage available for cyber security purposes, and the fraction of cyber data stored within distributed databases still exceeds analysis capabilities. Proposed research should focus on dynamic approaches to accelerate cyber hunting via extraction of the right data from the right device at the right time.

If successful, CHASE will transition technologies to defensive cyber protection teams, enabling warfighters to dynamically collect data from mission critical parts of a network, actively hunt for threats that evade routine security measures, and disseminate protective measures that semi-automatically bolster the collective cyber defense posture.

B. Problem

Networks within the United States and abroad face increasingly broad-spectrum cyber threats from numerous actors and novel attack vectors. Malicious activity also crosscuts organizational boundaries, as nefarious actors use networks with less protection to pivot into networks containing key assets. Detection of these threats requires adjustments to network and host sensors at machine speed. Additionally, the data required to detect these threats may be https://www.fbo.gov/

HR001117S0035 CYBER-HUNTING AT SCALE (CHASE) 6

distributed across devices and networks. In all of these cases, the threat actors are using technology to perpetrate their attacks and hide their activities and movement, both physical and virtual, inside DoD, commercial, and Internet Access Provider (IAP) networks.

Enterprise-sized networks present challenges in terms of both their size and distributed structure.

Today’s state-of-the-art commercial tools do not directly address the scale and speed needed to provide the best defense for multiple networks. Networks lack robust mechanisms to collect, share, and respond to threat intelligence. Data required to detect and characterize malicious activities may be diffused and may be located across network and endpoint devices. Further, cyber-relevant data (including data that may contain information useful for detection and characterization) routinely exceeds the total available storage, bandwidth, and analysis capability, often by several orders of magnitude. Of data that is able to be stored, only some is currently analyzed, and of all alerts generated, only a fraction are threat related. Storage and processing limitations abound, so cyber defenders require tools that strategically direct resources toward the data that actually contains information about threats.

Current commercially available tools may output thousands of alerts and false positives per day that often cannot be verified due to a lack of processing capacity. Static data retention policies sometimes result in the deletion of relevant data prior to investigation. Additionally, current tools may neither proactively detect novel attack vectors nor detect coordinated attacks distributed across multiple organizations. Traditionally, cyber defense technologies focus predominantly upon either host data or network data; however, malicious activity crosscuts both networks and hosts.

Real-time detection of threats within or across very large enterprise networks is not simply an issue of scale, but also a challenge due to the variable nature of malicious activities and their presentations. Notably, the right data varies based on the nature of the threat. Detailed data may be on one or a few devices and vary depending on the type of intrusion or attack. Sometimes detection and characterization requires detailed data from a few hosts and other times clues may be distributed across thousands of devices. For example, diagnosis of an in-memory exploit might require detailed data from a few devices, while characterization of a global botnet could require summary data from hundreds of thousands of devices.

Additionally, this data must be collected at the right time. Today’s systems rely on static data collection and repositories that delete data after fixed time windows. These systems are further limited in that they require individuals to manually reconfigure sensors in order to collect new data.

While the data related to malicious activity is small relative to legitimate activity, it is impossible to know what data will be important a priori. Although only a fraction of cyber data is threat related, today’s tools fail to accurately localize, extract, and distinguish cyber threat data from data associated with legitimate activity. The defense of large-scale and distributed networks requires dynamic tools that iteratively converge on the right data, from the right devices, and at the right time.

C. Program Scope

The CHASE program will develop automated tools to detect and characterize novel attack vectors, collect the right contextual data, and disseminate protective measures both within and

HR001117S0035 CYBER-HUNTING AT SCALE (CHASE) 7

across enterprises. CHASE will prototype components that enable network owners to reconfigure sensors and disseminate protective measures at machine speed with appropriate levels of human supervision. CHASE technologies will conduct real-time investigations of potential cyber threats through adaptive data collection. Threat detection algorithms developed under CHASE may be tailored to characterize and react to specific classes of threats in the context of different data types and data sources. Additionally, these algorithms may work in concert to determine probabilities of the reality of threats, as well as indicate requirements for additional data that should be collected. As such, CHASE will develop foundational technologies for detection, characterization, and strategic data management.

Enhanced threat detection will cue the generation of automated protective measures. CHASE will focus on protective measures that a network owner has the authority to execute within her own environment. CHASE will explore how next generation protective measures affect the accuracy and efficiency of threat detection techniques. Protective measures of interest include, but are not limited to honeypots, auto-patching techniques, and active modification of the network. Adaptations of existing protective measures are in scope. However, fundamental or basic research that focuses solely on new protective measures is out of scope. Likewise, research that fails to address how protective measure usage affects threat detection and characterization efficacy is out of scope.

CHASE will develop components that integrate into existing systems. To ensure realism, the program will provide data from enterprise networks to inform, test, and evaluate CHASE methods. CHASE components that efficiently detect and characterize threats within real datasets will become candidates for transition.

Ultimately, CHASE technologies will be evaluated to identify components that work well within a variety of settings and determine which combinations of components function best within select network environments. Components selected for transition should provide novel capabilities and interact effectively with platform(s) supported by transition partners.

Components should complement or extend capabilities commonly found in cyber defense platforms, rather than duplicate existing functionality.

Transition of CHASE technologies is a priority for the program. Transition partners may include organizations such as U.S. Cyber Command, DoD Cyber Protection Brigades, the Defense Information Systems Agency, the Department of Homeland Security, and/or commercial cybersecurity firms. Early and continuous input from the CHASE transition partners will ensure relevance and provide CHASE with a continuous understanding of the rapidly evolving state of cyber attacks. Proposers should explain how their approach will support experts in Security Operations Centers (SOCs) and/or DoD Cyber Protection Teams.

D. Program Structure

CHASE is a four-year effort divided into an initial 24-month phase (Phase 1) followed by two 12-month phases (Phase 2 and Phase 3). Phase 1 will focus on developing, demonstrating, and evaluating individual technology components. Phase 2 will focus on comparative evaluations and evaluation of feedback loops formed by integrating prototype components. Finally, Phase 3 will extend functionality and transition capabilities to military partners and/or Federal agencies.

Not all efforts will align directly with program phases, so proposers should plan to adjust schedules based on results of Government evaluations and transition opportunities.

HR001117S0035 CYBER-HUNTING AT SCALE (CHASE) 8

The selected CHASE performers are required to collaborate with each other. Associate Contractor Agreements (ACAs) are necessary to help facilitate an open exchange of information and ensure compatibility between software components, the system architecture, data, and other program elements. The ACA will be used to share scientific results, prevent unnecessary duplication of effort, encourage appropriate coordination, and facilitate integration of technologies. All selected performers will be required to have an ACA in place prior to the program kick-off meeting.

The Government will assess performer progress using formal and informal technology evaluations, as well as potentially adversarial engagements driven by specific operational scenarios. The operational scenarios may incorporate voice-of-the-offense teams and defensive cyber analysts. The scenarios will incorporate aspects of real-world cyber operations. DARPA encourages technical efforts that allow for adaptability in testing and evaluation to take advantage of time-sensitive opportunities to access data sets, while also supporting program planned research and development.

E. Technical Areas

Figure 1: CHASE concept. Boxes represent CHASE Technical Areas.

DARPA seeks innovative proposals in the following Technical Areas (TAs), as shown in Figure 1 and described in detail below:

HR001117S0035 CYBER-HUNTING AT SCALE (CHASE) 9

TA1 - Threat Detection and Characterization

TA2 - Informed Data Planning

TA3 - Global Analysis

TA4 - Protective Measure Generation and Dissemination

TA5 - Infrastructure for Evaluation Exercises

Proposals may address a single technical area for TA1 through TA5. Proposals may also address TA1 and TA2 jointly; TA1 and TA4 jointly; or TA1, TA2 and TA4 jointly. Multiple prime performers are envisioned for TA1 through TA4; and a single prime performer for TA5 is preferred.

Proposers are encouraged to read descriptions of all TAs to ensure a full understanding of the program context and anticipated relationships among performers. Please pay special attention to the joint TA1 TA2 and TA1 TA4 requirements, as well as the general requirements listed in the evaluation and metrics sections. The evaluation, metrics, and Government-furnished data sections (Section I, subsections F, G, and K) provide more information about datasets that the Government may provide for evaluation purposes.

TA1 - Threat Detection and Characterization

The goal of TA1 is to detect malicious actions that threaten or compromise the integrity of cyber infrastructure within an enterprise network. TA1 has four objectives: (1) detect and characterize cyber threats; (2) accelerate cyber hunting operations; (3) adapt to data and resource limitations;

and (4) inform data collection and protective measure generation. TA1 will develop techniques that integrate information from both networks and hosts, using data sources that include, but are not limited to raw packets, processed packet capture (PCAP), NETFLOW, host logs, process logs, memory logs, and crash dumps.

TA1 proposals must describe an innovative, data-driven approach to detect and characterize malicious activities within enterprise networks. Approaches should distinguish between malicious activities and benign anomalies with high confidence. Activities of interest include, but are not limited to web sessions, firewall anomalies, session anomalies, network activities, and irregular patterns of life indicative of malicious intent. TA1 methods may require theoretical advances in multimodal models of malicious activity and development of new cyber threat indicators. Proposers are encouraged to consolidate information across network and host sensors in novel ways.

Methods should detect malicious activities that evade conventional defense technologies, such as antivirus software, firewalls, intrusion detection systems, and access control systems. TA1 approaches are encouraged to advance feature extraction techniques beyond current state-of-the-art to improve detection. Proposers are encouraged to challenge standard assumptions and crosscut boundaries in ways that commercial offerings do not. Topics of interest include exploration of nonstandard features and automated convergence on features that enhance detection accuracy. Approaches that detect cyber threats within encrypted data are desirable, but not necessary.

HR001117S0035 CYBER-HUNTING AT SCALE (CHASE) 10

TA1 solutions should accelerate cyber hunting within enterprise networks. Techniques should minimize the time required to detect comprised assets. TA1 proposals should address how to accelerate detection even if algorithms begin with incomplete information. TA1 solutions should quickly triage Terabytes of data to identify activities or devices that require deeper examination.

Proposed methods should drastically reduce the time cyber analysts spend gathering contextual information and addressing false alarms. Fully automated hunt workflows are strongly encouraged. When human-in-the-loop workflows are required, proposers are encouraged to explain the rationale behind assigning tasks to humans or to machines. Workflows that take into account the relative strengths of both humans and machines are desirable.

Methods should support both real-time incident response and forensic analysis. Proposers are also encouraged to describe how their technologies would enable search, reasoning, and identification of patterns within petabytes of data in a cloud-based repository. Furthermore, proposers are encouraged to address how methods will adapt to data and resource limitations.

TA1 proposals must address how detection and characterization algorithms will contribute to both data collection and generation of protective measures. Regarding data collection, TA1 proposers should provide a technical approach that connects threat indicators to data requirements for TA2. TA1 algorithms should specify the data type, device, and collection time interval required to validate the presence or apparent absence of malicious activity with high confidence. Regarding protective measures, TA1 proposers should provide a technically sound mechanism that uses detection events to construct decision manifolds for TA4. Please refer to the sections on joint TA1 TA2 and TA1 TA4 requirements for more details.

TA1 proposers should provide a plan to obtain datasets from real networks to support proof-of-concept evaluations early in the program. Proposals should include information about data fidelity and scale. Proposals should also describe plans to support evaluations on Government-furnished datasets starting early in Phase 1. Please refer to the sections on evaluation, metrics, and Government-furnished data for more information.

TA2 - Informed Data Planning

The goal of TA2 is to manage data resources in a manner that makes threat detection faster and more reliable. TA2 methods should perform cost-value trade-offs to optimize utilization of distributed resources with limited data collection, storage, and processing capabilities. TA2 will develop innovative methods that assess data value, plan and execute data collection, and construct optimal data transmission and retention policies.

TA2 proposals must include an innovative technical approach for data valuation. Data has high value when it maximizes the likelihood of accurate cyber threat detection and minimizes the likelihood that adversaries will compromise important assets without anyone noticing. Proposers are encouraged to develop methods that identify essential parts of the network and assign high value to data that helps protect critical assets. Proposers should describe quantitative mechanisms that dynamically adjust data valuation in response to threat assessments from TA1.

Data valuation methods should also quantify costs associated with data collection. While a piece of data might maximize the likelihood of accurate cyber threat detection, a measurement might tip off an adversary or require an impractical amount of storage. Proposers should outline a technical approach that quantifies data collection costs, including but not limited to human labor, HR001117S0035 CYBER-HUNTING AT SCALE (CHASE) 11 bandwidth, network degradation, disruption to legitimate users, and disadvantageous effects on adversary behavior. Together, cost and value appraisals should provide automated mechanisms that dynamically highlight data with maximum information utility, given knowledge of network conditions and observations of malicious activity.

TA2 proposals should discuss strategies for automated data collection within an enterprise network. Innovative methods will automate collection of the right data within enterprise networks based upon cost assessments, data valuation, and evolving threat intelligence from TA1. Methods should generate clear data collection recommendations for a variety of malicious activities, taking into consideration cases in which data must be co-located to support TA1 detection and characterization.

Data collection plans will specify data types, devices, and time intervals required to collect the right data from the right device at the right time. Proposers are encouraged to describe dynamic planning techniques that direct sensor placement, establish default data collection policies, and generate timetables to implement new collections based upon evolving threat intelligence.

Methods that monitor measurement quality are encouraged. Strong TA2 proposals will also address data quality issues, such as mistimed collections, subsampling, and events missed due to conflicting data requests received at the same time.

TA2 proposals should provide an innovative technical approach for executing data collection plans. Topics of interest include, but are not limited to novel sensor types, efficient reconfiguration of sensors, sensor randomization to obfuscate noisy measurements, honeypots, honeynets, and covert techniques that do not alert the adversary that they have been detected.

TA2 approaches should specify when dynamic data collections can be automated, and when they require manual intervention. Proposers are encouraged to develop fully automated workflows for high value data. For semi-automated workflows, proposers are encouraged to minimize the human operator’s skillset requirements and reduce manual labor.

In addition to data collection, TA2 methods will strategically direct data transfer and retention to best utilize resources within an enterprise network. Data of greatest interest exists across variable time horizons, so TA2 policies should consider data value and costs when formulating intelligent recommendations. The TA2 approach must address three challenges: (1) What data should be stored on the host? (2) What data should be retained at the sensor? (3) What data should be pulled back to one or more repositories for TA1 detection and characterization? This approach should consider tradeoffs between local storage capacity, transmission costs, data processing requirements, and threat intelligence from TA1. For the purposes of the proposal, assume that the enterprise network has PCAP storage at the sensor, limited data storage on most hosts, and a central integrated data repository for multiple data types. Also, proposers should assume that some traffic may be encrypted.

Proposals should address extension or adaptation of data collection, retention, and transmission techniques to operate across enterprise networks, regional gateways (e.g. Internet exchange points), and global data repositories (e.g., Information Sharing and Analysis Centers within the commercial sector or Acropolis within the Department of Defense (http://www.disa.mil/Cybersecurity/Analytics/Acropolis). Topics of interest include, but are not limited to privacy and coordination that enhance security across enterprises.

http://www.disa.mil/Cybersecurity/Analytics/Acropolis

HR001117S0035 CYBER-HUNTING AT SCALE (CHASE) 12

TA2 proposers should provide a plan to evaluate methods using data from real networks.

Proposals should include information about data fidelity and scale. TA2 teams will likely obtain access to Government-furnished datasets early in Phase I to support development of planning components. Teams should also develop compelling demonstrations for data collection components that do not rely on Government-furnished datasets. Please refer to the sections on evaluation, metrics, and Government-furnished data for more information.

Requirements for Proposals Addressing TA1 and TA2

Ultimately, TA1 and TA2 algorithms will form a feedback loop, as shown in Figures 1 and 2.

The feedback loop will enable detection and characterization algorithms to collect additional information about threats under investigation. TA1 proposals should explain how detection and characterization algorithms will be used to generate data requests, while TA2 proposals should explain strategies that efficiently reconfigure sensors and collection methods in response to common types of malicious activity. Thus, both TA1 and TA2 proposals should discuss methods to connect malicious activities with data requests. During the program, TA2 performers will coordinate closely with TA1 performers to expand upon proposed feedback mechanisms.

Proposals that address both technical areas should combine TA1 and TA2 capabilities in compelling ways.

Figure 2: All proposals addressing TA1, TA2, or both TA1 and TA2 should address how to convert malicious activities into data requests.

TA3 - Global Analysis

The objective of TA3 is to construct global situational awareness to detect otherwise invisible threats. TA3 approaches should focus on malicious activity that is invisible from the perspective of an isolated network, but becomes apparent when multiple systems share the right data. Thus, the primary goal of TA3 is to discover such globally visible, locally invisible activity.

Proposers will develop distributed algorithms that detect global events when it is not possible to move all relevant data into a centralized location. Approaches should delineate computations to perform at the sensor, at distributed repositories, and at a centralized repository. Proposers may assume that local computations occur in data repositories distributed across dozens of enterprise networks and/or regional Internet exchange points. Global analysis may occur at a centralized repository, but data transfers between locations are limited (see Figure 3). TA3 efforts should

HR001117S0035 CYBER-HUNTING AT SCALE (CHASE) 13

explore what local computations, global computations, and data transfers need to occur to maximize the breadth and depth of global phenomenon detected.

Proposers are encouraged to construct indicators of global activity and describe the observables required for detection. Indicators should construct a global situational awareness and discover global events that may otherwise go undetected at the local level. Methods should provide a means for local networks to share partial observations to determine whether local anomalies are indicative of global attacks. Proposers should provide rigorous mechanisms to validate or invalidate conclusions.

The TA3 approach should describe how to evaluate cross-organizational occurrences of novel attack vectors, and how to use global attack trends to detect unknown unknowns within local networks. Threat indicators should incorporate information across organizations to aggregate models and detect subtle signals from threat activities. Weak, incomplete indicators across multiple sites may combine to formulate stronger conclusions. Proposers should discuss whether a basis set of statistical computations might maximize the breadth of global phenomenon detected.

The above discussions should incorporate topics such as: (a) how to fuse data from multiple sources; (b) how to deal with out-of-sequence data; and (c) how to handle event data when the causal order may be uncertain.

Proposers should characterize how detection of global phenomena depends on data limitations.

First, TA3 efforts should establish the breadth of global phenomena that distributed algorithms can detect, given liberal data collection, transfer, and processing policies. Then, TA3 approaches should explore how detection depends on data sharing constraints, such as bandwidth limitations, privacy considerations, and data collection restrictions. Assessments should evaluate what types of data sharing constraints cause global phenomena to become undetectable.

Proposers are encouraged to define compelling scenarios that explore a breadth of global detection capabilities. For example, TA3 proposers might consider scenarios in which algorithms execute arbitrary computations and data collects at all data repositories, but minimize data transfers between local and global sites. Alternatively, algorithms might execute arbitrary computations at the global data repository, but rely on fixed data collection and processing at local repositories. Similarly, algorithms might execute arbitrary computations at all locations, but only share privacy-preserving aggregate statistics across sites. TA3 proposers are encouraged to select scenarios that elucidate theoretical limitations to global detection.

TA3 proposers should provide a plan to evaluate methods using data from real networks. TA3 proposals should describe a strategy to obtain compelling datasets for evaluation during all phases of the program. Proposals should include information about data types, data fidelity, sensor locations, and scale. Please refer to the sections on evaluation, metrics, and Government-furnished data for more information. Development of feedback loops between TA3 and the other technical areas will occur primarily during years 3 and 4 of the program.

HR001117S0035 CYBER-HUNTING AT SCALE (CHASE) 14

Figure 3: For the purposes of the proposal, TA3 proposers should assume their solutions would operate independently of TA1, TA2, and TA4 solutions. Development of feedback loops between TA3 and the other technical areas will occur primarily during years 3 and 4 of the program. TA3 proposals should assume access to a global data repository, local data repositories within enterprise networks, and local repositories at regional Internet gateways. TA3 proposals should address what computations need to occur at each location and what information must move between locations.

TA4 - Protective Measure Generation and Dissemination

TA4 will generate and disseminate protective measures in response to threat intelligence, as well as develop novel methods to cue protective actions within and across enterprises. Collectively, TA4 will develop core technologies to bolster defenses at machine speed with appropriate levels of human supervision.

Traditionally, protective measures consist of triggers (e.g., destination port 23) and actions (e.g., connection blocked at the firewall). Under TA4, performers will replace traditional rule-based triggers with decision manifolds, i.e., high dimensional decision boundaries. Decision manifolds will leverage the high-dimensional expressivity of behavior detection, signal processing, and machine learning to construct nuanced representations of malicious activities. High-dimensional manifolds should encode the space of activities that generate a protective response when detected in the network. When deployed to a device, decision manifolds should make it more difficult for adversaries to evade defenses than intrusion signatures and IP blacklists.

TA4 should develop innovative methods that are resilient against adversarial targeting. TA4 methods should use TA1 threat detection outputs to generate decision manifolds. Together, TA1 detection algorithms and TA4 decision manifolds will form a feedback loop that automates dissemination of protective measures. Proposed TA4 methods should ensure that the combination of TA1 detection algorithms and TA4 decision manifold generators create a robust system that counters adversarial manipulation.

Proposers should identify and address vulnerabilities exhibited by machine learning algorithms.

For example, causative attacks may introduce vulnerabilities during the learning phase and exploit them during classification. Exploratory attacks may find and subsequently exploit existing vulnerabilities. Adversarial methods may violate data integrity to misclassify malicious samples as legitimate, use targeted samples to circumvent protections, or increase overall misclassification rate so that classifiers becomes unusable. TA4 proposers should characterize machine learning vulnerabilities and recommend countermeasures. Ultimately, TA4 methods should also detect when TA1 machine learning algorithms are under attack and provide countermeasures that reduce the vulnerability of machine learning algorithms to adversarial attack.

HR001117S0035 CYBER-HUNTING AT SCALE (CHASE) 15

TA4 proposals should outline a sound technical approach that recommends protective actions in response to observed malicious activity. Given TA1 detections, TA4 methods should generate decision manifolds paired with recommended actions. Fully-automated actions and actions that require human intervention are both of interest. Methods should monitor and adjust protective measures as new information becomes available. Protections of interest include, but are limited to, actions that: block or monitor connections, enforce strict authentication, modify network segmentations, adjust sensor placements, share protective measures with partner organizations, auto-patch vulnerabilities, direct traffic to honeypots, perform probabilistic penetration testing, and isolate devices from the network.

TA4 proposers should provide innovative methods to share complex protective measures with partner networks. Automated dissemination of protective measures will require TA4 performers to develop a syntax and semantics for decision manifolds. TA4 proposers should address how the manifold description language will enable recipients to deploy protective measures even when the feature detection algorithms in the recipient’s environment differ from those used by the sender. Techniques that enable protective measures to be communicated across classification boundaries are of interest.

The manifold description language must be machine interpretable and promote dissemination of protective measures at machine speed. Human interpretable languages are of interest, but are not required. If decision manifold descriptions are too complex for humans to understand, proposers must provide a mechanism to gain human trust. Techniques should enable recipients to verify the integrity of both the source and the protective measure.

TA4 will evaluate whether select protective measures can improve the performance of TA1 detection and characterization algorithms. Protective actions of interest include, but are not limited to auto-patches, Honeypot Turing Tests, and automated isolation of devices from the network. Proposers should outline metrics to evaluate the ability of protective measures to detect novel attack vectors, block known attack vectors more efficiently than conventional methods, reduce TA1 false alarm rates, and significantly improve TA1 detection and characterization accuracy.

TA4 proposers should provide a plan to evaluate methods using data from real networks.

Proposals should include information about data fidelity and scale. TA4 teams will likely obtain access to Government-furnished datasets during year two to support development of manifold generation and protective measure recommendation components. Teams should also develop compelling demonstrations for TA4 components that do not rely on Government-furnished datasets. Please refer to the sections on evaluation, metrics, and Government-furnished data for more information.

Requirements for Proposals Addressing TA1 and TA4

Working collaboratively with TA1, TA4 will define methods to share protective measures across enterprises. As shown in Figure 4, proposals addressing TA1, TA4, or both TA1 and TA4 must outline automated mechanisms that convert TA1 detections into decision manifolds. TA1 will propose mechanisms specific to their proposed detection and characterization algorithms, while TA4 will develop holistic techniques that reduce vulnerability to adversarial attack. Proposals that address both technical areas should combine TA1 and TA4 capabilities in compelling ways.

HR001117S0035 CYBER-HUNTING AT SCALE (CHASE) 16

Figure 4: Proposals addressing TA1, TA4, or both TA1 and TA4 should provide a technical approach to covert malicious activity detections into decision manifolds. TA1 proposals should provide mechanisms to convert detection algorithm outputs into decision manifolds. TA4 proposals should address what types of decision manifolds are robust against adversarial attacks and propose techniques that convert vulnerable detection methods into resilient ones.

TA5 - Infrastructure for Evaluation Exercises

TA5 performers must establish and maintain an analysis environment (e.g., software) for exploration of host and network data at scale. The analysis environment must facilitate evaluation of TA1 - TA4 technologies and enable experiments to assess whether TA1 - TA4 technologies extract the right data from the right device at the right time.

The TA5 team should adapt and extend open source tools to create an analysis environment for cyber data sets. The analysis environment must be designed to deploy rapidly onto Government-provided infrastructure, which may consist of commercial and private cloud instances. The core environment should provide an open source alternative to large cyber data stores, such as those used by Security Information and Event Management Systems (SIEMs).

The TA5 environment will support research, development, internal testing, and formal evaluation exercises using enterprise network data. TA5 proposals must include a sound, detailed technical plan that describes user management, data management, collaboration, and continuous development tools for TA1 - TA4 performers and Government evaluators. TA5 proposers should delineate which technical requirements necessitate innovative extensions to existing tools.

Sections describing innovative extensions should include a clear technical plan that first outlines current capabilities and then explains how tools will be extended, adapted, or developed to achieve TA5 technical objectives.

TA5 user-management plans must address access to and provisioning of cyberinfrastructure.

Methods should carefully manage storage and compute resources for the dual purposes of controlling costs while maximizing performance for users. User-management plans should describe a technical approach that assesses algorithm performance before devoting significant resources to an algorithm's execution. Ideally, user-management approaches will have

HR001117S0035 CYBER-HUNTING AT SCALE (CHASE) 17

automated mechanisms to track jobs and ensure that scripts are first tested for scalability on small data samples before running on large data tables. Solutions that adjust baseline compute thresholds based on script history or user expertise are encouraged. Solutions that automatically pause or kill jobs that exceed projected compute profiles are also encouraged.

TA5 data-management plans should explain in detail how proposers will facilitate data ingest, indexing, search, and filtering operations for TA1 - TA4 evaluation exercises. Proposals should address how proposers will accomplish the following technical objectives:

• Efficiently ingest multi-TB Government-provided cyber datasets;

• Store data in manner that facilitates exploration and algorithmic processing by TA1 -

TA4 performers;

• Track data provenance for raw and pre-processed data;

• Provide data APIs that enable TA1 - TA4 performers to track data provenance;

• Archive or flag data for deletion based on TA2 recommendations; and

• Retain raw data in a manner that would support re-processing as needed.

Novel approaches to data provenance tracking are encouraged. Data provenance tracking should enable TA1 - TA4 to optimize algorithms based upon data collection information, such as sampling rates or sensor locations.

During evaluation exercises, TA5 performers will rapidly ingest cyber datasets and provide status updates to exercise participants. TA5 performers should discuss robust mechanisms to assess data quality and broadcast information about datasets in near real time. TA5 performers should also support streaming ingests to facilitate evaluation of TA1 TA2 feedback loops.

Similarly, a capability that replays batch datasets in a streaming manner is desirable for evaluation of TA1 TA4 protective measures.

TA5 collaboration plans should explain in detail tools that will facilitate TA1 - TA4 performers and evaluators to share context, analysis, and results. Proposed collaboration environments should address the following technical objectives:

• Develop, extend, and/or deploy tools that generate data summaries, trend analyses, and exploratory visualizations on newly loaded datasets;

• Enable exploration of results in a manner that does not require substantial data downloads;

• Track usage statistics on software and data; and

• Establish communication processes that facilitate discussion, clarification of ideas, and evaluation.

TA5 proposals must include a maintenance plan for hardware and software. Maintenance plans should address how to perform efficient software upgrades, respond to bugs, maintain software repositories, manage cloud configurations, and upgrade other elements as needed. Proposers should explain how the technical approach will minimize the human resources required for maintenance.

HR001117S0035 CYBER-HUNTING AT SCALE (CHASE) 18

TA5 proposals must include a robust strategy to support continuous development and integration of TA1 - TA4 software. The continuous development processes should provide researchers and Government evaluators with a consistent baseline for code execution. The proposed continuous development plans should address the following technical objectives:

• Establish continuous integration processes for regular code updates, automated unit testing, and integration testing to ensure that new code does not break the build;

• Perform quality checks prior to staging code for delivery. Quality checks of interest include, but are not limited to functional tests, regression tests, and pre-generated acceptance tests;

• Ensure that code builds can support demonstrations and evaluations;

• Support manual exploratory evaluations and user acceptance tests;

• Provide developers with immediate feedback and status updates; and

• Work with TA1 - TA4 performers to prepare code for evaluation and transition.

Once the Government team selects a component for formal evaluation, TA1 - TA4 performers will be required to submit their source code, build scripts, and unit tests to the TA5 performer on a regular basis. The TA5 performer will verify that each component passes its own unit and integration tests as well as any automated tests developed by the Government evaluation team.

TA5 continuous development tools should facilitate automated execution of TA1 - TA4 technologies and streamline evaluation experiments.

TA5 performers will assist the Government evaluation team in establishing analysis environments for evaluation exercises in the Washington DC Area. As such, TA5 performers should have a local presence in the vicinity of the evaluation exercises to facilitate coordination with the Government team.

F. Evaluation and Schedule

Within CHASE, each technical component will progress through performer-led evaluations and Government evaluations. All proposals should describe a detailed evaluation strategy for the performer-led evaluations. During the performer-led evaluations, proposers should evaluate methods using data from a real network. Proposers should describe how they plan to obtain current network data to test components. Options include, but are not limited to regular data collects from the proposer’s organization, data from multiple organizations in the proposal team, or data from an independent partner. Proposals should describe a compelling strategy to obtain data with appropriate types, fidelity, sensor locations, and scale. In addition to evaluation data, the proposal should describe metrics and a compelling evaluation strategy for each functional component.

During the Government evaluations, performers will demonstrate methods on enterprise network data furnished by the Government. Government evaluations may involve near real-time data feeds. Evaluators may also compare results generated via CHASE algorithms with results generated by cyber hunt teams comprised of human experts in the cyber domain. TA1 teams should propose to provide at least one experienced analyst who could contribute to a cyber hunt team during Government evaluation workshops. TA2 and TA4 teams are likewise encouraged to

HR001117S0035 CYBER-HUNTING AT SCALE (CHASE) 19

provide analysts who could contribute to a cyber hunt team during Government evaluation workshops.

Once a component produces compelling results within the performer-led evaluations, the component may be selected for Government evaluation. In preparation for Government evaluation, performers will be required to perform regular code checks coordinated by TA5. The TA5 team will test to ensure that components run within the Government evaluation environment. Performers will work with the TA5 team to develop demonstration scripts that will enable Government evaluators to apply the method to novel datasets. TA1 - TA4 performers will work with the Government evaluation team to minimize the amount of hand tuning required for the methods to perform optimally.

Potential transition partners may also be involved in evaluation. Components that perform well during the Government evaluation and meet prospective user needs may be selected for transition. Program transition will include regular code checks, test automation, and demonstrations on data of interest to the transition partner. Performers will also work with transition partners and the Government evaluation team to support integration into a transition system on an as needed basis.

Technologies will progress from the performer-led evaluations to the Government evaluations based upon assessments of technology maturity and impact. It is anticipated that TA1 and select TA2 components will begin Government evaluations during year 1. TA2 and TA4 methods will begin Government evaluations in year 2. TA3 components may begin Government evaluations in year 3 or year 4, contingent upon performer-led evaluation outcomes and technology maturity.

Proposers should submit a schedule that is consistent with the maturity of their approaches and the risk reduction required for their concepts. These schedules will be synchronized across performers, monitored, and revised as needed throughout the CHASE program.

For budgeting purposes, use January 1, 2018, as an estimated start date.

G. Metrics

CHASE will evaluate methods on data from real networks, which empirically contain malicious activity. CHASE will develop ranking quality metrics that an average sized security team could use to evaluate security methods in an operational setting. Discounted Cumulative Gain (DCG) is one such metric, because it only requires security teams to provide ground truth assessments for the top N results in a ranked list. Each term of DCG measures the gain, or usefulness, of a result discounted by its position in the list. Thus, DCG measures the quality of ranked lists.

CHASE will develop variations on ranking quality metrics, such as DCG, to assess how accurately TA1 algorithms detect real cyber threats and characterize threats by type. For TA3, CHASE will measure how accurately algorithms detect globally visible, locally invisible activity. During the first six months, CHASE will use ranking quality metrics to establish

HR001117S0035 CYBER-HUNTING AT SCALE (CHASE) 20

baseline performance. Subsequent phases will improve upon initial baselines and will define additional metrics for the program to evaluate effectiveness.

All proposers should draft and submit metrics appropriate to their respective technical area. For example, TA1 metrics should assess the extent to which detection algorithms push high priority alerts to the top of an alert list. Performers may assume that a human expert will evaluate the accuracy of the top items in a ranked list. Thus, information retrieval metrics might provide a good inspiration for CHASE metrics.