HR001117S0033-Amendment-01.pdf

Broad Agency Announcement Cyber Assured Systems Engineering (CASE)

HR001117S0033

May 12, 2017

Amendment 1

Amended on May 24, 2017

Defense Advanced Research Projects Agency Information Innovation Office 675 North Randolph Street Arlington, VA 22203-2114

HR001117S0033 CYBER ASSURED SYSTEMS ENGINEERING (CASE) 2

Table of Contents

PART I: OVERVIEW INFORMATION

PART II: FULL TEXT OF ANNOUNCEMENT

I. Funding Opportunity Description

A. Introduction

B. Program Description

C. Technical Areas

D. Schedule and Milestones

E. Deliverables

F. Government-Furnished Property/Equipment/Information

G. Intellectual Property

II. Award Information

A. Awards

B. Fundamental Research

C. Disclosure of Information and Compliance with Safeguarding Covered Defense Information Controls

III. Eligibility Information

A. Eligible Applicants

B. Organizational Conflicts of Interest

C. Cost Sharing/Matching

D. Other Eligibility Requirements

IV. Application and Submission Information

A. Address to Request Application Package

B. Content and Form of Application Submission

C. Submission Date and Time

D. Funding Restrictions

E. Other Submission Requirements

V. Application Review Information

A. Evaluation Criteria

B. Review and Selection Process

VI. Award Administration Information

A. Selection Notices

B. Administrative and National Policy Requirements

HR001117S0033 CYBER ASSURED SYSTEMS ENGINEERING (CASE) 3

C. Reporting

VII. Agency Contacts

VIII. Other Information

A. Frequently Asked Questions (FAQs)

B. Proposers Day

C. Submission Checklist

D. Associate Contractor Agreement Clause (ACA)

HR001117S0033 CYBER ASSURED SYSTEMS ENGINEERING (CASE) 4

PART I: OVERVIEW INFORMATION

Federal Agency Name: Defense Advanced Research Projects Agency (DARPA), Information Innovation Office (I2O)

Funding Opportunity Title: Cyber Assured Systems Engineering (CASE)

Announcement Type: Initial Announcement

Funding Opportunity Number: HR001117S0033

Catalog of Federal Domestic Assistance Numbers (CFDA): Not Applicable

Dates o Posting Date: May 12, 2017 o Proposers Day: May 15, 2017 o Proposal Due Date: June 28, 2017, 12:00 noon (ET) o BAA Closing Date: June 28, 2017, 12:00 noon (ET)

Anticipated Individual Awards: DARPA anticipates multiple awards in technical areas (TAs) 1, 2, 3, 4, 5, and 6; and a single award in TA7.

Types of Instruments that May be Awarded: Procurement contracts or Other Transactions

Agency Contacts o Technical POC: Dr. Raymond Richards, Program Manager, DARPA/I2O o BAA Email: CASE@darpa.mil o BAA Mailing Address:

DARPA/I2O

ATTN: HR001117S0033

675 North Randolph Street Arlington, VA 22203-2114 o I2O Solicitation Website: http://www.darpa.mil/work-with-us/opportunities mailto:CASE@darpa.mil http://www.darpa.mil/work-with-us/opportunities

HR001117S0033 CYBER ASSURED SYSTEMS ENGINEERING (CASE) 5

PART II: FULL TEXT OF ANNOUNCEMENT

I. Funding Opportunity Description

DARPA is soliciting innovative research proposals in the area of advanced design and analysis tools that can establish cyber resiliency as an explicit system property. Proposed research should investigate innovative approaches that enable revolutionary advances in science, devices, or systems. Specifically excluded is research that primarily results in evolutionary improvements to the existing state of practice.

This Broad Agency Announcement (BAA) is being issued, and any resultant selection will be made, using procedures under Federal Acquisition Regulation (FAR) 6.102(d)(2) and 35.016.

Any negotiations and/or awards will use procedures under FAR 15.4. Proposals received as a result of this BAA shall be evaluated in accordance with evaluation criteria specified herein through a scientific review process.

DARPA BAAs are posted on the Federal Business Opportunities (FBO) website (https://www.fbo.gov/).

The following information is for those wishing to respond to this BAA.

A. Introduction

Embedded computing systems are ubiquitous in critical infrastructure, vehicles, smart devices and military systems. Conventional wisdom once held that cyberattacks against embedded systems were not a concern since they seldom had traditional networking connections on which an attack could occur. However, attackers have learned to bridge air gaps that surround the most sensitive embedded systems, and network connectivity is now being extended to even the most remote of embedded systems. In short, embedded systems are now subject to cyberattacks, either as the end goal of the cyber assailant or as a means to a greater end, and there is a critical need to protect and defend embedded systems in a cyber-context. The mechanisms currently employed to secure embedded systems include development of software using cyber best practices, adapting mechanisms from information technology (IT) systems, and penetration testing followed by patching; however, these methods have proven to be generally ineffective.

Systems engineers are responsible for producing designs with a required set of functional and non-functional properties. Non-functional properties include properties such as safety, performance, durability, survivability, availability, manufacturability, maintainability, etc. The system designer is responsible for managing conflicts and making tradeoffs between the desired non-functional properties.

Critical systems are built by requirements-based engineering and it is an accepted axiom of systems engineering that requirements are positive, testable statements about the system -statements on the systems’ functional behaviors and non-functional properties often captured as ‘shall’ statements. This style of engineering has proven to be ineffective in engineering cyber resilient systems because cyber requirements are often statements on what the system should not do, i.e., shall not statements.

https://www.fbo.gov/

HR001117S0033 CYBER ASSURED SYSTEMS ENGINEERING (CASE) 6

B. Program Description

The goal of CASE is to develop the necessary design, analysis and verification tools to allow system engineers to design-in cyber resiliency and manage tradeoffs as they do the other non-functional properties when designing complex embedded computing systems. Cyber resiliency means that the system is tolerant to cyberattacks in the same way that safety critical systems are tolerant to random faults - they recover and continue to execute their mission function.

Achieving this goal requires research breakthroughs in:

the elicitation of cyber resiliency requirements before the system is built;

the design and verification of systems when requirements are not testable (i.e., when they are expressed in shall not statements);

tools to automatically adapt software to new non-functional requirements; and techniques to scale and provide meaningful feedback from analysis tools that reside low in the development tool chain.

The CASE program will operate over three phases. Phase 1 will be 12 months in duration, consisting of initial research and tool development. Option Phase 2 will be 18 months in duration, consisting of technology enhancements and testing of the emerging tools on one or more experimental platform(s). Option Phase 3 will be 18 months in duration, during which CASE tools will be used to re-engineer selected representative demonstration systems for cyber resiliency.

For the duration of the program, tool assessments will be completed every six months with a baseline assessment of the platforms(s) to be completed at month six of the program.

Subsequent tool assessments will judge technical progress relative to the baseline assessment;

these assessments will be used in making any potential option exercise decisions at the end of every phase. In addition, platform assessments to measure progress toward program goals will be completed at the end of Phase 2 and Phase 3.

The CASE program has seven Technical Areas (TAs):

TA1: Derive Cyber Requirements TA2: Design for Resiliency TA3: Support Legacy Components TA4: Explainable Formal Methods TA5: Engineering Workflow Integration TA6: Platform Provider TA7: Evaluation

The relationships between technical areas 1-4 and a rough outline of a notional system design process is shown in Figure 1.

HR001117S0033 CYBER ASSURED SYSTEMS ENGINEERING (CASE) 7

Figure 1: Technical Areas 1-4 supporting a CASE system design process

Each proposal should address a single technical area with Phase 1 as the baseline effort and Phases 2 and 3 proposed as options. A submitter may propose to any number of technical areas.

However, please note that a TA7 performer may not participate on any other technical area.

Multiple awards are envisioned in TAs 1-6, with a single award in TA7.

Close coordination among the various technical areas is critical to program success; the work in CASE will be centered around the TA6 platform(s). In Phase 1, TA1, TA2, TA3, and TA4, will provide test inputs for their tools. The tool test inputs are to evaluate the technical progress towards their respective evaluation goals. The tests should be challenging enough to demonstrate the tools’ capabilities and to measure their limitations. TA7 will review and potentially augment these tests. In subsequent phases, performers in TA1, TA2, TA3 and TA4 will be required to tune their research activities to support the needs of the experimental platform(s) and the demonstration platform(s). The experimental platform(s) will be available for all performers, providing challenge problems and use as a technology testbed. The demonstration platform(s) will only be accessible by TA6 performer(s). Similarly, performers in TA7 will use the TA6 experimental platform(s) as a test bed to exercise their metrics and analysis tools for execution on the demonstration platform in Phase 3. In turn, TA6 performers will tune their plans for their experimental system(s) to optimize the research opportunities for the TA1, TA2, TA3, TA4 and TA7 performers and for the integration efforts of TA5 to the extent that such flexibility makes sense in the context of the experimental platform(s) being built and the demonstration platform(s).

Because no TA can succeed on its own, proposals must include a Collaborative Research Team Concept section that describes how the work would fit within the context of potential collaborative research teams (see Section IV.B.1.a.v). The proposer’s Collaborative Research Team Concept section provides the opportunity to outline the dependencies between the work proposed and work in other TAs. The Collaborative Research Team Concept section should describe:

a) the working assumptions about features or capabilities their proposal requires from any of the other TAs; and

b) other additional features or capabilities that may be accommodated within the scope of their proposed approach.

Table 1 specifies necessary collaboration and interactions between technical areas.

HR001117S0033 CYBER ASSURED SYSTEMS ENGINEERING (CASE) 8

TA Collaborations and Interactions 1 TA2: Concur on format, content and structure of derived cyber requirements

TA4: Work to integrate scalable analysis engines in Phase 2 TA5: Provide instruction on the proposed requirements derivation tool; facilitate workflow integration in Phase 2; provide tool enhancements and modifications in Phase 3 TA6: Provide instruction on necessary inputs to the requirements derivation tool TA7: Facilitate fair assessment of Cyber Requirements tools

2 TA1: Concur on format, content, and structure of derived cyber requirements TA3: Concur on format, content and structure of derived component requirements TA4: Provide instruction on defining analysis problem sets and their scalability; develop interfaces for the exchange of contextual information and analysis feedback; integrate enhanced analysis engines in Phase 2; collaborate on tool enhancements in Phase 3

TA5: Provide instruction on developed tools; facilitate workflow integration in Phase 2; provide enhancements and modifications in Phase 3

TA6: Provide an understanding and working knowledge of developed tools TA7: Facilitate fair assessment of Design for Resiliency tools

3 TA2: Concur on format, content and structure of derived component requirements TA4: Provide instruction on defining analysis problem sets and their scalability TA5: Provide instruction on developed tools; facilitate workflow integration in Phase 2;

provide enhancements and modifications in Phase 3 TA6: Provide instruction on developed tools TA7: Facilitate fair assessment of Legacy Software tools

4 TA1: Provide analysis engines with improved scalability for targeted problems sets TA2: Provide analysis engines with improved scalability for targeted problems sets; develop interfaces for the exchange of contextual information and analysis feedback TA3: Provide analysis engines with improved scalability for targeted problems sets TA6: Collaborate on tool semantic feedback TA7: Facilitate fair assessment of tool scalability

5 TAs 1-34: Build knowledge and understanding of developed tools and develop detailed plans for facilitation of workflow integration

TA6: Demonstrate an understanding of the proposed system engineering process to facilitate integration of tools; provide input as to engineering cost metrics captured in Phase 3

6 TA1: At the beginning of Phase 2, provide the necessary design data of the experimental platform to derive cyber requirements. Provide subject matter expertise on the experimental platform in Phase 2

TA2: At the beginning of Phase 2, provide the necessary design data of the experimental platform for testing design tools; provide subject matter expertise on the experimental platform in Phase 2

TA3: At the beginning of Phase 2, provide the necessary experimental platform software to test legacy software tools; provide subject matter expertise on the experimental platform

TA4: Collaborate on tool sematic feedback TA5: Provide subject matter expertise on system engineering workflow TA7: Provide subject matter expertise on experimental and demonstration platform

7 All: Develop challenge problems and benchmarks; Perform tool and platform assessments at the conclusion of Phases 1, 2 and 3

Table 1: Collaborations and Interactions among Technical Areas

HR001117S0033 CYBER ASSURED SYSTEMS ENGINEERING (CASE) 9

C. Technical Areas

This section contains detailed descriptions of the research sought in each of the technical areas.

TA1: Derive Cyber Requirements

The goal of this technical area is to build a tool(s) that takes an early representation of the system as an input and produces a set of cyber requirements. The proposer will determine what early representation they will need as input. Possible representations include, but are not limited to requirements documents and functional design models. One research outcome of this technical area is how to determine what details about a system are needed to derive requirements. The answer to this question will affect how the TA1 tools are integrated into a system design workflow.

The resulting set of requirements, when implemented in the system, should protect the system against many classes of cyber threats. In order to avoid redundancy and over constraining the design, the number of requirements should be kept to a minimum. There is a balance to be struck between minimizing the number of requirements and producing a usable set of requirements that provide broad threat coverage. For example, a requirement stated as “The system shall execute its functional specification and nothing else,” covers most classes of cyber threats but is difficult to formally verify and therefore, is not usable. The evaluation goals for proposals to this technical area include broad coverage of the threat space and number of derived requirements. Strong proposals will include an argument on threat space coverage.

During Phase 1, TA1 performers will provide all the necessary test inputs to perform the Phase 1 tool assessments. The assessment goals can be found in Table 2. For subsequent phases, TA7 will provide tool assessment test inputs. It is anticipated that the TA1 level of effort in Phase 3 will be drastically reduced from the previous two phases.

Strong TA1 proposals will:

Describe system representations believed to be useful for capturing cyber requirements.

This should include describing the necessary level of detail needed to support their analysis to derive cyber requirements.

Describe the approach for deriving a minimal set of cyber requirements from analysis results. A detailed use case example is required.

Describe the class of attacks that are mitigated by the derived requirements, and of what types of attacks not covered by the proposed approach. Describe how attacks not addressed by the approach could be mitigated.

Provide detailed plan to meet commitments to other TAs, as specified in Table 1.

TA2: Design for Resiliency

The goal of this technical area is to provide tools for system design, working in a design-with-verification paradigm with respect to the cyber requirements. The output of the human-driven design activity supported by TA2 tools is a designed system with verified cyber properties, a set of cyber requirements levied against components in the system design and a set of tests to validate the analysis against the system implementation.

HR001117S0033 CYBER ASSURED SYSTEMS ENGINEERING (CASE) 10

Proposers should assume a model-based system design paradigm. Proposers should state which modeling language they will develop their technology against in Phase 1. DARPA will choose the modeling language(s) used in Phases 2 and 3 of the program.

As the system design is developed, TA2 performers will provide system designers with feedback on the ability of the proposed design to meet the identified cyber requirements. This includes identifying inherent weaknesses in the design and suggesting resiliency-improving design patterns while also providing input on the impact of design decisions on the ability to meet requirements.

This TA encompasses a variety of potential topics and it is not expected that proposals cover every topic; additionally, TA2 proposals may include topics not discussed in this document.

However, proposals that include topics not discussed here must include a strong argument on how their topic supports the goals for this technical area and how their approach contributes to improving the assessment metrics for this technical area, specified in Table 2. Topics of interest in this technical area include:

Verification of cyber requirements with feedback to the designer:

Perform formal verification of requirements over designs and partial designs. Provide feedback to the designer on the analysis. This can include but is not limited to: which component properties are being used to prove particular cyber requirements; when a verification of a requirement fails; what part of the design that should be reworked to meet the requirement; and which stipulated component properties do not support cyber requirements.

Validation of design model:

CASE verification of cyber requirements occurs through the analysis of abstract design models. This analysis typically includes a set of assumptions and properties provided by constituent components. To increase assurance it is necessary to validate the analysis against the system implementation once the system is complete. One of the goals for TA2 is the creation of validation tests from the design model to be run on the completed system as part of system validation.

Runtime assurance for requirements:

Cyber requirements that are onerous to verify can be assured by run-time mechanisms;

that is, mechanisms to validate necessary properties hold during system execution. A cyber property failure at run-time should trigger a recovery mechanism designed into the system. The design for the runtime assurance needs to be included in the cyber assurance guarantees provided by the system design.

Design analysis for structural patterns that generally support cyber resiliency:

Identify structural design patterns that support TA2 goals and also analyze that design and suggest applications of design patterns that help to meet cyber requirements with minimal requirements burden on legacy software. Examples of design patterns include redundancy and load balancer.

During Phase 1, TA2 performers will provide all necessary test inputs to perform the Phase 1 tool assessments. The assessment goals can be found in Table 2. For subsequent phases, TA7 will provide tool assessment test inputs. It is anticipated that the TA2 level of effort in Phase 3 will be drastically reduced from the previous two phases.

HR001117S0033 CYBER ASSURED SYSTEMS ENGINEERING (CASE) 11

Strong TA2 proposals will:

Describe the proposed tool’s intended function, and how it helps in improving the TA2 assessment metrics. In a use case example, provide detailed description of the use of the tool and describe the proposed tool’s expected inputs and outputs.

Discuss which lower level provers are used. Descriptions of how feedback from these underlying tools can be made relevant to the human designer.

Describe limitations to the proposed tool. Substantiate and quantify the scalability of the proposed approach. Describe the necessary human inputs to the proposed tool.

Describe the inputs and outputs of the proposed tool. Discuss the level of detail that must exist in the system design before the tool can be applied to that design, e.g., what information does that tool need about the system being designed? Discuss the proposed modeling language in terms of strengths and weaknesses, e.g., how do the language constructs facilitate the proposed approach?

Provide a detailed plan to meet commitments to other TAs, as specified in Table 1.

TA3: Support Legacy Components

The goals of TA3 are to develop tools to analyze legacy software against new requirements, adapt legacy software to meet the new requirements or insert run-time assurance mechanisms into the legacy software. These run-time assurance mechanisms are to trigger system-level recovery mechanisms. It is likely that these newly imposed requirements are not functional requirements; the legacy component was chosen because of its functionality. Approaches that operate on software binaries are preferred over source code approaches, but strong source code approaches will be considered. Re-implementation of legacy software is beyond the scope of the CASE program.

During Phase 1, TA3 performers will provide all necessary test inputs to perform the Phase 1 tool assessments. The assessment goals can be found in Table 2. For subsequent phases, TA7 will provide tool assessment test inputs. It is anticipated that the TA3 level of effort in Phase 3 will be drastically reduced from the previous two phases.

Strong TA3 proposals will:

Describe a technical approach for analyzing legacy software against new requirements, adapting legacy software to meet new requirements, or inserting run-time assurance mechanisms into legacy software.

Characterize the class of requirements that the proposed approach verifies; provide examples of verification of these requirements along with explanations substantiating the soundness of the proposed analysis. Discuss the scalability limits of the proposed approach.

Characterize the class of requirements that the proposed approach can meet with design-time modifications, to include a use case example.

Characterize the class of requirements that the proposed approach meets with run-time assurance along with a use case example.

Provide detailed plan to meet commitments to other TAs, as specified in Table 1.

HR001117S0033 CYBER ASSURED SYSTEMS ENGINEERING (CASE) 12

TA4: Explainable Formal Methods

It is anticipated that the work done by performers in technical areas 1-3 will result in tool chains that have, at their lowest level, an analysis engine. Historically, such tools chains have presented two main challenges. First, the analysis engine is far removed from the domain at the top of the tool chain. The output from the tools (other than the binary pass/fail output) is difficult to relate back to the domain in which the user of the tool chain is working. Second, the analysis tools near the bottom of the tool chain tend to be the limiting factor in scalability.

This TA seeks approaches to address both of these challenges. First, this TA looks to bridge the semantic gap between the analysis domain and the design domain. This challenge is not unlike the challenge that was surmounted by attaching symbolic debugging tools to executable binaries, thereby bridging the gap between the binary domain and the high-level language domain. The innovation of symbolic debugging allowed software programmers to debug binaries at the same level of abstraction that they wrote the implementation. Likewise, TA4 should present relevant analysis feedback to the system design in the same semantic context that is being used in design.

Second, this TA looks to improve the scalability of analysis engines for the set of problems conforming TAs 1-3. While it is not necessary or desired to improve the general-case scalability of such tools, targeting scalability to a class of analysis problems will require a strong argument that the class of problems being considered is relevant to the CASE program. TA4 proposals my address either or both of these challenges.

During Phase 1, TA4 performers will provide all the necessary test inputs to perform the Phase 1 tool assessments. The assessment goals can be found in Table 2. For subsequent phases, TA7 will provide tool assessment test inputs. It is anticipated that the TA4 level of effort in Phase 3 will be drastically reduced from the previous two phases.

Strong TA4 proposals will:

Provide detailed plan to meet commitments to other TAs, as specified in Table 1.

For proposals addressing the semantic gap between design and analysis:

o Identification of the low-level analysis tool(s) to be enhanced; description of the envisioned enhancements to its user output, as well as necessary additions to its user inputs.

o Use case examples of how the tools’ enhanced output can provide useful information on the design resiliency to higher-level tools and users. Description of the interactions between the higher-level tools and the proposed enhanced tool to include examples of both successful and failed analyzes.

For proposals addressing scalability:

o Description of the approach for improving the tools’ computational efficiency and scalability, discussing the tradeoffs being made, and providing evidence that the approach will be effective. This is to include a characterization of the analysis problems that are the focus of the improvements, as well as a characterization of any analysis capabilities that are being degraded.

o Concrete examples of what problems will be more efficiently solved if the proposed work is successful. If possible, include examples of scaling these problems in complexity while providing a clear-cut description on the metric of complexity (e.g., number of inputs, increase in state, etc.).

HR001117S0033 CYBER ASSURED SYSTEMS ENGINEERING (CASE) 13

TA5: Engineering Workflow Integration

Performers in this technical area have two areas of responsibility: first, tool integration, with the objective of pulling all of the CASE tools together and integrating them into an example engineering workflow and second, facilitating transition of the CASE technologies into practice on a demonstration system.

TA5 performers will develop a plan for integration of technologies developed by technical areas 1-4 into a general systems engineering workflow; this is to be delivered at the end of Phase 1. At the end of Phase 2, TA5 performers will deliver integrated tool suites tailored for the TA6 platform engineering to the TA6 performer(s). Throughout the course of Phase 3, TA5 will support the use of the CASE tools by the TA6 performers, coordinating enhancements and updates with the TA1-4 performers. Throughout the course of all three phases, TA5 performer(s) will study the TA6 engineering process(es) and estimate changes to system development costs when adding CASE tools, and will validate their estimates by suggesting engineering cost metrics to TA6 performers in Phase 3.

To support technology transition, TA5 performers will perform a standardization study to make recommendations on standardization of specification languages and tool interfaces to be delivered by the end of Phase 3. TA5 will look at industry standards that could be amended or modified, or what new standards should be proposed, to facilitate adoptions of CASE developed and similar technologies. TA5 performers will also produce recommendations on the support, sustainment, and commercialization of the CASE developed tools.

The evaluation goal for TA5 is a subjective measure on how effectively the CASE technologies are integrated into the workflow produced by the TA6 performer(s). It is anticipated that this TA will require a low-level of effort in Phase 1 and will ramp up significantly in Phases 2 and 3.

Strong TA5 proposals will:

Provide a detailed technical plan to integrate TA 1-4 tools into a realistic engineering workflow. Include a walkthrough of a detailed use case example.

Describe previous experience in supporting tooling for complex engineering workflows;

describe the workflows and characterization of the types of systems developed with those workflows.

Describe previous experience of supporting design tools as they are being used. Describe the systems engineering process to be used including what commercial tools might be used in that workflow; this should include a discussion of tool compatibility issues and approaches to overcome these issues.

Describe previous experience working with standards bodies. Discuss which standards are relevant and describe previous level of participation in standards bodies.

Discuss previous experience in commercializing emerging tool technologies, along with detailed examples, description of relevant challenges encountered and how they were overcome.

HR001117S0033 CYBER ASSURED SYSTEMS ENGINEERING (CASE) 14

TA6: Platform Provider

The goal of the CASE program is to learn how to build cyber resiliency into complex cyber physical systems. TA6 is where this is tested and demonstrated. Each TA6 proposal should propose two related platforms – an experimental platform and a demonstration platform. In Phase 1, TA6 performer(s) will prepare the necessary experimental platform design artifacts that will be used as a testbed needed by the other performers in Phase 2. In Phase 3, the efforts of all performers will be directed to the demonstration platform and engineering support to redesign a portion of that platform.

The experimental platform should be a distributed embedded system containing multiple computing elements with various interconnecting busses, reflective of the types of architectures found in military platforms. It is acceptable for the experimental platform to be either a lab bench integration of components or a system integration lab. At the end of Phase 1, the TA6 performer(s) will provide the needed design data and experimental platform access to all other performers across the program; this includes the initial design representations required by TAs 1 and 2, and legacy software implementations for TA3. The experimental platform should offer relevant technical challenges and drive the development of CASE technologies toward success.

These challenges include deriving cyber requirements, re-engineering portions of the system for resiliency and accommodating legacy software implementations in a cyber-resiliency framework.

Performers in TAs 1, 2, and 3, will address challenge problems proposed to them by the TA7 performer and focusing on their individual experimental platform. At the end of Phase 1, the Government, in conjunction with TA6 performer(s), will choose the best-fit technologies to modify the experimental platform.

At the end of Phase 2, the TA5 performer(s) will provide the TA6 performer(s) with a tool suite tailored for the existing TA6 engineering workflow; this tool suite will be comprised of a selection of TAs 1-4 tools that have proven their efficacy on the TA6 performers’ experimental platform(s).

In Phase 3, a demonstration platform, anticipated to be a relevant military platform of non-trivial complexity, will be partially re-engineered by the TA6 performer(s) using the integrated CASE tool suite; this will be done in conjunction with TA5 performers who will provide tool support and enhancements. This allows the TA6 performer(s) to control access to critical platform data while demonstrating the usability and transitionability of CASE technologies. The TA6 performer(s) will identify a small set of system-level function(s) to be re-designed using CASE technologies. This set of system-level functions should provide a challenging, yet reasonably sized, subset of the overall system. An example of such a function is the sanitization (or zeroization) function that many military systems have defined in their system-level requirements.

This function removes sensitive mission data from a platform - coordinating and interacting with many, but not all, components. The TA6 performer(s) will collect engineering cost metrics and, using relevant parametric costing models, determine estimates of the cost of making the CASE engineering changes to the system vice using traditional engineering practices.

At the end of Phase 3, the TA6 performer(s) will provide a detailed report on the user experience with the CASE tool suite. Questions to be answered in the report include:

How did the CASE tools integrate into the workflow?

How can the integrated workflow be improved?

HR001117S0033 CYBER ASSURED SYSTEMS ENGINEERING (CASE) 15

How did the CASE tools provide understandable, actionable feedback?

How did the CASE tools help diagnose weakness in the design?

How did the CASE tools provide insights on which portions of the design supported cyber requirements?

How can the CASE tools be improved to provide better insights to the designer?

The evaluation goals for TA6 platform assessments are detailed in Table 2.

Strong TA6 proposals will:

Describe the proposed experimental and demonstration platforms; characterize these systems in terms of complexity, and cyber-attack surfaces. Hypothesize potential cyber adversaries’ goals in attacking the systems.

Explain why the proposed experimental platform was chosen. Describe technical challenges that will drive the other TAs toward success with the demonstration platform.

Describe how the other TAs will be provided access to the experimental platform.

Discuss the development tools that are necessary to modify the experimental platform.

Describe system-level functions that can be re-engineered for both platforms. Describe any restrictions to modifications to either platform, i.e., intellectual property restrictions.

TA7: Evaluation

The Evaluation Team is responsible for conducting tool assessments at six-month intervals throughout the program, as well as platform assessments at the end of phases 2 and 3. The tool assessments will capture and quantify the progress made by technical areas 1, 2, 3, and 4 against their individual assessment goals. The platform assessments will capture and quantify the progress made toward the TA5 and TA6 goals. Table 2 provides technical area specific goals and measures.

(Please see next page for Table 2).

HR001117S0033 CYBER ASSURED SYSTEMS ENGINEERING (CASE) 16

TA Goals Measures of Effectiveness 1 Maximize attack space coverage Minimize number of requirements

Number of evaluation team generated attacks not covered by generated requirements

Number of derived requirements

2 Design meets cyber requirements

Analysis assumptions are validated

Minimize new requirements on legacy components

Percentage of assured requirements either at design-time or at run-time

Percentage of the model validated Number of new requirements on legacy components

3 Correctness of verification analysis

Software modifications meet new requirements

Analysis tested against benchmark suite Modified software tested for compliance to requirements

4 Increased scalability measured against calibrated benchmark

Minimize semantic gap

Analysis tested against a benchmark suite that has a partial order ranking of complexity

Subjective analysis of semantic gap 5 Efficacy of tools within TA6 engineering workflows Subjective report on tool integration

6 Completeness of the threat space coverage

Implementation meets cyber requirements

Capture impact of proposed approach on development costs

Number of demonstrated attacks that are not addressed by cyber requirements

Number of demonstrated attacks against the cyber requirements

Comparison of costs metrics against a parametric costing model

Table 2: Evaluation Goals and Measures

In Phase 1, TA1, TA2, TA3, and TA4, will provide test inputs for their tools. The tool test inputs are to evaluate the technical progress towards their respective evaluation goals. The tests should be challenging enough to demonstrate the tools’ capabilities and to measure their limitations. TA7 will review and potentially augment these tests. This is intended to facilitate the TA7 performer’s learning curve on the various tools, while not burdening them with test input development. However, TA7 will be executing and evaluating the tests in Phase 1. For subsequent assessments in Phases 2 and 3, the TA7 performer will develop test inputs for the various tools drawn from the TA6 experimental platform. This will include a suite of benchmark legacy software components for TA3 to verify and modify and a suite of analysis problems to test the scalability of TA4 tools. Activity by performers in all technical areas other than TA7 will conclude 90-days before the end of Phase 3, allowing ample time for completion of the final tool and platform assessments.

Strong TA7 proposals will:

Describe proposed penetration testing process(es). Discuss pertinent experience performing penetration testing. Describe types of systems tested and vulnerabilities discovered, including systems where no vulnerabilities were uncovered.

Discuss experience performing technology assessments on DARPA programs and other similar R&D activities. Highlight experiences assessing development and analysis tools, including the development of tests and challenge problems.

HR001117S0033 CYBER ASSURED SYSTEMS ENGINEERING (CASE) 17

Discuss experience using and assessing formal methods tools; this should include the range of tools and level of expertise with each tool.

Discuss experience in performing software verification testing, including the development and maintenance of benchmark suites.

Describe the proposed approach for assessing the technical development of TAs 1-6;

include a discussion of timelines and plans for performing assessments and providing measures for discussion at the PI meetings.

D. Schedule and Milestones

DARPA anticipates a January 2, 2018, start date for the CASE program. Subject to the availability of funding, the program is intended to last for 48 months (4 years), and is structured as one 12-month base phase (Phase 1) and two 18-month option phases (Phase 2 and Phase 3), as shown in Figure 2.

Figure 2: Case Schedule and Milestones

There will be a kickoff meeting following finalization of all contract awards, and regular Principal Investigator (PI) meetings beginning at month 7 and every six months thereafter. The goals of the PI meetings will be to: (a) review technical progress; (b) enhance collaboration and coordination across performers; (c) demonstrate accomplishments of each phase; and (d) review plans for the upcoming period. TA7 tool assessments will be conducted one month prior to PI meetings in order for results to be available for discussion.

DARPA will specify the locations for (PI) meetings. For proposal purposes, assume that PI meetings will require 2.5 days, and will alternate locations between the east coast and west coast of the United States. Proposers should include travel to support two collaborative meetings per year with other performers.

In addition to occasional site visits, monthly or bi-monthly teleconference meetings will be held with each PI to enhance communications with the Government team. Should important issues arise between these meetings the Government team will be available to support informal interim technical interchange meetings.

Phase 1 (Tool Development) will focus on initial tool development and core technical capabilities. There will be a baseline assessment of tool capabilities at month 6. A second tool assessment will be completed by month 12. Technical areas 5 and 6 will use this phase to track development of the various tools and prepare for the subsequent phases.

Phase 2 (Tool Enhancement and Integration) will apply the developed tools against the experimental platform and make enhancements to the technologies to meet the challenges

HR001117S0033 CYBER ASSURED SYSTEMS ENGINEERING (CASE) 18

provided by this platform. At the end of this phase, modifications will be made to the experimental platform(s) to improve cyber resiliency. Tool assessments will continue every six months, and an assessment of the experimental platform(s) will occur at the end of the phase.

In Phase 3 (System Engineering), TA6 performers will re-engineer a set of system-level functions on the demonstration platform using tools integrated into their engineering workflow in conjunction with TA5 performers. TA5 performers will provide direct tools support to TA6, working with TAs 1-4 to provide necessary tool enhancements. Tool assessments will continue every six months and a final platform assessment will be made one month before the end of the phase.

E. Deliverables

All performers (all TAs) shall be required to provide the following deliverables:

Commented source code, other necessary data, and accompanying documentation for all software developed under this program.

Regular code drops provided to the Evaluation Team (TA7) to include source code, build scripts, development environments, and test inputs.

For all performers developing tools, a document discussing concepts of operation for conducting evaluations, and user guidance for conducting tool assessments. This is to be delivered 30 days before each tool assessment.

Slide Presentations - Annotated slide presentations shall be submitted within one month after the program kickoff meeting and after each program event (program reviews, assessments, PI meetings, and technical interchange meetings).

Monthly Progress Reports - A monthly progress report describing technical progress made, resources expended, major risks, planned activities, trip summaries, changes to key personnel, and any potential issues and problem areas requiring the attention of the Government team shall be provided within 10 days after the end of each month.

A Technical and Management Work Plan with a project schedule including milestones, updated as required.

End of Phase Report after each program phase. The final report shall concisely summarize the effort conducted.

The final delivery of platforms and tools will be at the end of the month 15 of Phase 3, allowing the evaluation team 90 days to complete final tool and platform assessments.

In addition, the following deliverables are required for particular technical areas:

TA5: A report in the results of the standardization study and a recommend plan on the support, sustainment and commercialization of the developed tools. This is to be delivered at the end of Phase 3.

TA6: A report on the users’ experience with the tools and the impact to system engineering costs, comparing and analyzing collected cost metrics against estimated costs using traditional methods. This is to be delivered at the end of Phase 3.

HR001117S0033 CYBER ASSURED SYSTEMS ENGINEERING (CASE) 19

TA7: A report on each tool assessment and each platform assessment. The reports must include quantitative results, including data collected in digital format, and results of each performer’s performance with respect to the goals and measures outlined in Table 2.

This is to be delivered 90 days after the start of each assessment. Preliminary results of tool assessments shall be presented at each PI meeting.

F. Government-Furnished Property/Equipment/Information

None anticipated.

G. Intellectual Property

The program will emphasize creating and leveraging open source technology. Intellectual property rights asserted by proposers are strongly encouraged to be aligned with open source regimes. See Section VI.B.1 for more details on intellectual property.

A key goal of the program is to establish an open, standards-based toolset that allows for interoperability and integration. This includes the ability to easily add, remove, substitute, and modify software components. This will facilitate rapid innovation by providing a base for future users or developers of program technologies and deliverables. Therefore, it is desired that all noncommercial software (including source code), software documentation, hardware designs and documentation, and technical data generated by the program be provided as deliverables to the Government, with a minimum of Government Purpose Rights (GPR), as lesser rights may adversely impact the lifecycle costs of affected items, components, or processes.

HR001117S0033 CYBER ASSURED SYSTEMS ENGINEERING (CASE) 20

II. Award Information

A. Awards

Multiple awards are anticipated for TA1, TA2, TA3, TA4, TA5 and TA6, with a single award in TA7. Proposers interested in multiple TAs should submit multiple proposals, one for each TA.

Performers in all TAs must demonstrate their willingness and capability to adapt and integrate their efforts and plans into tightly-coupled, multi-performer, collaborative research teams. The level of funding for individual awards made under this solicitation has not been predetermined and will depend on the quality of the proposals received and the availability of funds. Awards will be made to proposers whose proposals are determined to be the most advantageous to the Government, all factors considered, including the potential contributions of the proposed work, overall funding strategy, and availability of funding. See Section V for further information.

The Government reserves the right to:

select for negotiation all, some, one, or none of the proposals received in response to this solicitation;

make awards without discussions with proposers;

conduct discussions with proposers if it is later determined to be necessary;

segregate portions of resulting awards into pre-priced options;

accept proposals in their entirety or to select only portions of proposals for award;

fund proposals in increments and/or with options for continued work at the end of one or more phases;

request additional documentation once the award instrument has been determined (e.g., representations and certifications); and remove proposers from award consideration should the parties fail to reach agreement on award terms within a reasonable time or the proposer fails to provide requested additional information in a timely manner.

Proposals selected for award negotiation may result in a procurement contracts or Other Transactions (OT) depending upon the nature of the work proposed, the required degree of interaction between parties, and other factors.

Proposers looking for innovative, commercial-like contractual arrangements are encouraged to consider requesting Other Transactions. To understand the flexibility and options associated with Other Transactions, consult www.darpa.mil/work-with-us/contract-management#OtherTransactions.

In all cases, the Government contracting officer shall have sole discretion to select award instrument type, regardless of instrument type proposed, and to negotiate all instrument terms and conditions with selectees. DARPA will apply publication or other restrictions, as necessary, if it determines that the research resulting from the proposed effort will present a high likelihood of disclosing performance characteristics of military systems or manufacturing technologies that are unique and critical to defense. Any award resulting from such a determination will include a requirement for DARPA permission before publishing any information or results on the program. For more information on publication restrictions, see the section below on Fundamental Research.

http://www.darpa.mil/work-with-us/contract-management#OtherTransactions http://www.darpa.mil/work-with-us/contract-management#OtherTransactions

HR001117S0033 CYBER ASSURED SYSTEMS ENGINEERING (CASE) 21

B. Fundamental Research

It is DoD policy that the publication of products of fundamental research will remain unrestricted to the maximum extent possible. National Security Decision Directive (NSDD) 189 defines fundamental research as follows:

‘Fundamental research’ means basic and applied research in science and engineering, the results of which ordinarily are published and shared broadly within the scientific community, as distinguished from proprietary research and from industrial development, design, production, and product utilization, the results of which ordinarily are restricted for proprietary or national security reasons.

As of the date of publication of this BAA, the Government expects that program goals as described herein may be met by proposers intending to perform fundamental research and proposers not intending to perform fundamental research or the proposed research may present a high likelihood of disclosing performance characteristics of military systems or manufacturing technologies that are unique and critical to defense. Based on the nature of the performer and the nature of the work, the Government anticipates that some awards will include restrictions on the resultant research that will require the awardee to seek DARPA permission before publishing any information or results relative to the program.

Proposers should indicate in their proposal whether they believe the scope of the research included in their proposal is fundamental or not.