HR001119S0074.pdf

Broad Agency Announcement Intent-Defined Adaptive Software (IDAS)

HR001119S0074

July 10, 2019

Defense Advanced Research Projects Agency Information Innovation Office 675 North Randolph Street Arlington, VA 22203-2114

HR001119S0074 INTENT-DEFINED ADAPTIVE SOFTWARE (IDAS) 2

Table of Contents Part I: Overview Information……………………………………………………………………...………4

Part II: Full Text of Announcement………………………………………………………………………..5

I. Funding Opportunity Description

A. Introduction & Background

B. Program Description & Scope

C. Program Structure

D. Technical Areas

E. Schedule/Milestones

F. Deliverables

G. Government-furnished Information

H. Intellectual Property

II. Award Information

A. Awards

B. Fundamental Research

C. Disclosure of Information and Compliance with Safeguarding Covered Defense Information Controls

III. Eligibility Information

A. Eligible Applicants

B. Organizational Conflicts of Interest

C. Cost Sharing/Matching

D. Other Eligibility Requirements

IV. Application and Submission Information

A. Address to Request Application Package

B. Content and Form of Application Submission

C. Submission Dates and Times

D. Funding Restrictions

E. Other Submission Requirements

V. Application Review Information

A. Evaluation Criteria

B. Review and Selection Process

VI. Award Administration Information

A. Selection Notices

B. Administrative and National Policy Requirements

HR001119S0074 INTENT-DEFINED ADAPTIVE SOFTWARE (IDAS) 3

C. Electronic Funds Transfer information (e.g., proposer’s bank account number, routing number, and bank phone or fax number).Reporting

VII. Agency Contacts

VIII. Other Information

A. Frequently Asked Questions (FAQs)

B. Proposers Day

C. Submission Checklist

D. Associate Contractor Agreement (ACA)

HR001119S0074 INTENT-DEFINED ADAPTIVE SOFTWARE (IDAS) 4

PART I: OVERVIEW INFORMATION

Federal Agency Name: Defense Advanced Research Projects Agency (DARPA), Information Innovation Office (I2O)

Funding Opportunity Title: Intent-Defined Adaptive Software (IDAS)

Announcement Type: Initial Announcement

Funding Opportunity Number: HR001119S0074

Catalog of Federal Domestic Assistance Numbers (CFDA):

12.910 Research and Technology Development

Dates o Posting Date: July 10, 2019 o Proposers Day: July 9, 2019 o Abstract Due Date: July 24, 2019, 12:00 noon (ET) o Proposal Due Date: September 10, 2019, 12:00 noon (ET) o BAA Closing Date: September 10, 2019, 12:00 noon (ET)

Anticipated Individual Awards: DARPA anticipates multiple awards for Technical Area 1 and a single award for Technical Areas 2 – 4.

Types of Instruments that May be Awarded: Procurement contracts or cooperative agreements.

Agency Contacts o Technical POC: Jacob I. Torrey, Program Manager, DARPA/I2O o BAA Email: IDAS@darpa.mil o BAA Mailing Address:

DARPA/I2O

ATTN: HR001119S0074

675 North Randolph Street Arlington, VA 22203-2114 o I2O Solicitation Website: http://www.darpa.mil/work-with-us/opportunities mailto:IDAS@darpa.mil http://www.darpa.mil/work-with-us/opportunities

HR001119S0074 INTENT-DEFINED ADAPTIVE SOFTWARE (IDAS) 5

PART II: FULL TEXT OF ANNOUNCEMENT

I. Funding Opportunity Description

DARPA is soliciting innovative research proposals to create novel software engineering technologies that enable automated adaptation of the resulting software system to radical changes in requirements and/or the computational environment. Proposed research should investigate innovative approaches that enable revolutionary advances in science, software technologies, or systems. Specifically excluded is research that primarily results in evolutionary improvements to the existing state of practice.

This Broad Agency Announcement (BAA) is being issued, and any resultant selection will be made, using procedures under Federal Acquisition Regulation (FAR) 6.102(d)(2) and 35.016.

Any negotiations and/or awards will use procedures under FAR 15.4 (or 32 CFR § 200.203 for cooperative agreements). Proposals received as a result of this BAA shall be evaluated in accordance with evaluation criteria specified herein through a scientific review process.

DARPA BAAs are posted on the Federal Business Opportunities (FBO) website (https://www.fbo.gov/) and the Grants.gov website (https://www.grants.gov/).

The following information is for those wishing to respond to this BAA.

A. Introduction & Background

The Department of Defense (DoD) is highly dependent on software. The increasing complexity and scale of this software is creating new attack surfaces for adversaries and reducing the DoD’s ability to react to new threats1. As currently practiced the cost of software engineering is constraining the ability of the U.S. Government to deploy new software-based capabilities, as more than 70% of the federal information technology (IT) budget is dedicated to operations and management (O&M)2. Software today is brittle with respect to changes in requirements and/or computing resources, requiring frequent and ultimately unaffordable modernization efforts to maintain adequate functionality.

Management of complexity is a central problem in software engineering. A common approach is concretization, in which the software engineer chooses from a set of apparently or almost equivalent options a particular choice that enables the resulting code to compile. Ideally, the engineer makes this choice by considering its impact on the system when running under a variety of likely future conditions. For example, the engineer might assume that network latency will never exceed a particular value, select a 16-bit integer to represent the horizontal velocity of a rocket, or implement data management tools using the Application Programming Interfaces (APIs) of a specific cloud service offering. Concretization makes the process of software development tractable, allowing the engineer to define and implement an architecture, split up the development tasks into manageable parts, establish conventions to enable their integration, and integrate them into a cohesive software system. However, this process occurs at design time, 1 E.g., many mission-critical tactical IT systems in the DoD are still reliant on Windows XP 2 White House Office of Management and Budget, "Information Technology," US Federal Government, Washington, DC, 2017.

https://www.fbo.gov/

HR001119S0074 INTENT-DEFINED ADAPTIVE SOFTWARE (IDAS) 6

when information about all possible future environments of the operational system is not available to guide the choice of concrete values or types. A substantial fraction of these choices will be wrong at some point in the system’s lifecycle, in the sense that they will require adaptation to unanticipated requirements or changes in computing resources. Concretization also creates inefficiencies in software, as engineers tend to choose conservative values that account for worst-case scenarios or future uncertainty but waste resources when handling the far more common average cases. Software engineers make dozens of concretization decisions every day of development. In the vast majority of cases, they do not document the rationale for their choice of a particular concrete value, so the context of their decisions is lost. Even if these rationales are documented, they are likely expressed in human-readable format that is not semantically accessible to automation and can quickly become out-of-sync with the code.

Agile software development methodologies address the problem of complexity in software design by structuring development in short sprints of two to four weeks that culminate in demonstrations of developed capabilities to the end users of the system. This process can produce better concretization decisions, because the engineers and end users can spot problems earlier in the development process, when the rationale for particular choices is still readily available, but it does not prevent concretizations and the brittleness these decisions cause in software systems. In the context of some unknown future change in requirements, each sprint adds more technical debt, increasing the cost of future maintenance and adaptation.

As large software projects mature, they often become locked to specific versions of the resources on which they depend. System maintainers of these projects often struggle to keep up with changes in protocols and APIs in software system dependencies. To fix critical security weaknesses, developers maintaining large software systems often have to adapt or backport the fixes to old software, increasing their work load. Upgrading systems to take advantage of modern computational resource features often requires many of hours of effort to either completely reengineer the software, or meticulously change and test interfaces across dozens or hundreds of dependencies.

B. Program Description & Scope

The Intent-Defined Adaptive Software (IDAS) program will develop technologies that capture the intentions of software engineers, to enable rapid code generation to support the continual adaptation of DoD software-enabled systems. In practice, changes in requirements and resources are a common occurrence. The program will develop new methods for representing the intent of software and its abstract constraints separately from its concrete instantiation, and will leverage automated methods to adjust to a particular instance.

Technologies developed on the IDAS program will enable rapid adaptation of software to changes in requirements and/or operating environments. A feasibility study framed one possible approach in terms of a constraint satisfaction problem. This study explored problems such as efficiently storing large amounts of streaming data in a distributed cluster, while maintaining a guarantee of lookup within a fixed time bound. A set of constraints defined the problem that the resulting software would address, and the software was specified in terms of the programmer’s intentions. A partially-automated software generation process took these constraints and intentions as input and produced compiled code. To adapt to changes in either requirements or computational resources, system maintainers were able to modify the constraints and/or the

HR001119S0074 INTENT-DEFINED ADAPTIVE SOFTWARE (IDAS) 7

intentions of the software system, and then generate a new version of the software with limited human effort.

Key findings from the study include:

1. Creating separate representations of the problem to be addressed by the software (e.g., the constraints on a viable solution), and the actual solution (e.g., a specific software architecture and program that addresses the problem), is essential for scalability. If the problem constraints include aspects of the intended solution (e.g., opting for the use of a particular algorithm that now implicitly fixes certain data types and concurrency requirements when there are viable alternatives that do not add such constraints), the resulting constraint satisfaction problem is drastically more difficult to solve, because such aspects ramify the dimensions of the problem, creating a more complex search space.

2. Representing the program as a set of higher-level programmer intentions rather than just as specific, concrete source code that addresses a current set of problem constraints is essential for enabling rapid future changes.

3. APIs and pre-defined interfaces are concretizations that hamper software flexibility.

This third point requires some unpacking. Conventional APIs and interfaces hide the underlying implementation of a software module from its users, creating an abstraction boundary that enables the module users and module developers to conduct their development activities independently. For this to work in practice, however, the APIs must not change in ways that invalidate the assumptions of the API users. The specification of an API therefore requires extensive commitments to design choices, in other words, many concretizations and conventions (typically described in human-readable documentation). The feasibility study explored the use of dynamic controllers embedded within software modules as a way to provide an invariant abstraction (i.e., an API) that is independent of a specific implementation, thus allowing for greater flexibility of the resulting codebase. Such a controller can either take inputs at runtime or be adapted at compile time to meet new requirements and/or use new computational resources.

The controllers were automatically generated and allowed for compositional stability analysis to verify that the specific dependencies between modules were upheld, separately from the specific implementation automatically generated.

To illustrate how a controller can generate a simple solution that addresses uncertainty, consider a restaurant. The restaurateur must provide tables for parties coming to dine, and desires to maximize the efficiency of table use, via a seating algorithm. A naïve approach would be to statically over-provision for the expected worst-case scenario (e.g., New Year’s Eve) by ordering many large tables that can accommodate parties of any size up to a set maximum, say 12 people (note the concretization here in the problem description). On most nights, many of the tables would be under- or unutilized. However, the seating algorithm would have to make efficient use of the tables on each night, as it would have no way to forecast a surge in demand.

The simplest seating algorithm would assign parties as they arrive to the first open seat at a table.

However, this would split some parties across two tables, creating customer dissatisfaction. The seating algorithm must therefore take into account the constraint that each party must be seated at the same table. To maximize efficiency, the restaurant could wait to seat people until enough parties of the right sizes had arrived to optimally occupy each table (the NP-hard bin packing problem). However, customers dislike long wait times, particularly when they can see empty tables in the restaurant, so the seating algorithm will be unable to efficiently use the statically

HR001119S0074 INTENT-DEFINED ADAPTIVE SOFTWARE (IDAS) 8

provisioned table resources.

A controller, or in this example a maître d′, can provide a more efficient solution. Rather than statically allocating an abundance of overly large tables, the restaurant can get by with a collection of smaller (e.g., two-seat) tables. The controller then adapts to the flow of customers by dynamically moving sufficient tables together to accommodate parties of any size. In effect, the controller provides the seating algorithm with a way to dynamically allocate optimally-sized table resources as needed. As a result, the complexity of the seating algorithm no longer scales with the size of the table resources (i.e., this approach is not tied to the number of tables in the restaurant). Such a seating algorithm could serve the same number of customers with a much smaller footprint, while respecting the problem constraints of keeping parties together and seating them soon after they arrive. This solution focuses on only the essential constraints of the problem, while ignoring don’t care properties, such as the size of tables or the table arrangement at the end of the night, which enables it to scale up to handle complex problems.

When using a constraint satisfaction approach, solving time can scale exponentially, limiting the size of problems that can be attempted. There are two conventional approaches for simplifying a problem: (a) removing constraints, and (b) adding degrees of freedom. Removing constraints makes the problem easier; for example, removing the constraint that parties should sit together simplifies the process of producing an acceptable seating arrangement. However, adding degrees of freedom can (counter-intuitively) at times implicitly introduce new dimensions to the problem, making the solving process more difficult. For example, adding degrees of freedom by having more tables in a restaurant (more options for an algorithm to seat parties) actually increases solve time compared to the controller-based approach that can ignore the implied dimensions of total seating capacity and allocation to specific table sizes. Premature concretization into a problem (e.g., bin packing to fixed sized resources) tends to increase the difficulty of generating a solution by adding constraints (fixed table sizes) that now scale in the size of the solution (restaurant size) rather than solely in the size of the problem. A controller-based solution will work for any (reasonable) number of tables, because the problem and the solution representations are not as tightly coupled.

In addition to the feasibility study, other research efforts in this field3 have identified the following elements of an approach to generate software that can be readily adapted to be flexible to future changes:

1. A problem representation, in terms of constraints on any viable solution, that is separate from the solution representation that consists of programmer intentions for how the software should behave. Defining the problem’s uncertainties and solution constraints separately, instead of as parts of a single solution, creates a solution space. Within this defined space, solvers can ignore don’t care variables, in effect relaxing constraints without increasing the dimensionality. The feasibility study discovered that additional degrees of freedom can increase the difficulty for constraint solvers by implicitly adding dimensions, whereas don’t care variables simplify solving. Separating problem and solution representations assists in the identification of these don’t care variables by preventing a single concretization from dictating further constraints (as in the example of

3 For example: P. Hawkins, A. Aiken, K. Fisher, M. Rinard and M. Sagiv, "Concurrent data representation synthesis," in PLDI'12 Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation, Beijing, 2012.

HR001119S0074 INTENT-DEFINED ADAPTIVE SOFTWARE (IDAS) 9

choosing a specific algorithm that requires certain considerations of data formats and concurrency models). In the restaurant example, the dynamic controller enables one to treat table size and number of tables as don’t care variables. As a result, searching for a viable seating solution becomes much simpler.

2. A controller generator. Once programmer intentions are captured and problem constraints identified, sources of uncertainty must be controlled and resolved. Controllers allow for a partial abstractions to be considered complete, because they stabilize behaviors below the controller to offer a guarantee that can be relied upon without needing the details.

Controllers may be able to resolve uncertainties at compile time, or may incorporate logic to sense and respond to runtime changes. In the restaurant example, the controller creates tables of approximately the needed size at runtime, effectively removing the need for the seating algorithm to run an NP-hard bin packing routine to seat parties efficiently at tables of fixed size.

3. An analyzer, which determines if the controllers meet the problem requirements by verifying that their stability guarantees are within the operating bands for controllers at higher levels of abstraction, and that no constraints could be violated by a solution in the defined allowable space.

4. An optimizer/generalizer, which either adds constraints to increase the efficiency of the software system or removes constraints to create a more general solution.

In summary, the IDAS program (envisioned in Figure 1) will enable adaptation of software to radical changes in requirements or its computational environment with an order-of-magnitude reduction in the effort required. The key idea of IDAS is the separation of problem description (in terms of intentions and constraints) from any particular, concrete instantiation. This intent and constraint model must be semantically accessible to an IDAS toolchain, yet expressive enough to capture the relationships between the problem and the method by which generated software can solve and validate a solution. For IDAS to transition, this capture process should be done to the greatest extent possible within the familiar process of writing software, and impose minimal additional tasks on developers who may not understand formal methods. Through additional automation of specific implementation generation, software sustainment effort should be drastically reduced, freeing engineers to focus on the design of the software and adding new functionality.

HR001119S0074 INTENT-DEFINED ADAPTIVE SOFTWARE (IDAS) 10

Figure 1: Intent-Defined Software Adaptation

DARPA anticipates that achieving the goals of IDAS will require research breakthroughs in:

Capturing, learning, or annotating software intent and constraints separate from the concrete decisions required to create a specific instance of software.

Using captured intent to drastically reduce the human-in-the-loop effort needed to adapt software to new requirements, platforms, and resources.

Verifying that the newly-adapted software provides the functional needs of the customer/end user and that the instance does not violate any requirements.

Integrating a new intent-defined software development paradigm into existing Agile workflows to enable adoption and transition into the greater programmer community.

C. Program Structure

IDAS will consist of three phases (see schedule in Section E). Phase 1 will be 18 months in duration and will emphasize research and initial development of the tools and technologies needed to realize a deferred-concretization development paradigm. Phase 2 will be 18 months and focus on iterative exercises to evaluate prototype Technical Area (TA)1 technologies against the current state of the art represented by TA4. Phase 3 will scale up and provide the TA1-developed technologies to TA4 to measure the learning curve and adoption likelihood for traditional developers to develop real-world software in an IDAS paradigm.

Phase 1 will have two testing exercises that will not be used to evaluate performers, but offer an opportunity to practice the evaluation exercise flow. Phase 2 will focus on repeated exercises across the problem domains to measure and differentiate how software is built and adapted to address changing requirements and platforms by TA1 performers and the TA4 control. Phase 3 aims to both scale up the IDAS technologies to real-world problems, measure the learning curve, and assess transition strategies across the commercial and DoD software landscape.

HR001119S0074 INTENT-DEFINED ADAPTIVE SOFTWARE (IDAS) 11

D. Technical Areas

The IDAS program comprises four Technical Areas (TAs):

1. TA1 – Automated software generation

2. TA2 – Problem set generation

3. TA3 – Integrated test and evaluation

4. TA4 – Experimental control and transition

The Government anticipates multiple awards for TA1, and single awards in each of TA2, TA3, and TA4. Proposals shall address only one technical area. Proposers may submit multiple proposals for any or all four TAs, but only TA2 and TA3 may be awarded to the same organization to prevent conflicts of interest (see Section III.D.1 for more details).

TA1 – Automated Software Generation

The goal of TA1 is to create technologies that enable software engineers to develop and verify adaptive software through a deferred-concretization methodology. The core challenge will be to enable traditional developers to work at a higher level of abstraction than currently possible with minimal additional effort. TA1 will produce an augmented development pipeline that can capture abstract descriptions of a problem and its solution (in terms of intentions), and assist in the generation of either source or binary code that meets all stated requirements, with assurance evidence. TA1 technologies should remove or minimize the human-in-the-loop effort when adapting software to address changes in requirements and/or the computational environment. The process of adaptation must produce evidence that the resulting software satisfies the new requirements. DARPA encourages diverse approaches for TA1.

While many possible approaches could achieve the IDAS goals, DARPA expects that, at a high level, there will be a process or approach for the IDAS system to learn or capture the core problem that the software or component is aiming to solve, coupled with automation to adapt the solution to changes in either requirements or computational resources. Today the problem is partially expressed in terms of requirements. Software engineers compensate for the imprecision of this representation with conventions for how components should interact, and programming methods and intuitions acquired through study and experience. A primarily manual engineering process produces a single software instance that embodies a point solution to the problem.

Skilled engineers who are familiar with the system are required to address any substantial changes in the system’s requirements.

The IDAS workflow should in some way instrument, augment, and/or change the development workflow to capture these requirements, conventions, and human intuitions as constraints and intentions, a formalized, semantically-analyzable body of knowledge that represents the problem and constraints on the solution space in a manner that an IDAS development pipeline or source code generator can use to create a viable solution instance. These formalisms could be captured in a number of ways. TA1 proposals should describe how a traditional developer could make use of IDAS tools with minimal additional effort. When there are changes to the requirements or resources available, minor manual edits to these formalized constraints and intentions should enable rapid adaptation and generation of a new version of the software.

Proposed approaches should facilitate the production of verification or assurance evidence that

HR001119S0074 INTENT-DEFINED ADAPTIVE SOFTWARE (IDAS) 12

any generated or synthesized implementation will respect the stated problem constraints and programmer intentions. The abstracted problem description should be far smaller than specific instantiation of the software, potentially enabling substantial scaling up of formal methods approaches to assurance.

A high-level example of how this workflow differs from today’s development practices is shown in Figure 2.

Figure 2 Traditional versus IDAS workflow

Strong TA1 proposals should, at a minimum, address the following:

Methods to capture programmer intent at design, development, or build time while keeping that intent separate from the concrete decisions needed to execute in a specific instance. These methods must be sufficiently intuitive to enable programmers to readily learn and apply them, and should not impose burdens that discourage adoption and use.

A compelling argument that the proposed methods can be learned and adopted by traditional developers without advanced training in topics such as formal logic.

Automation technologies to generate or adapt software to new requirements, resources, and platforms. These technologies should reduce human-in-the-loop effort in the sustainment tail of software.

Evidence generation to provide assurance that the adapted software satisfies the new environment/requirements.

Approaches that are likely to exhibit exponential performance in the size of the solution (e.g., naïve applications of SMT or ILP solvers, see Section B for more details) are considered non-responsive to the TA1 requirements. TA1 proposers should describe an approach that allows developers to express the problem to be solved without coupling it to a partial or complete solution, as is done currently. When a solution space is embedded into the problem

HR001119S0074 INTENT-DEFINED ADAPTIVE SOFTWARE (IDAS) 13

representation, it includes known facts or assumptions that, while correct for a certain use-case, are unlikely to hold for a timescale of DoD-relevance.

Research performed on TA1 may be considered fundamental in nature. TA1 may propose a 12-month, 1.5 full-time equivalent (FTE) optional Phase 4 for transition partner-funded efforts; this option will not be considered fundamental research as it would apply research findings to specific DoD needs.

TA2 – Problem Set Generation

Beginning in Phase 1, the TA2 performer will develop sets of requirements and environments that are comparable in complexity to, but lack the security sensitivities of, actual DoD systems.

These surrogate problem sets should map to DoD use cases, but will be temporally compressed into an evaluation period to test the ability of TA1 and TA4 performers to keep pace with changing requirements and environments. These problem sets will be held back from the TA1 and TA4 performers, to prevent familiarization prior to an evaluation exercise. After each exercise, the TA2 performer will submit the problem sets to the Government for review to be publically released. The publication of these problem sets will allow other researchers in the community to use them as representatives of real-world use cases beyond that of the IDAS program. Due to the potential for interactions between TA2 and DoD operational elements, the TA2 performer team will be required to have a minimum of two individuals with a minimum of a SECRET clearance (TOP SECRET preferred) at time of award in order to meet with stakeholders and understand all aspects of DoD-relevant software systems. In Phase 1, TA2 must prepare for the two planned test exercises. These can be drawn from any domain and do not need a high degree of realism.

Two of the three IDAS problem domains are Logistics and Cloud Agility. Each requires computing an optimal or near-optimal allocation of fixed or limited resources in changing situations based on software (rather than hardware) limitations. This might entail, for example, determining the best method of routing supplies to an area of responsibility via multiple transport means with evolving political and military realities. Over time, the requirements for transport and the means available may change substantially, requiring TA1 and TA4 to produce new software versions. The TA2 performer must understand these problem domains at a granular level, so that they can produce relevant technical requirements and resource changes without reference to sensitive specifics. During the evaluation exercises, the TA2 performer will act as the customer for TA1 and TA4 performers. TA2 must be able to describe the requirements of each problem domain in the preferred formats and/or representations of each TA1 and TA4 performer.

Logistics domain:

Global supply chains require complex optimization and routing, as well as fault-management when mistakes or failure occur. These supply chains are also highly susceptible to disruption or changes in policy. As an example, certain types of cargo (e.g., lithium ion batteries) create constraints on their transportation. Changes in policies are hard to anticipate, so many modern logistics organizations have fragmented, separate systems for managing their supply chains.

Cloud agility domain:

The DoD is migrating its operations to the cloud. However, the cloud is far from a location-independent, vendor-neutral environment. Choosing a specific cloud provider or service API

HR001119S0074 INTENT-DEFINED ADAPTIVE SOFTWARE (IDAS) 14

may lock in the DoD for an extended period of time, reducing agility, and increasing costs.

Problems of this type should demonstrate how migration of complex Software as a Service (SaaS) stacks (such as machine learning frameworks) can be done rapidly from one service API to another. Additionally, changes in international policies are breaking the abstraction that the cloud is location-independent. For example, the recent European privacy laws (GDPR) now dictate how personal information must be handled, requiring software behavior to change based on the physical location of the host server.

Proposals for TA2 must include a third problem domain that is of comparable complexity to the logistics and cloud agility domains. The proposal should, at a minimum, include an explanation of the proposed domain, and how it is relevant to DoD interests. Compelling problem domain proposals should show a clear focus on software in environments undergoing frequent requirements churn and/or changes in computational resources, along with a description of the process for gathering domain ground-truth data.

Strong TA2 proposals should, at a minimum, address the following:

Developing evaluation exercises that subject all portions of the software development lifecycle to repeated changes in requirements, platforms, and computational resources.

Methods for protecting any security sensitivities of the problem domains to ensure that the generated exercises can be publicly released (e.g., within an academic paper).

Identification of ways to reduce overall engineering effort during exercises that does not provide experimental value, while still exploring the scalability of approaches (e.g., a problem that only requires a system of 100s of source lines of code [SLOC] may not reveal exponential behavior in a solver, but a system with 10,000s of SLOC that does not provide insight into key IDAS problems adds cost without utility).

Representation of the end customer’s needs in a variety of means to suit each TA1 and the TA4 performer, from a written requirements document to meetings with developers in a more agile approach.

TA3 – Integrated Test & Evaluation

TA3 will ensure the proper execution of experimentation by deeply understanding the nuances of the TA4 and each TA1 approach, and adjusting the specifics of timing for the release of changed requirements during a 1-4 month evaluation exercise engagement. A sufficient understanding of the technical approaches will enable TA3 to guide TA2 to tailor the exercises to stress each performer system just enough to provide useful feedback. The TA3 performer will coordinate the test and evaluation activities of all other performers in order to ensure that TA2-produced problems sets are neither too simple nor too complex. The TA3 performer will act as referee and judge during each evaluation exercise, and will evaluate the TA1 performers against the benchmark that the TA4 experimental control team establishes. The TA3 performer will establish an Associate Contractor Agreement (ACA) that spans all performers (See Section VIII.D. for additional ACA details).

TA3 proposals should discuss measurement of IDAS metrics, shown in Figure 3. DARPA expects that such measurement for complex, distributed systems will be challenging. After each evaluation exercise, TA3 will report on the performance of each TA1 team, and provide feedback to TA2 and the Government team for improvements of future evaluation exercises.

HR001119S0074 INTENT-DEFINED ADAPTIVE SOFTWARE (IDAS) 15

Additionally, TA3 must produce a report after each exercise on the extent to which the TA1 R&D systems are progressing relative to the TA4 baseline.

Figure 3 IDAS Metrics and Goals

Strong TA3 proposals should, at a minimum, address the following:

Identifying problem set weaknesses and directing TA2 to adapt problem sets to differentiate and maximize exercise value to the IDAS program.

Evaluating TA1 compliance with the TA2 problem sets when requirements may be ambiguous or difficult to measure.

Measuring performance of TA1 approaches, both compared to TA4, as well as ranking strengths and weaknesses between each TA1 approach.

The Government anticipates between one and a maximum of ten TA1 awards. For costing purposes, TA3 proposals should assume a single TA1 performer, and provide up to nine (9) additional costed options spanning the entire IDAS period of performance. Each option’s pricing is for supporting a single, additional TA1 performer. Include any pricing ground rules and assumptions based on the award of any number of options that could increase/decrease the overall value (e.g. economies of scale could reduce the costs by an amount if X number of options are selected for award). The government will determine the number of these TA3 options to negotiate for award based on the actual number of TA1 performers selected for award.

TA4 – Experimental Control and Transition

In order to properly evaluate and measure TA1 performer approaches, the TA4 experimental control and transition team will establish a baseline of performance, against which TA1-developed software and workflows will be compared. The TA4 performer should apply current software engineering best practices to develop software that addresses the same requirements and environmental constraints as TA1 during each evaluation exercise, and will respond to all changes in requirements or computational resources. The TA4 team must therefore have skilled software engineers who have a deep command of the current state of the art in software architecture, operating systems, middleware frameworks, distributed, cloud and web-based computing, build processes, and agile development methods. During Phase 3, this team will adopt the most successful of the TA1 technologies in order to measure the learning curve required for traditional developers to adopt emerging IDAS capabilities.

TA4 proposals should describe the proposer’s track record of applying agile development paradigms to produce large software systems, preferably across a broad spectrum of platforms and architectures. Strong proposals should include detailed past use cases in which agile development practices were used to rapidly address changes in broad requirements or resources (e.g., OSes, data back ends, UIs). Research performed on TA4 may be considered fundamental in nature as it will be the application of state of practice agile development routines to IDAS-

HR001119S0074 INTENT-DEFINED ADAPTIVE SOFTWARE (IDAS) 16

generated evaluation problems for the express purpose of understanding the types of requirements changes that modern agile paradigms struggle to cope with. TA4 proposals may also include a Phase 4 option to support transition, at a scale of 1.5 FTE for 12 months.

E. Schedule/Milestones

For cost estimation purposes, the IDAS program kick off is tentatively scheduled for February 2020, and will run for 48 months. The program will conduct tri-annual Principal Investigator (PI) meetings. Over the course of the program the eight evaluation exercises that increase in complexity, scale (ranging from one to four months in duration, worked from performer locations), and realism will be the primary focus of the IDAS evaluation. Participation is also required at the three demonstrations (single-day technology highlights to Government attendees).

For costing purposes, proposers should assume that all PI meetings will alternate between the Washington, D.C. metro area and San Francisco metro area, and will run for 1.5 days. Assume that demonstrations will require one day in addition to the PI meeting.

The program is structured to contain three phases (Figure 4); proposals should be scoped for all three phases:

a) Phase 1: Research and initial prototypes

b) Phase 2: Robust prototypes

c) Phase 3: Scaling to real-world problems and transition

Figure 4 IDAS Program Schedule

During Phase 1 (18 months), TA1 performers will design and implement an initial proof-of-concept of their toolchain, while the TA4 performer will develop abstraction layers and other frameworks to support evaluation in later phases. The TA2 performer will generate two test exercises and to work closely with mission partners across the three problem domains to generate representative surrogate problems for Phases 2 and 3. The TA2 performer will work with the Government team and TA3 to plan the Phase 2 evaluation exercises, and will work closely with each TA1 performer to understand the nuances of their approach and how best to differentiate the performance of each team. Phase 1 will contain two test exercises (the first of which is optional for TA4) to prepare performers for later phases; TA3 will score these test exercises, but the objective is to familiarize performers with the mechanics and operation of evaluation rather than to measure performance.

In Phase 2 (18 months), performers will conduct a set of development exercises across the multiple problem domains. The exercises will use the TA2 problem domains over multiple weeks

HR001119S0074 INTENT-DEFINED ADAPTIVE SOFTWARE (IDAS) 17

to months to develop software systems and respond to changing requirements, environmental constraints, and other confounding factors. The TA3 performer will release a TA2-generated set of system requirements to all TA1 performers and the TA4 control team, followed by revisions and alterations to those requirements with successively shorter deadlines. 4 TA3 will monitor the progress of all TA1 performers and the TA4 control team to validate implementations and measure the effort required to overcome each change in requirements or computational resource availability. In addition to modifying the Phase 2 problem sets and supporting the evaluation exercises as a client, TA2 will develop larger-scale problems for Phase 3.

The evaluation structure is depicted in Figure 5. TA2 transforms DoD problems from the three diverse domains into representative time-series of changing requirements and environments and delivers them to TA3 for feedback, per the needs of a specific experiment. During evaluations, TA3 will deliver these requirements to all TA1 and TA4 performers concurrently, with TA2 acting as the end customer. As satisfying candidate software is delivered or as timelines dictate, TA3 will publish changes to requirements and/or computational resource availability, testing the ability of TA1 and TA4 performers to rapidly adapt their software systems.

Figure 5 IDAS Evaluation Structure

In Phase 3 (12 months), the IDAS program will focus on preparing the TA1 technologies that have empirically out-performed TA4 in Phase 2 exercises for transition. This will entail providing these TA1 systems/tool-chains to TA4 and measuring the learning curve for conventional developers to understand and apply the IDAS approach and technologies.

Additional exercises will assist TA4 in learning the TA1 tools. During Phase 3, TA1 performers will improve scalability of their technologies and mature the user experience to guide developers to use the developed technologies most effectively. The goal for Phase 3 is for the TA4 performer to become as effective as TA1 performers in the evaluation exercises at handling requirements and environmental changes.

4 The requirements release process must be responsive to each TA1 and TA4 performers’ needs. For example, instead of a formal requirements document, a performer applying agile methods may request an in-person or remote meeting with the “client” (TA2) to understand their needs and problem, from which to derive their requirements.

HR001119S0074 INTENT-DEFINED ADAPTIVE SOFTWARE (IDAS) 18

F. Deliverables

All performers awarded procurement contracts or OTs will be required to provide, at a minimum, the following deliverables:

All technical papers derived from work funded by IDAS;

Commented source code, any other necessary data and documentation (including at minimum user manuals and a detailed software design document) for all IDAS technologies developed under this program;

Annotated slide presentations must be submitted within one month after the program kickoff meeting and after each program event (program reviews, PI meetings, and technical interchange meetings);

Monthly technical status reports detailing progress made, tasks accomplished, major risks, planned activities, trip summaries, changes to key personnel, and any potential issues or problem areas that require the attention of the Government Team must be provided within 15 days of the end of each calendar month;

Monthly financial status reports and copies of invoices must be provided within 15 days of the end of each calendar month;

A final report for each program phase that concisely summarizes the effort conducted, technical achievements, and remaining technical challenges will be due 30 days after the end of each phase; and

A final report at the end of the overall period of performance that summarizes the project.

In addition, the following deliverables are required for particular technical areas:

TA1 & TA4: All code developed during evaluation engagements must be provided to TA3 per a jointly-established individual exercise schedule, to include all source code, binaries, build scripts, test harnesses, development environments, unit tests and system tests. Performers in TA1 and TA4 are also expected to track FTE effort for each of the evaluation exercises and report these metrics to TA3 for evaluation.

TA2: Develop and provide to the government a detailed security plan for providing realism of development/evaluation problems. This plan must describe procedures for abstracting system requirements and constructing test cases representative of DoD needs without reliance on DoD-specific data or information. The TA2 performer must submit this plan to DARPA no less than 90 days prior to commencing analysis of a DoD relevant software system, to allow sufficient time for discussion with the Government team and the completion of any necessary revisions.

TA2 must provide evaluation problems (sets of changing requirements and resources) to TA3 prior to each exercise with an anticipated duration for each development cycle between issuance of the next change in requirements. The mapping from each exercise to a DoD need should be delivered via appropriate channels to DARPA.

TA3: Develop and provide an evaluation plan prior to each exercise, defining metrics for testing and evaluation, and discussing a concept of operations for conducting evaluations of any software that requires user interaction, to be produced in collaboration with TA2. Facilitating the establishment of an ACA for the program is the responsibility of TA3.

HR001119S0074 INTENT-DEFINED ADAPTIVE SOFTWARE (IDAS) 19

Proposers receiving assistance instrument awards may be requested to make similar deliveries or provide software solutions for program test and evaluation before being returned to the performer by the Government.

G. Government-furnished Information

TA2 proposals should suggest sources of data relevant to their particular proposed third problem domain, and describe how they would obtain such datasets. The proposal should discuss the extent to which the Government would be able to obtain such data more easily and/or more quickly. Absent such discussion, the Government does not intend to furnish data.

H. Intellectual Property

The program will emphasize creating and leveraging open source technology and architecture.

Intellectual property rights asserted by proposers are strongly encouraged to be aligned with open source regimes. See Section VI.B.1 for more details on intellectual property.

A key goal of the program is to establish an open, standards-based, multi-source, plug-and-play architecture that allows for interoperability and integration. This includes the ability to easily add, remove, substitute, and modify software and hardware components. This will facilitate rapid innovation by providing a base for future users or developers of program technologies and deliverables. Therefore, it is desired that all software (including source code), software documentation, hardware designs and documentation, and technical data generated by the program be provided as deliverables to the Government, with a minimum of Government Purpose Rights (GPR), as lesser rights may adversely impact the lifecycle costs of affected items, components, or processes.

HR001119S0074 INTENT-DEFINED ADAPTIVE SOFTWARE (IDAS) 20

II. Award Information

A. Awards

Multiple awards are anticipated. The level of funding for individual awards made under this solicitation has not been predetermined and will depend on the quality of the proposals received and the availability of funds. Awards will be made to proposers whose proposals are determined to be the most advantageous and provide the best value to the Government, all factors considered, including the potential contributions of the proposed work, overall funding strategy, and availability of funding. See Section V for further information

The Government reserves the right to:

select for negotiation all, some, one, or none of the proposals received in response to this solicitation;

make awards without discussions with proposers;

conduct discussions with proposers if it is later determined to be necessary;

segregate portions of resulting awards into pre-priced options;

accept proposals in their entirety or to select only portions of proposals for award;

fund proposals in increments and/or with options for continued work at the end of one or more phases;

request additional documentation once the award instrument has been determined (e.g., representations and certifications); and remove proposers from award consideration should the parties fail to reach agreement on award terms within a reasonable time or the proposer fails to provide requested additional information in a timely manner.

Proposals selected for award negotiation may result in a procurement contract, or cooperative agreement depending upon the nature of the work proposed, the required degree of interaction between parties, and other factors.

In all cases, the Government contracting officer shall have sole discretion to select award instrument type, regardless of instrument type proposed, and to negotiate all instrument terms and conditions with selectees. DARPA will apply publication or other restrictions, as necessary, if it determines that the research resulting from the proposed effort will present a high likelihood of disclosing performance characteristics of military systems or manufacturing technologies that are unique and critical to defense.