W911NF-22-S-0001 Advanced Graphic Intelligence Logical Computing Environment (AGILE).pdf

U.S. ARMY RESEARCH OFFICE

In partnership with

The Intelligence Advanced Research Projects Activity (IARPA)

BROAD AGENCY ANNOUNCEMENT FOR

Advanced Graphic Intelligence Logical Computing Environment (AGILE)

W911NF-22-S-0001

Issued by:

U.S. Army Contracting Command-Aberdeen Proving Ground

Research Triangle Park Division

P.O. BOX 12211

Research Triangle Park, NC 27709-2211 i

I. OVERVIEW OF THE FUNDING OPPORTUNITY 1

A. Required Overview Content 1 A.1 Federal Agency Name(s) 1 A.2 Funding Opportunity Title: Advanced Graphic Intelligence Logical computing Environment (AGILE) 1 A.3 Announcement Type 1 A.4 Research Opportunity Number 1 A.5 Catalog of Federal Domestic Assistance (CFDA) Number 1 A.6 Response Dates 1

B. Additional Overview Information 1

II. DETAILED INFORMATION ABOUT THE FUNDING OPPORTUNITY 2

A. Technical Overview 2 A.1 Funding Opportunity Description 2

A.1.1 Introduction and Background 2 A.1.2 Program Scope 3 A.1.3 Program Out-of-Scope 4 A.1.4 Team Expertise 5

A.2 Program Structure 5 A.2.1 Technical Challenges and Research Areas 5 A.2.2 Program Phases 8 A.2.3 Program Metrics and Goals 11 A.2.4 Program Waypoints, Milestones, and Deliverables 15

A.2.5 GFI/GFE 19

A.2.6 Test and Evaluation (T&E) 20

A.3 Program Timeline and Deliverables 24 A.4 Meeting and Travel Requirements 29

A.4.1 Technical Review Meetings 29 A.4.2 Site Visits 29 A.4.3 Visits to Government Testbeds 30 A.4.4 Place of Performance 30 A.4.5 Period of Performance 30

B. Federal Award Information 30

C. Eligibility Information 31 C.1 Eligible Applicants 31 C.2 Cost Sharing or Matching 32

D. Application and Submission Information 32 D.1 Address to View Broad Agency Announcement 32 D.2 Content and Form of Application Submission 32

D.2.1 General Information 32 D.2.2 Preparing an Application 33 D.2.3 Submission of Complete Research Proposals 38

D.3 Unique Entity Identifier and System for Award Management (SAM) 41 D.4 Submission Dates and Times 41 D.5 Intergovernmental Review 42 D.6 Funding Restrictions 42 D.7 Other Submission Requirements 42

E. Application Review Information 42 E.1 Criteria 42 ii

E.1.1 Overall Scientific and Technical Merit 43 E.1.2 Effectiveness of Proposed Work Plan 43 E.1.3 Contribution and Relevance to the IARPA and ARO Mission and Program Goals 44 E.1.4 Relevant Experience and Expertise 44 E.1.5 Resource Realism 44

E.2 Review and Selection Process 44 E.3 Recipient Qualification 45

F. Award Administration Information 48 F.1 Award Notices 48 F.2 Administrative and National Policy Requirements 48

F.2.1 Required Certifications 48 F.2.2 Policy Requirements 50

F.3 Reporting 52

G. Agency Contacts 53

H. Other Information 54 H.1 Outline of requirements for a cost proposal 54 H.2 Sample Elements of a Cost Proposal 59

H.2.1 Direct Labor 59 H.2.2 Other Direct Costs (ODCs) 60 H.2.3 Indirect Rates 63 H.2.4 Cost of Money (COM) 64 H.2.5 Fee/Profit 64 H.2.6 Subcontracting Plan 64

Appendix A: AGILE Workflows and Benchmark Descriptions 65 Appendix A.1 Knowledge Graphs - Groups, Relationships and Interests 65 Appendix A.2 Detection – System and Event Patterns 66 Appendix A.3 Sequence Data – Identification and Clustering 67 Appendix A.4 Network - Cyber-Physical Systems 68

Appendix B: FAR 52.209-11 70

Appendix C: Definitions 71

Appendix D: Table of Abbreviations 73

I. OVERVIEW OF THE FUNDING OPPORTUNITY

A. Required Overview Content

A.1 Federal Agency Name(s)

U.S. Army Research Office

Issuing Acquisition Office:

U.S. Army Contracting Command-Aberdeen Proving Ground, Research Triangle Park Division (ACC-APG RTP Division)

A.2 Funding Opportunity Title: Advanced Graphic Intelligence Logical computing Environment (AGILE)

A.3 Announcement Type

Initial Announcement

A.4 Research Opportunity Number

W911NF-22-S-0001

A.5 Catalog of Federal Domestic Assistance (CFDA) Number

12.431 – Basic Scientific Research

A.6 Response Dates

Proposals: 4:00 PM Eastern Standard Time on: January 18th, 2022. See Section II.D.4 for additional information.

B. Additional Overview Information

This Broad Agency Announcement (BAA) which sets forth research areas of interest to the Army Research Laboratory- Army Research Office (ARL-ARO) and the Intelligence Advanced Research Projects Activity (IARPA) is issued under paragraph 6.102(d)(2) of the Federal Acquisition Regulation (FAR), and 10 USC 2358 which provides for the competitive selection of basic research proposals. Proposals submitted in response to this BAA and selected for award are considered to be the result of full and open competition and in full compliance with the provision of Public Law 98-369, "The Competition in Contracting Act of 1984" and subsequent amendments.

The Department of Defense (DoD) agencies involved in this program reserve the right to select for award; all, some, or none of the proposals submitted in response to this announcement. The participating DoD agencies will provide no funding for direct reimbursement of proposal development costs. Technical and cost proposals (or any other material) submitted in response to this BAA will not be returned. It is the policy of participating DoD agencies to treat all proposals as sensitive, competitive information and to disclose their contents only for the purposes of evaluation.

II. DETAILED INFORMATION ABOUT THE FUNDING OPPORTUNITY

A. Technical Overview

A.1 Funding Opportunity Description

The U.S. Army Research Office (ARO) in partnership with IARPA seeks research and development of innovative, energy-efficient, reliable computer architectures for the DoD and the Intelligence Community (IC) to implement large-scale data-analytic applications, as well as other classes of large problems. The primary focus of this program, Advanced Graphic Intelligence Logical computing Environment (AGILE), is the development of new system-level intelligent mechanisms for processing, moving, accessing, and storing large, unfiltered, time-varying data streams and structures that allow for the scalable and efficient execution of dynamic graph analytic workflows.

The overarching goal for the AGILE program is to create novel computer architectures and designs that overcome the challenges specified below. The AGILE program will result in the delivery of Detailed Designs, whose performance has undergone rigorous and independent Testing and Evaluation. Pending results of this program, IARPA and ARO may support a follow-on program to develop prototypes of the proposed architectures.

A.1.1 Introduction and Background

Today’s new era of explosive data growth poses serious challenges for the DoD and the IC in monitoring adversarial activities and predicting future adversarial actions. This burgeoning data, which is increasing exponentially not only in volume, but in velocity, variety, and complexity, already far outpaces the abilities of current computing systems to execute the complex analytics needed to extract meaningful insights in a timely manner. The key computational problem is that today’s computers were designed to address yesterday’s compute-intensive problems rather than today’s data-intensive problems. Transforming the massive, unstructured, heterogeneous data streams and structures into actionable knowledge could benefit from a reimagining of computing architectures and technologies – one that places primary focus on data movement, storage, and access of irregular and time-varying structures.

The data of interest is increasingly sparse, unstructured, and heterogeneous, with minimal locality (it is distributed across the computer), poor data re-use, streaming updates flowing into the system, and fine-grain data movement and parallelism. The computations to be performed are determined by the data with multiple applications simultaneously accessing the same data. These are very different conditions than those characteristic of yesterday’s compute-intensive applications.

The AGILE program seeks innovative, energy-efficient, reliable computer architectures which can address the DoD’s and the IC’s large-scale data-analytic applications, as well as other classes of large problems. This solicitation focuses on developing new system-level intelligent mechanisms for processing, moving, accessing, and storing large, unstructured, time-varying data streams and structures that allow for the scalable and efficient execution of dynamic analytics workflows. Acceptable AGILE system designs must emphasize optimizing the fully integrated system, rather than the independent optimization of individual functionalities (e.g., memory or computation). Supporting research and development for these proposed designs do not need to be constrained by existing component interfaces and protocols, legacy architectures, or current practices. It is anticipated that a “clean sheet” approach to designing a computer system for data-intensive applications will be required to meet the AGILE goals.

The protection of the data from observation and modification by unauthorized agents and concurrently running applications must be a fundamental capability of any AGILE architecture and must be considered during all design phases, not layered on afterward. Structures and mechanisms must be included (e.g., hardware root of trust and encryption) that can impede an adversary’s ability to obtain or modify data or software or alter the hardware/software to create backdoors or malicious circuitry. AGILE computers must be able to withstand remote hacking attempts and insider threats.

A.1.2 Program Scope

The AGILE program is envisioned as a 36-month effort with two phases. Phase 1 will last 18 months, and Phase 2 will last 18 months.

The overall goals for the AGILE program are to create novel computer architectures and designs that overcome the challenges specified below. The AGILE program will result in the delivery of Detailed Designs, whose performance has undergone rigorous and independent Testing and Evaluation (T&E) using the application modeling and simulation environment described in Section A.2.6 Test and Evaluation (T&E). The proposed designs must have the following characteristics:

1. Efficient and scalable when executing large-scale data analytics including streaming analytics;

2. Energy efficient, reliable, and able to support scaling from a deskside system to large multi-cabinet configurations. Energy efficiency should be at least equal to today’s computer systems but higher is preferred;

3. Cost effective. The price of computation should be at least equal to today’s computer systems but less expensive is preferred;

4. Secure from an adversary attacks;

5. Realizable in silicon prior to approximately 2030;

6. The IP utilized in the design must be open sourced or licensable by the US Government; and

7. Able to meet the metrics described in Section A.2.3 Program Metrics and Goals.

We will release four workflows plus their derivative kernels, and three industry standard benchmarks to drive the research and development of the AGILE system designs. Offerors should propose a research project that develops a computer design driven by the four AGILE Workflows, Kernels and Benchmarks. These workflows are described in Section A.2.5 GFI/GFE and Appendix A. Benchmarks are described in Section A.2.3 Program Metrics and Goals.

Offerors must propose a full system design and should describe a system-level performance model, including the performance factors. The proposal should describe how the data analytics problem provided in this BAA impacts the performance factors. The proposal should also tell how the proposed performance model will drive the design decisions.

Achieving AGILE program goals will require utilization of the four end-to-end AGILE Workflows, Kernels and Benchmarks in the co-design process. In this document, AGILE Workflows, Kernels, and Benchmarks will be referred to as AGILE Applications. Performers will be provided a collection of Government Furnished Information (GFI) workflows and kernels based on AGILE workflows. However, Performers will need to develop their own micro-kernels to initiate the co-design of their components and to substantiate their capabilities. The AGILE program will provide Performers with GFI source code for workflows written in C++, plus workflow-based kernels and links to industry-standard benchmark codes or benchmark descriptions to be used in both the co-design development process and to evaluate the performance of the resulting AGILE designs. For each AGILE workflow and kernel, a test suite at different scales will be provided.

Performers’ designs will be tested and evaluated using the AGILE-enhanced Structural Simulation Toolkit (A-SST), which is based on the open-source software toolkit, Structural Simulation Toolkit (SST), developed by Sandia National Laboratories. A-SST provides the infrastructure for modeling and simulating components, node designs, and system designs. Although Performers will be responsible for reporting the performance of their designs on the AGILE metrics, their results will need to be independently verified using A-SST. Performers’ design characteristics and performance will be further evaluated using an FPGA hardware emulation platform and any other tools deemed appropriate by the T&E Team. The FPGA emulation platform will primarily be used for the validation of and detailed circuit-level timing analysis for the components. All Performers will be provided the A-SST software and given access to an FPGA emulation platform by the T&E Team for the Performers to use at their own discretion. Alternative to using of the FPGA emulation platform, Performers may instead use the equivalent Berkeley FireSim emulator running on an Amazon F1 FPGA instance but will need to pay for its use out of their project funds. The Performers must supply their design specification using design-tool-neutral modeling or hardware description artifacts that can be imported to A-SST (SystemC, C/C++, SST, Verilog, or System Verilog). More details on the design and model descriptions are provided in Section A.2.6 Test and Evaluation (T&E).

Funded AGILE projects will be openly discussed in public forums, such as workshops and conferences. Topics discussed will include the research approaches, the architecture and designs, and the performance estimates' design or components. Background IP may remain proprietary, but Performers must be willing to discuss openly how the IP is used in their design and must make a version of the background IP available for use within A-SST, with terms and conditions that T&E partners can accept, and with correct timing results.

Definitions for most of the AGILE program related technical terms used in this BAA are in Appendix C: Definitions

A.1.3 Program Out-of-Scope

The following areas of research or technical approaches are out of scope for the AGILE program, and proposals that include such initiatives are not awardable:

Approaches that are likely to result in only incremental improvements over the current state-of-art.

Approaches that do not leverage the AGILE Applications or cannot be tested and evaluated using A-SST.

Approaches with significant limitations on operating conditions or operational parameters, such as cryogenic computers.

Design of power distribution and cooling systems for the proposed architecture.

Designs that cannot be realized in silicon prior to approximately 2030.

Designs that consist of Intellectual Property (IP) that is not open-sourced or licensable by the

US Government.

Designs that do not provide efficient and scalable execution of large-scale data analytics, including streaming analytics.

Development of integrated circuits that are not required for the Offeror’s design.

Energy inefficient design.

Partial solutions that cannot be independently tested and evaluated against program metrics.

Research that does not have strong theoretical and experimental foundations, or plausible scientific support, for the Offeror’s claims.

Resulting system that will not be cost effective.

Solutions that cannot be made sufficiently robust for real-world use.

System designs that do not scale from a deskside system to large multi-cabinet configurations.

The AGILE program does not involve the development of a complete programing environment.

However, the runtime system and memory models will clearly have an impact on programming.

Offerors should provide a vision for the anticipated programming environment and describe the methodology for converting a program written in a high-level language into an optimal executable.

Proposals should address issues concerning how programmers will develop applications for the new architecture, and how the programming model contributes to improving performance, scalability, and productivity.

A.1.4 Team Expertise

The Performer teams should be multidisciplinary and may include expertise and experience in multiple fields related to the AGILE program goals. Therefore, collaborative efforts and teaming is highly encouraged. Teams are encouraged to include researchers with experience in innovative, leading-edge computing technologies and may consist of academic and industrial researchers and developers. The Principal Investigator (PI) should have experience managing a substantial, complicated research and development project. This individual could come from an academic institution, a small company, or a large company.

A.2 Program Structure

A.2.1 Technical Challenges and Research Areas

AGILE systems must be capable of processing large-scale data-analytic applications at least 10x faster than today’s departmental-level multi-cabinet computer systems. The proposed AGILE architecture should have a hierarchical and modular design. The design must scale from a single node to a multi-node system, and should not be constrained by industry commodity components, industry standard protocols, or execution policies. However, the design may include interfaces to commodity components such as accelerators or memory devices.

Proposed computer system designs will consist of four fundamental functions:

1. Communication – the web that permeates the computer structure to provide mechanisms for moving data and executing message-driven remote actions (between and within the nodes in the AGILE System).

2. Memory – mechanisms for accessing and storing the data.

3. Computation – mechanisms for the execution and flow control of tasks.

4. Runtime – infrastructure for executing system and computational tasks.

For AGILE, these four functions must be developed in coordination with one another to ensure that innovations in one function are not negated by bottlenecks in another (i.e., the fully integrated system must demonstrate the cooperative advantages of its subsystems). The key technical challenges are as follows:

1. Processing elements designed to execute efficiently a broad spectrum of data-driven methods.

Proposed AGILE systems should not rely on one or two assumptions regarding the regularity and predictability of memory accesses or instruction mix because no one or two assumptions apply universally to all the workloads. AGILE data-analytic applications generate workloads where the execution patterns have poor spatial and temporal locality, and instruction streams have numerous branch-conditions that cannot be predicted ahead-of-time. Furthermore, the amount of work executed by any given thread, and/or the amount of data produced from any kernel invocation, may be data dependent and not predictable a priori.

2. Fine-grain, irregular data movement ranging from the processing elements to the system level. Applications with irregular, unpredictable, and data driven memory requirements will need systems to support alternative approaches to minimize/hide latencies from processing element to system level. Current approaches – multithreading and partitioning – have advantages and disadvantages. For example, multithreading can generate a sufficient number of fine-grain memory requests to saturate communication channels; however, the efficiency of data movement can suffer from the overhead of setting up the data transport and the inefficiency of transmitting a package or packet. These efficiency problems, plus the diversity of workload requirements, will require some manner of reconfigurability and adaptive control of dataflow, bandwidth allocation, and resource allocation/disaggregation. The sheer volume of data managed by this system is expected to be more than a petabyte (PB) and could require innovations in memory address translation, hybrid volatile/non-volatile storage systems, and potentially novel data compression and recoding technologies.

3. Parallelism at all levels (fine-to-coarse grain). This includes self-discovered parallelism since, for example, graph processing for irregular, time-varying structures has parallelism that can only be discovered in real time. The high percentage of fine-grain tasks may necessitate asynchronous operations.

4. New memory and storage architectures for unstructured, dynamic irregular data structures.

AGILE Applications may have non-regular structures whose number, shape, and size are not known until runtime and may change dynamically during runtime. This creates challenges for conventional page-based memory management and data layout in memory structures.

Moreover, the creation and deletion of these data structures may need to happen asynchronously at large scale, which creates huge challenges for memory consistency models at both, the node and system scale. The data properties (the base type) may be highly varied and unknown until runtime.

5. An I/O subsystem capable of ingesting high-velocity streams of data from multiple external sources. POSIX filesystem1 semantics make it very challenging to ingest data from multiple data sources into a file interface that has sequential semantics while still maintaining global consistency across all clients. The software interfaces and layered abstractions to access non-volatile and high-capacity storage are increasingly mismatched with the underlying technology.

1POSIX.1-2008, IEEE 1003.1-2008, The Open Group Base Specifications Issue 7. URL:

http://pubs.opengroup.org/onlinepubs/9699919799/

6. System security, data integrity, and compliance services to support multiple applications working cooperatively in the same memory space. For AGILE systems, the access control apparatus must allow a common approach to managing authentication and authorization, which works across nonvolatile storage and volatile memory. This contrasts with modern high-performance computing systems, in which multi-tenancy usually involves many different users and applications with different access credentials that are able to share a common database. Both efficient data transport and end-to-end data integrity protection are essential.

The runtime system presents an abstraction of the computing system software structure and operation for a specific system architecture. The design provides a fundamental conceptual framework for the co-design of technology: architecture, programming interfaces, and system software. The designs of the runtime system should therefore be used as the conceptual framework for overcoming the above challenges.

When developing an AGILE design, Performers must utilize the co-design methodology (see Figure A.2.1-1) at all levels of the AGILE architecture to ensure that: (a) components are designed appropriately, (b) all subsystems cooperate to optimize performance and eliminate data-movement bottlenecks at the system level, and (c) both system designs and AGILE Applications are simultaneously optimized. The co-design process will enable the system design and the application codes to be iteratively optimized. The T&E Team will utilize the A-SST environment to perform metric measurements of the design.

Figure A.2.1-1 AGILE Co-Design Process.

Possible areas of research for AGILE include, but are not limited to:

Tightly integrated design (communication, memory, compute & runtime) resulting in a unified, efficient, and scalable system with transparent access to data and compute resources throughout the system.

Data-driven compute elements optimized for the AGILE Applications, supporting efficient data movement that provides low-latency, high-bandwidth data accesses, and moving the compute to the data.

Distributed memory management and security system that efficiently supports fine-grained addressing and protection of objects and data across the system.

System-level intelligent mechanisms for processing, moving, accessing, and storing large, unstructured, time-varying data streams and structures.

Global name space/global adaptive data transfer mechanism to enable efficient data movement and adaptation of information flows to match complex AGILE Applications requirements.

Dynamic adaptive runtime systems to match activity demands to changing resource availability.

Offerors should address these technical challenges and propose additional technical challenges they plan to investigate. Offerors must provide an explanation of how their proposed architecture will address the technical challenges.

A.2.2 Program Phases

The AGILE program will proceed in two phases, which are designed to enable Performers to create innovative, scalable, and efficient computer architectures driven by data-intensive problems (as described in Section A.2.5, GFI/GFE), and to develop those architectural concepts into progressively capable designs described by increasingly accurate models that will be quantitatively evaluated for performance on AGILE program metrics (as described in Section A.2.3). Table A.2.2-1, AGILE Program Phases, summarizes the two phases and lists their characteristics.

Phase 1: Architectural Design (18 months)

The objective of this eighteen-month Phase 1 is to provide a system-level architectural design in the form of a model that will meet the performance metrics shown in Section A.2.3 of this BAA.

This Architectural Design must be characterized by a set of high-level behavioral models (see Appendix C: Definitions) suitable for estimating all aspects of its performance on the AGILE Applications and program metrics in the A-SST environment. The Phase 1 high-level Architectural Design may need to be a multi-resolution set of models, ranging from fine granularity at the circuit level to coarse granularity at the multi-node system level. More details on the development of this suite of models are provided in Section A.2.6. During the course of Phase I, Performers will be provided GFI AGILE Applications. Performers will be expected to utilize the GFI AGILE Applications to identify system bottlenecks and create innovative solutions that improve the overall performance and efficient resource utilization of the new system. It is anticipated that some Detailed Designs will include new integrated circuits designs; the accuracy of the timing models for these circuits will need to be validated by the Performers and verified by the T&E Team.

Performers will generate comprehensive test plans and test suites required to evaluate their system architecture designs. This includes completeness, correctness, functionality, system characteristics, and performance estimates on the AGILE Applications. Any modifications to the GFI application suite (AGILE Applications) codes to optimize them for the new architecture being developed is the responsibility of the Performers. While Performers are free to use any industry standard architectural development environment that they select, the performance evaluations and their verification by the T&E Team will be conducted using A-SST. The description of the high-level behavioral model must be provided in design-tool-neutral IEEE-standard format (either

SystemVerilog, SystemC, or C/C++) that can be imported into the A-SST environment for independent testing and verification by the T&E Team. It is the performer’s responsibility to ensure interoperability with the A-SST framework.

Table A.2.2-1 AGILE Program Phases

Area Phase 1 (18 months) Phase 2 (18 months)

Design review Preliminary design review Critical design review

Application Test Plans, with test suites

Validate the completeness & correctness of the AGILE Applications modified for the Architectural Design

Validate the completeness & correctness of the AGILE Applications modified for the Detailed Design

Hardware Test Plans, with test suites

Validate the completeness, correctness, functionality, system characteristics, and performance of the Architectural Design

Validate the completeness, correctness, functionality, system characteristics, and performance of the Detailed Design

Workflow and kernel codes

Optimize for Architectural Design Optimize for Detailed Design

Application software framework

Prototype development tools and libraries for the Architectural Design

Prototype development tools and libraries for Detailed Design

Models/designs Architectural designs, including high-level behavioral designs and abstracted node & system-level models

Detailed Design, including RTL designs/functional models, with abstracted node & system-level models

The use of A-SST for independent testing of the designs is described in Section A.2.6. A-SST and an FPGA emulation platform will be available to Performers at the start of the AGILE program. The AGILE T&E Team will assess the system architecture using the A-SST environment and the test suite generated by the Performers as well as sequestered test-sets that are developed by the T&E Team. This will allow IARPA and ARO to perform independent verification and validation to measure the completeness, correctness, functionality, system characteristics, and performance of the proposed Architectural Design.

At the conclusion of Phase 1, the Performers must make a compelling argument, supported by the T&E Team’s independent evaluation of their designs and simulated performance results, that their Phase 1 Architectural Design and future design approach will meet the AGILE metrics at the end of the program. The Architectural Design must be capable of being imported into the A-SST environment to enable performance estimates for a multi-node simulation of the AGILE Applications.

The diagram in Figure A.2.2-1 presents a possible structure of the Architectural Design.

Figure A.2.2-1 Representation of a potential structure of the Architectural Design

Phase 2: Detailed Design (18 months)

The objective of this eighteen-month Phase 2 is to provide a Register Transfer Level (RTL) design of the AGILE system developed in Phase 1 (plus continuing innovations). If Performers’ designs include Commercial-Off-The-Shelf (COTS) components, this design can be a hybrid design composed of Functional or High-level Behavioral models (see Appendix C: Definitions) for any COTS components in the system and RTL designs for each AGILE-developed integrated circuit.

The Phase 2 design may need to be a multi-resolution set of designs, ranging from RTL at the circuit level to coarse granularity at the multi-node system level. The Detailed Design will include components, node designs, and system-level interconnect models for a full system design. This might require a mixed-resolution system model, where both RTL models and higher-level models run in the same simulation, connected together with the system-level interconnects for system-level simulations.

Performers will focus on the detailed implementation of the AGILE system architecture developed in Phase 1, including the logic design of any Application Specific Integrated Circuits (ASICs) and/or Field Programmable Gate Arrays (FPGAs) contained in the system design. The Phase 1 Architectural Design shall be kept up to date with any architectural changes made to the Detailed Design.

As in Phase 1, Performers shall generate a comprehensive test plan and test suite required to evaluate their design. This includes completeness, correctness, functionality, system characteristics, and performance estimates on the AGILE Applications. Any modifications of the GFI application suite (AGILE Applications) codes to optimize them for the developing AGILE architecture shall be the responsibility of the Performers. As in Phase 1, although Performers are free to use any industry standard development environment, the hardware description must be provided to the AGILE T&E Team using a design-tool-neutral, IEEE-Standard hardware description language – Verilog or synthesizable SystemVerilog for Detailed Design components and SystemVerilog, SystemC, or C/C++ for High-Level Behavioral models. As in Phase 1, it is the performer’s responsibility to ensure interoperability with the A-SST framework. The AGILE T&E Team will assess the system design using the A--SST environment and the test suite generated by Performers. This will allow IARPA and ARO to perform independent verification and validation to measure the completeness, correctness, functionality, system characteristics, and performance of the Detailed Design.

Deliverables for Phase 2 include a node level design capable of being used to accurately simulate and model a multi-node system. Performers must provide tests results that verify that all IP used in the design has been validated and deliver a test plan and test suite for the system design and its components. The Performers will deliver the hardware description in either IEEE-Standard SystemVerilog or Verilog so that it can be imported into A-SST for evaluation by the T&E Team.

Any components that are not expressed in an HDL2 (Verilog or SystemVerilog) must have their high-level C/C++ behavioral model delivered as either SST or SystemC components. They shall also deliver their modified versions of the AGILE Applications codes for evaluation and use by the T&E Team. Finally, Performers must deliver design documents and modeling, and/or other calculations to demonstrate that the design, when manufactured, can be expected to meet all AGILE program goals.

The diagram in Figure A.2.2-2 presents a possible structure of the Detailed Design.

Figure A.2.2-2 Representation of a potential structure of the Architectural Design

A.2.3 Program Metrics and Goals

Achievement of metric goals is a critical factor that will be taken into account when determining whether to continue work under IARPA/ARO research contracts. IARPA/ARO have defined program metrics to evaluate effectiveness of the proposed solutions in achieving the stated program goal and objectives, and to determine whether satisfactory progress is being made.

The AGILE target metrics described in this BAA are shared with the intent to bound the scope of the effort, while affording Offerors maximum flexibility, creativity, and innovation in proposing solutions to the stated problem. Performance metrics are based on today’s departmental-level multi-cabinet computer systems and the goals were selected based on the following factors:

1. What is deemed technically feasible based on emerging microelectronics and computer system research but challenging based on current computer system capabilities.

2. What is useful to mission partners?

Proposals with a plan to exceed the defined metrics in one or more categories are desirable, provided that all of the other metrics are met. All proposals must provide clear justification as to why their proposed approach will be able to meet or exceed the enhanced metric goals. Offerors should provide an analysis of the fundamental algorithms for each AGILE Workflow metric and describe how their proposed architecture will provide the required performance improvement compared to the performance expectations on a computer system that is available at the time of this proposal. During the first phase of the AGILE Program, Architectural Design, Performers

2 HDL = Hardware Description Language will present a Preliminary Design document at Month 9 of their proposed architecture. The Preliminary Design document will provide the anticipated metrics values that will be achieved by the proposed architecture for each standard benchmark and AGILE Workflow metric. The AGILE targeted values for each of the Workflow components described in Section A.2.5, are shown below in Tables A.2.3-1 through A.2.3-4.

Table A.2.3-1 Workflow: Knowledge – Groups, Relationships & Interests

Metric Today AGILE Target

Data ingestion rate

0.1 G3

data-elements per second from a

Single source, single data type

10 G data-elements per second from

3 or more sources and data types(100x faster for 3 sources

Time to learn embedding (Graph Size > 1 PB) 1,440 minutes 30 minutes

(48x faster)

Time to classify vertices and edges > 1,440 minutes 30 minutes

(> 48x faster)

Time to predict and infer new relationship > 1,440 minutes 30 minutes

(> 48x faster)

Time to reason about higher-order relationships using multi-hop reasoning

1 – 2 hops (exact matches) in 30 minutes

3 – 5 hops (approximate/fuzzy matches) in 1 minute

(30x faster)

Table A.2.3-2 Workflow: Detection – System and Event Patterns

Metric Today AGILE Target

Size of graph 0.01 PB4

10 PB

(1000x larger)

Data ingestion rate 0.1 G data-elements per second from a

10 G data-elements per second from a

3 G = 109 4 PB = 1015 Bytes single source, single data type three or more sources and data types

(100x faster for 3 sources)

Insert/Delete/Modify rate for vertices and edges

0.01 G edits /

second (batched)

10 G edits / second (continuous)

(1000x faster)

Pattern Detection per minute Single event, linear paths, exact match

Multiple events, branches, prioritized approximate/fuzzy matching

Incremental analysis NOT DONE Commensurate with data rate

Time to complete multiple day / multiple location queries

NOT DONE

Completed in minutes

Table A.2.3-3 Workflow: Sequence Data – Identification and Clustering

Metric Today AGILE Target

Data ingestion rate 1 M records per second from a single source

10 M records per second from 3 or more sources

(10x faster for 3 sources

Records processing rate 0.1 M per second 10 M per second

(100x faster)

Time to construct similarity graphs using metrics such as Jaccard index

400 hours5 6 hours6

(67x faster)

Time to cluster similarity networks 500 hours[2] 10 hours[3]

5 Assumes a graph of 10 billion vertices and 100 billion edges running on departmental size cluster

6 Assumes a graph of 1 trillion vertices and 100 trillion edges

(50x faster)

Time to predict labels (functions) of new sequences

200 hours[2] 4 hours[3]

(50x faster)

Time to estimate uncertainties of labels and functions, and prioritize alerts

NOT DONE Completed

Table A.2.3-4 Workflow: Network – Cyber-Physical Systems

Metric Today AGILE Target

Construct 1 PB graph through game theoretic modeling

120 minutes 2 minutes

(60x faster)

Identification of top k influential nodes (simple model)

60 minutes 1 minute

(60x faster)

Identification of top k influential nodes (enhanced model)

600 minutes 30 minutes

(20x faster)

Propagate labels/confidence score 120 minutes 2 minutes

(60x faster)

Incremental analysis NOT DONE Never recomputed from scratch

Benchmark Codes

Breadth First Search (BFS) - https://graph500.org/ Triangle Counting - http://graphchallenge.mit.edu/ Jaccard Coefficients - https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7529958

Benchmark Assumptions

System has 1+ PB of storage for graph data

A core represents a basic computing element, within the system, capable of executing graph algorithms and loading data from a memory

Overall performance target metric is: 10x faster than a conventional system, that has 1 PB storage

Graph Data Properties

Utilizes the Graph500 Benchmark datasets (http://graph500.org/)

Scale 36: 236 vertices & storage 17.6 TB7

Scale 42: 242 vertices & storage 1.1 PB

Beyond the performance metrics specified in, Table A.2.3-5, Benchmarks, the Offeror’s proposal should include estimated wall clock time, memory and network metrics, and resource utilization also need to be reported.

Table A.2.3-5 Benchmarks

Benchmark Metric

Target Metrics

Scale 36 Scale 42 Report

Breadth First Search

(BFS)

1 GTEPS8/core 1 GTEPS/core Traversed edges per second (TEPS)

Triangle Counting 1 M9 triangles/sec/core 1 M triangles/sec/core Number of triangles & number of 2-paths

Jaccard Coefficients 1 M coefficients/sec/core

1 M coefficients/sec/core

Number of Jaccard Coefficients & number of Jaccard Coefficients per second

A.2.4 Program Waypoints, Milestones, and Deliverables

This section describes the program schedule, including Waypoints, Deliverables and Milestones.

Waypoints, Milestones, and Deliverables are established from the program’s onset to ensure alignment with AGILE objectives, organize research activities in a logical and reportable manner, and facilitate consistent and efficient communication among all stakeholders – IARPA, ARO, AGILE T&E Team, U.S. Government Stakeholders, and AGILE Research Performers. A schedule of key program Milestones and Deliverables is shown in Table A.3-1 and Table A.3-2. Milestones are significant events or actions that are required by the program.

Waypoints

Waypoints are events that are proposed by the Offerors and are a tangible measurement of Performer progress toward achieving the AGILE program objectives. Waypoints provide additional insight into the development of the key aspects of the proposed research beyond the measurement of deliverable performance metrics. They assist the program management team in providing guidance and assistance to Performer teams. Program waypoints may be refined during the various Phases of the program.

7 TB = 1012 Bytes 8 GTEPS = 109 traversed edges per second 9 M = 106

Offeror’s proposed technical and programmatic waypoints shall be included in the Offeror’s proposal. For each proposed waypoint, the proposal shall describe the waypoint, its relationship to program tasks(s) and criteria for successful achievement of the waypoint, and the date by which the waypoint shall have been achieved. It is preferred that this waypoint information be conveyed in tabular format as indicated below in Table A.2.4-1.

Table A.2.4-1 Sample Waypoint Table

Phase

Months after Start of Phase Waypoint Name Description

Success Criteria

Mandatory Program Waypoints

The Government has identified the following two waypoints for all Performers:

Waypoint 1: Preliminary Design Review (PDR). A PDR will occur during the second site visit, expected within the first nine (9) months of contract award. The Performer must deliver documentation of the preliminary design, updated target metric values for the AGILE Applications, and implementation plan, an accurate program schedule, and a mitigation plan for identified risks. The Performers shall describe the bottlenecks they identified during their analysis and their rationale for their design trade-offs. At the PDR, Performer progress, plans, proposed optimized AGILE Applications and architectures will be presented by the Performer team and assessed by the Government with input from the T&E Team and selected advisors. The preliminary design must include the design development process including the methodology for modeling the designs.

The output of the preliminary design would be deep performance analysis of the workloads, block diagrams of the proposed system, performance assumptions for the key subsystems, justifications for those performance assumptions, an analysis of the components, node and full-system modeling methodology and determination that the designs will be evaluated using A-SST. The PM will document any deficiencies in what was presented by the Performer at the PDR or within ten (10) business days thereafter. The exit criteria for the Critical Design Review (CDR) will be discussed at the PDR. The PM may also refine the Exit Criteria for the CDR as part of the PDR documentation following the PDR. The preliminary architectural specification and Preliminary Design are deliverables

The PDR will provide the Performer’s preliminary architecture specification. The CDR will also provide the methodology for modeling the architecture specification at all levels. That is, component, node and system levels. Finally, the CDR will present how the model of the architecture specification will be evaluated using the A-SST environment.

The PDR will provide the initial design of the runtime system. The proposed methodology for utilizing the runtime system in the co-design process and the evaluation of the proposed models will be presented.

Waypoint 2: Critical Design Review (CDR). A CDR will occur during a site visit to the Performer site within twenty-four (24) months of the program kick-off. At the CDR, Performer progress, final plans, Applications, and designs will be presented by the Performer and assessed by the Government PM, with input from the T&E Team and selected advisors. The Performer will provide evidence that all CDR Exit Criteria have been met. The hardware specification is also a deliverable. Performers must also report the results of their internal testing at the CDR and describe how any deficiencies in performance will be addressed. The PM will document any deficiencies in what was presented by the Performer at the CDR or within ten (10) business days thereafter.

The CDR will provide the hardware design specification. The specification of the runtime system will be provided. Also, the approach to utilizing A-SST will be presented. The architecture specification will be updated.

The architecture, hardware and runtime specifications will be updated throughout the program.

Deliverables

Below are details concerning the AGILE deliverables, which are in addition to the PDR and CDR Deliverables. A timeline of the program deliverables follows below (see also Section A.3).

Kickoff Meeting

Performers shall provide the presentation slides and any supporting documentation from the Kickoff Meeting.

Program Wide Review Meeting (PI Review)

Performers shall provide the presentation slides and any supporting documentation for each PI Review. Supporting documentation can include written responses to questions from the Government attendees. At the conclusion of Phase 1, Performers must also make a compelling argument, supported by their performance results on the program metrics as described at the end of Section A.2.3, that their Phase 1 Architectural Design and future Detailed Design approach will meet Phase 2 metrics by the end of the program.

Site Visits

Performers shall provide the presentation slides and any supporting documentation for each Technical Review Meeting. Supporting documentation can include written responses to questions from the Government attendees.

Technical Reporting

Performers shall provide monthly technical status reports no later than 10 days after the first of each month. The technical reports shall document the status of the project and progress towards achieving AGILE metrics, including results of internal performance tests. It will also track Performers’ self-determined technical project risks and corresponding mitigation strategies. Both the results presented at technical review meetings and technical reports will serve as an official record of progress.

Technical Review Meetings

Performers shall provide the presentation slides drawn from monthly status reports and any supporting documentation for each Technical Review Meeting. Supporting documentation can include written responses to questions from the Government attendees.

Financial Reporting

Performers shall provide monthly financial status reports (MSRs) not later than ten days after the first of each month. The MSRs shall summarize budget, spending plan and actual spending and an explanation of any variance, as well as identify any financial issues that may affect the program or put achievement of program objectives at risk.

Architecture Specification

Performers shall provide an architecture specification which, as a minimum, contains a block diagram, description of the major functions of each block, major timing requirements for each block and between blocks, anticipated metric values for the system, discussion of major risks and mitigation plan, and targeted process technology for each new chip that will be designed as part of this program. The Architecture Specification will be updated as necessary during Phase 2.

Hardware Design Specification

Performers shall provide a hardware specification which, as a minimum, contains a block diagram, detailed description of the functions of each block along with timing, anticipated metric values for the system, discussion of major risks and mitigation plan, and detailed description of each new chip that will be designed as part of this program including functions, input/outputs signals, packaging diagram, timing requirements and targeted process technology.

Hardware Test Plan

Performers shall provide test plans that will specify the evaluation methodology and process they will utilize to verify the completeness, correctness, functionality, system characteristics, and performance of the Architectural Designs in Phase 1 and the Detailed Designs in Phase 2. The test results must be included in the monthly technical status reports, as appropriate. Additionally, the test plan must inform the T&E Team on how the designs can be evaluated using the A-SST environment.

Application Test Plan

Performers shall provide a plan which describes how they will confirm that the models and designs are executing the optimized AGILE application software correctly and measure the performance.

Hardware Test Suite

Performers shall provide the test vectors, programs and/or other stimulus required to verify the correct functionality and timing of the Architectural Design in Phase 1 and the Detailed Design in Phase 2. An analysis of test coverage must be included in Phase 2.

Application Test Suite

Performers shall provide the test vectors, programs and/or other stimulus required to confirm that the Architectural Design in Phase 1 and the Detailed Design in Phase 2 are executing the optimized AGILE application software correctly and measure the performance. An analysis of completeness must be included in Phase 2.

Application Framework

Performers shall provide the framework to be used to develop and optimize AGILE Applications for their Architectural Design in Phase 1 and their Detailed Design in Phase 2. This most likely includes a bare-bones compiler and other simple tools.

Application Software

Performers shall…