IARPA-BAA-24-02 SoURCE CODE BAA V1.6 FINAL 04 16 2024_C.pdf

IARPA

BROAD AGENCY ANNOUNCEMENT

IARPA-BAA-24-02

SoURCE CODE

“SECURING OUR UNDERLYING RESOURCES IN CYBER ENVIRONMENTS”

Release Date:

November 15, 2023

AMENDMENT 001: December 13, 2023

AMENDMENT 002: January 19, 2024

AMENDMENT 003: February 2, 2024

AMENDMENT 004: February 28, 2024

AMENDMENT 005: March 12, 2024

AMENDMENT 006: April 16, 2024 i

(U) CHANGE LOG

REVISION DATE SOW

SECTION

PAGE # DESCRIPTION OF CHANGES POC

001 12132023 General

Information, 3.A, 4.A

1, 18-19, & 21

Update Question Submission and

Proposal Due Dates; Update Eligible

Applicants; Update Classified

Addendum Request Date

KJM

002 01192024 General

Information, 4.B.1.c

Section 3:

Detailed

Proposal

Information

1 & 28 Update Question Submission and

Proposal Due Dates; Update RDMP

KJM

003 02022024 General

Information

1 Update Proposal Due Dates KJM

004 02282024 General

Information

1 Update Proposal Due Dates KJM

005 03122024 General

Information

1 Update Proposal Due Date KJM

006 04162024 General

Information

1 Update Proposal Due Date KJM ii

BROAD AGENCY ANNOUNCEMENT: IARPA-BAA-24-02

TABLE OF CONTENTS

GENERAL INFORMATION

OVERVIEW

SECTION 1: FUNDING OPPORTUNITY DESCRIPTION

1.A. Program Overview

1.A.1 Technical Challenges and Objectives

1.A.2 Program Phases

1.A.2.a Phase 1

1.A.2.b Phase 2

1.B Team Expertise

1.C Program Scope and Limitations

1.D. Program Data

1.D.1 Development Data

1.D.1.a Government Research and Evaluation Data

1.D.1.b Performer Supplemented Data Sources

1.E Testing & Evaluation

1.F Program Metrics

1.G.1 Waypoints

1.G.2 Program Milestones and Deliverable timelines

1.G.3 Software Deliverable Formatting

1.G.4 Program API

1.H. Meeting and Travel Requirements

1.H.1 Technical Exchange Meetings and Workshops

1.H.2. Site Visits

1.I. Period of Performance

1.J. Place of Performance

1.K. References

SECTION 2: AWARD INFORMATION

SECTION 3: ELIGIBILITY INFORMATION

3.A. Eligible Applicants

3.A.1 Organizational Conflicts of Interest (OCI)

3.A.2 Multiple Submissions to the BAA iii

3.B. U.S. Academic Institutions

3. C. Other Eligibility Criteria

3. C.1 Collaboration Efforts

SECTION 4: DOCUMENT PREPARATION PRE-APPROVAL, PROPOSAL, AND SUBMISSION

INFORMATION

4.A Approval to Receive Classified Addendum and Prepare and Submit a Classified Proposal 21

4. A. 2 Proposal Information

4. B. Proposal Format and Content

4. B.1 Volume 1: Technical and Management Proposal

4. B.1.a Section 1: Cover Sheet & Transmittal Letter

4. B.1.b Section 2: Summary of Proposal (see below for page limit)

4. B.1.c. Section 3: Detailed Proposal Information

4.B.1.d. Section 4: Attachments

4.B.2. Volume 2: Cost Proposal (No Page Limit)

4. B.2.a Section 1: Cover Sheet

4. B.2.b. Section 2: Estimated Cost Breakdown

4. B.2.c. Section 3: Supporting Information

4. C. Submission Details

4. C.1. Due Dates

4. C. 2 Proposal Delivery

4. D. Funding Restrictions

SECTION 5: PROPOSAL REVIEW INFORMATION

5.A. Technical and Funding Availability Evaluation Factors

5.A.1. Technical Evaluation Factor (technical criteria listed below)

5. A.1.a. Overall Scientific and Technical Merit

5. A.1.b. Effectiveness of Proposed Work Plan

5. A.1.c. Contribution and Relevance to the IARPA Mission and Program Goal

5. A.1.d Relevant Experience and Expertise

5. A.1.e Resource Realism

5.A.2. Funding Availability Factor

5. A.2.a. Budget Constraints

5. A.2.b. Program Balance

5.B. Method of Evaluation and Selection Process

5.C. Negotiation and Contract Award

5.D. Proposal Retention iv

5.E. Permission to Share with USG Partners

SECTION 6: AWARD ADMINISTRATION INFORMATION

6.A. Award Notices

6.B. Administrative and National Policy Requirements

6.B.1. Proprietary Data

6.B.2. Intellectual Property

6.B.3. Human Use

6.B.4. Animal Use

6.B.5. Publication Approval

6.B.6. Export Control

6.B.7. Subcontracting

6.B.8. Reporting

6.B.9. System for Award Management (SAM)

6.B.10. Representations and Certifications

6.B.11. Lawful Use and Privacy Protection Measures

6.B.12. Public Access to Results

6.B.13. Other Contract Requirements

6.B.13.a. Provisions

6.B.13.b. Clauses

Appendix A: Templates for Volume 1: Technical Proposal

Appendix A.1 Cover Sheet for Volume 1: Technical Proposal

Appendix A.2 Academic Institution Acknowledgment Letter

Appendix A.3 Intellectual Property Rights

Appendix A.4 Organizational Conflicts of Interest Certification Letter

Appendix A.5 Three Chart Summary of the Proposal

Appendix A.6 Research Data Management Plan (RDMP) BAA 24-02

Appendix B: Templates for Volume 2: Cost Proposal

Appendix B.1 Cover Sheet for Volume 2: Cost Proposal

Appendix B.2 Prime Contractor/Subcontractor Cost Element Sheet for Volume 2: Cost Proposal .. 61

Appendix B.3 Software and IP Costs

Appendix B.4 Travel Costs Trip breakdown

Appendix B.5 Contract Deliverables Table

GENERAL INFORMATION

This notice constitutes a Broad Agency Announcement (BAA) and sets forth research of interest in the area of novel technologies to assist forensic experts in making determinations of the most likely attackers, based on coding styles. The solicitation process will follow Federal Acquisition Regulation (FAR) Part

35, Research and Development Contracting, as supplemented with additional information included in this notice. Awards based on responses to this BAA will be considered the result of full and open competition.

Important dates and contact information:

1. Federal Agency Name – Office of the Director of National Intelligence (ODNI)/Intelligence

Advanced Research Projects Activity (IARPA)

2. Funding Opportunity Title – Securing our Underlying Resources in Cyber Environments “SoURCE

CODE” Program

3. Announcement Type – Initial

4. Funding Opportunity Number – IARPA-BAA-24-02

5. Catalog of Federal Domestic Assistance Numbers (CFDA) – Not applicable

6. Questions:

Submit unclassified technical, administrative or contracting questions by email to: Unclassified:

DNI-IARPA-BAA-24-02@iarpa.gov. Do not send classified content or questions with proprietary content to the above email alias. Alternatively, contact the unclassified DNI-IARPA-

BAA-24-02@iarpa.gov e-mail BAA alias to coordinate an appropriate medium of delivery for ALL classified questions. All questions and delivery requests must include the full name and affiliation of the requester with contact information. Unclassified questions and answers and BAA amendments will be will be posted to SAM.gov. Classified questions and answers and BAA amendments will be distributed via a classified medium. No answer will go directly to the submitter.

7. Dates:

7.1 BAA Release Date: November 15, 2023 (Classified Addendum to be released separately, See

BAA Section 4.A)

7.2 BAA Question Submission Due Date: November 30, 2023; UPDATE: Tuesday, January 2, 2024 @ 5:00 PM EST

7.3 Proposal Due Date:

• Technical Area (TA) 1 only: January 22, 2024 @ 5:00 PM EST

• UPDATE: TA 1 only: Friday, February 16, 2024 @ 5:00 PM EST

• UPDATE: TA 1 only: Friday, March 1, 2024 @ 5:00 PM EST

• UPDATE: TA 1 only: Friday, March 15, 2024 @ 5:00 PM EST

• UPDATE: TA 1 only: Friday, March 29, 2024 @ 5:00 PM EST

• UPDATE: TA’s 1, 2, and 3: Friday, February 16, 2024 @ 5:00 PM EST

• UPDATE: TA’s 1, 2, and 3: Friday, March 1, 2024 @ 5:00 PM EST

• UPDATE: TA’s 1, 2, and 3: Friday, April 5, 2024 @ 5:00 PM EST

• UPDATE: TA’s 1, 2, and 3: Friday, April 19, 2024 @ 5:00 PM EST

• UPDATE: TA’s 1, 2, and 3: Friday, May 3, 2024 @ 5:00 PM EST

• UPDATE: TA’s 1, 2, and 3: Friday, May 31, 2024 @ 5:00 PM EST

8. Anticipated individual awards – Multiple awards anticipated

9. Types of instruments that may be awarded – Procurement Contracts or Other Transactions1

1 Procurement Contract: This is a standard government contract that follows the processes, format and terms and conditions as outlined in the Federal Acquisition Regulations (FAR) and supplementing Agency specific regulations.

mailto:DNI-IARPA-BAA-24-02@iarpa.gov

10. Agency Points of Contact

ATTN: IARPA-BAA-24-02

Office of the Director of National Intelligence

Intelligence Advanced Research Projects Activity

Washington, DC 20511

Electronic mail: DNI-IARPA-BAA-24-02@iarpa.gov (unclassified)

11. Program Manager (PM) – Dr. Kristopher Reese

12. Program Website – www.iarpa.gov/research-programs/source-code

13. BAA Summary – The SoURCE CODE program is seeking to provide novel technologies to assist forensic experts in making determinations of the most likely attackers, based on coding styles. The program will explore full feature spaces in binary code and source code files to measure the similarity between files and provide information to forensic experts to the likely origins (country, groups, individuals, etc.). This capability would enable the automated matching of similar binaries from known samples, allowing analysts to speed up the attribution of malicious attacks to improve law enforcement and intelligence community responses.

OVERVIEW

SECTION 1: FUNDING OPPORTUNITY DESCRIPTION

The Intelligence Advanced Research Projects Activity (IARPA) often selects its research efforts through the Broad Agency Announcement (BAA) process. The use of a BAA solicitation allows a wide range of innovative ideas and concepts. The BAA shall appear first under Contract Opportunities on https://SAM.gov, then the IARPA website at http://www.iarpa.gov/. The following information is for those wishing to respond to this Program BAA.

This BAA (IARPA-BAA-24-02) is for the Securing our Underlying Resources in Cyber Environments

(SoURCE CODE) Program. IARPA is seeking innovative solutions for the SoURCE CODE Program in this BAA, which is envisioned to be a 30-month effort across all focus areas, beginning approximately

July 2024. Technical Area 1 is unclassified. Technical Areas 2 and 3 are classified efforts. A Classified

Addendum to this BAA, including the Description for Technical Areas 2 and 3 will be distributed separately to qualified offerors. See SECTION 4.

1.A. Program Overview

The volume, methods, and complexity of cyberattacks on companies and infrastructure has grown significantly and will continue to evolve over time. These cyberattacks – whether on government systems or private companies – pose a serious threat to national security. At the same time, there is a shortage of cyber expertise to fill all necessary cyber-focused positions in the commercial world. This challenge holds true with cyber-forensic experts, who play an important role in attributing these attacks to assist with informing companies and governments on the threats that are facing the U.S.

Attribution of these malicious attacks can work to disrupt criminal cyber capabilities and improve law enforcement and intelligence community responses to attacks. The Diamond Model of Forensic attribution

[1] is a common methodology used by forensic experts to attribute attacks on systems. The diamond model captures the process that forensic experts go through by looking at multiple aspects of the information

Other Transactions: Agreements generally are not subject to the federal laws and regulations governing procurement contracts and thus are not required to comply with the Federal Acquisition Regulations (FAR), its supplements or laws that are limited in applicability to procurement contracts. They may be used with non-traditional contractors under certain circumstances.

http://www.iarpa.gov/research-programs/source-code available to include each point on the diamond (as shown in Figure 1) – this includes the Victim of the attack, the capabilities used in the attack, and infrastructure used in the attack. With this information, the diamond model helps to infer the likely attacker. The SoURCE CODE program is not seeking to automate the entirety of attribution (the entirety of the diamond model of forensic attribution). Instead, SoURCE

CODE will seek to automate and assist forensic experts on the Attacker-Capability edge (as shown in the green oval in Figure 1).

Figure 1. The Diamond Model of Forensic Attribution: showing the edge that the SoURCE CODE program is seeking to assist forensic experts.

SoURCE CODE will create novel methods and technologies that go beyond the common lexical and syntactic features that have been explored in prior literature; these features might include semantic and behavioral features that may become salient features for similarity and demographic traits. The Program will create scientifically validated forensic similarity and analytic technologies that allow for a measure of similarity of code and binaries as well as provide additional information or components of the code that may assist in analyzing hidden demographic information (groups, countries, or individuals where possible).

SoURCE CODE systems shall provide evidence or explanations, beyond just a similarity score, to assist forensic experts in making a final attribution. Figure 2 shows the expected tasks in this effort – including the expected outputs explored by performers.

Figure 2. The SoURCE CODE vision: Forensic systems capable of measuring similarity and providing additional information about the likely demographics of the author(s).

SoURCE CODE seeks to go beyond the limitations of existing academic literature in the space that utilizes limited subsets of hand-crafted features in largely the lexical and syntactic domains. SoURCE CODE seeks to explore the semantic and behavioral features found within code to extract information and determine similarity. SoURCE CODE will also explore methods to provide evidence or explanations to the forensic experts to assist in their final determination.

Offerors to the SoURCE CODE program shall propose against the main goals of the similarity and demographic focus areas to ensure that their models will be able to meet the goals of the program. Offerors shall propose methods to explore a fuller feature space (to include the possibility of learned features) to identify salient features that may be invariant across datasets. Using those feature sets, offerors will also propose similarity or demographic analytic algorithms for measuring source code to source code, binary to binary, and in later phases source code to binary and vice versa. Additionally, offerors shall propose methods in which significant evidence or explanations can be provided back to forensic experts to make a final determination.

A robust, unclassified data collection initiative will be part of the SoURCE CODE Program to assemble diverse text types for use in Research and Development (R&D) for algorithm training, and in T&E for comparing the algorithm performance against ground truth. The T&E Team will conduct dataset development activities throughout the life of the program using Institutional Review Board (IRB)-approved

Human Subject Research (HSR) protocols. A portion of the data will be held out for T&E, the remaining portion will be made available to Performers for R&D. The Test and Evaluation cycle is described in more detail in 1.E Testing & Evaluation. Additional details on program data can be found in 1.D Program Data.

The SoURCE CODE Program will pursue rigorous and comprehensive independent T&E to ensure that research outcomes are well characterized, deliverables are aligned with program objectives, and performance is measured across the full range of data conditions. T&E activities will not only inform

Government stakeholders on SoURCE CODE research progress but will also serve as valuable feedback to the Performers to improve their research approaches, algorithm training practices, and system development.

The SoURCE CODE Program will work closely with experts in the field of Engineering and Forensic attribution of software to continually refine and improve T&E methodologies.

1.A.1 Technical Challenges and Objectives

Offerors shall address three focus areas within the Technical Area 1 (TA1) to meet the goals of SoURCE

CODE to develop novel, human interpretable similarity and demographic algorithms. Offerors must propose novel approaches to each of these focus areas, and if selected as a Performer, Performers will be required to create an end-to-end technology with components from each focus area. These focus areas include:

Feature Space Generation and Extraction: The goal of this focus area is to research and develop an encoding scheme that projects source code and binary representations of software into an optimized feature space wherein observations from the same classes are projected near one another, regardless of the representation of the software (binary or source code). Approaches may use entirely learned features (i.e., Neural Network approaches), hand-crafted, or a combination of features. Successful approaches must utilize features that are predictive of class similarity and/or demographic information (country, group, individual). Deliverables will be evaluated by the T&E team using metrics outlined in 1.F Program

Metrics.

Similarity and Demographic Analytic Algorithms: The goal of this focus area is to research and develop a system that will successfully identify similar binaries within a corpus using the stylistic characteristic of the software being analyzed. Systems must leverage representations from the feature space encoder to determine whether two or more software representations are similar or come from a specific author, group, or country. The software similarity and demographic analytic algorithms will be evaluated by the T&E team using metrics outlined in 1.F Program Metrics.

System Explainability: Systems developed in TA1 must provide human-interpretable feedback providing evidence or explaining why specific documents likely are, or are not, similar or from a specific demographic group. Offerors shall propose novel evidentiary or explainability techniques to address this goal, and methods to evaluate this approach that are potentially generalizable to other performer approaches, including justifiable metrics. T&E Teams will finalize the protocol for evaluating explanations in Phase 1 and implement this protocol in evaluations during subsequent phases.

Figure 3 provides an example of the TA1 software components that will be expected under this program.

T&E teams will design component interfaces to ensure that systems are modularized, allowing for addition or removal of components to meet government transition needs and additional evaluations.

Figure 3. Performer Outputs for TA1. Inputs will be source code and/or binary representations of software. The performers will be expected to featurize and create similarity and/or demographic analytic systems with explainability components.

1.A.2 Program Phases

The SoURCE CODE program is anticipated to be a 2.5-year (30 month) effort, comprised of two (2) phases.

Both phases are being solicited under this BAA. Phase 1 will be 18 months in duration and phase 2 will be

12 months. The phases will encompass all of the TA1 focus areas described in previous sections (1.A.1

Technical Challenges and Objectives and 1.A.2 Program Phases).

Offerors are required to propose to both phases and all focus areas within TA1 of the Program under this BAA. Proposals that omit any of phases or focus areas will be considered non-compliant.

SoURCE CODE aims to create systems that can accomplish similarity and demographic analytics under multiple conditions encountered by Law Enforcement and U.S. Intelligence Community stakeholders.

Program phases are designed to test Performer systems against increasingly challenging data scenarios.

Differences in phases are shown in Table 1.

Table 1. SoURCE CODE Program Phases and progression of challenges between both phases, data permitting. * Closed set means the target is within the set; an Open set indicates that the target may not be within any class in the set (unseen example).

Area Phase 1 (18 months) Phase 2 (12 months)

Attribution domains Source Code → Source Code

Binary → Binary

Source Code → Binary

Binary → Source Code

Number of groups and demographic attributes Fifty (50) Seventy or more (70+)

Type of Set * Closed Set Open set

Each program phase will have multiple milestone deliverables at which performers will submit software deliverables to the T&E team for independent evaluation against datasets, to include hidden datasets.

Deliverables will occur in 4 month increments and include featurization methods for testing, along with similarity and demographic analytic algorithms. Deliverables are outlined in Figure 4.

Figure 4. Mapping of TA milestones and activities to Phases. Also shows a schedule of key milestones and deliverables.

Evaluations will be performed on existing and sequestered datasets using performer-submitted, containerized software components. Aspects of evaluation that involve human adjudication (i.e.

explainability) or feedback will be performed by the T&E team, but performers may self-test using human evaluators.

In Phase 1, IARPA will provide development datasets and an evaluation harness developed by the T&E team and provided at program kick-off, and any updates will be provided as they are released during the life of the program. The same harness will be used for testing and evaluation of Performer-submitted software components at each subsequent evaluation milestone. The purpose of the evaluation harness is to enable performer-site training and testing for both the feature space, similarity, and demographic analytic algorithms. IARPA will also furnish documentation of the program Application Programming Interface

(API) developed by the SoURCE CODE T&E Team. This Government Furnished Information (GFI) will facilitate productive and useful research in a consistent and standardized manner.

Datasets developed by the T&E Team for each program phase will be partitioned into development and evaluation datasets. Although the T&E Team will deliver development data, Performers may seek to supplement their own development datasets aggregated from open source or self-generated and using creative techniques and heuristics to infer classes. These datasets must be submitted to the T&E Team to be combined into a single dataset, such that it is releasable to all Performers as discussed in 1.D.1.b External

Data Sources.

Evaluation of Performer progress towards program metrics will primarily utilize evaluation datasets that are sequestered. A portion of these datasets will be shared with Performers after each evaluation to enable

Performers to conduct error analyses. The baseline API provided in Phase 1 will standardize system and subcomponent design and streamline comparative evaluations. More details on the datasets are available in

1.D. Program Data. and more details on the API are available in 1.G.4 Program API.

1.A.2.a Phase 1

The goal of Phase 1 is to develop new methods and explore the feature space between source code to source code and binary to binary representations of software. Featurization, similarity matching, and demographic analytic systems in this phase should succeed at performing rank and retrieval and open search tasks using inputs of both source code and binary representations. Systems will only be expected to query like representations (source code to source code, binary to binary) matches against a large corpus of software.

Throughout the first phase of the program, Performer research will focus on foundational explorations and investigations into different approaches, theories, and concepts to establish the building blocks of their

SoURCE CODE system(s). During this time, performers and the T&E team will collaborate on a protocol to evaluate human explainability. This protocol will be implemented towards the end of Phase 1 and carry into Phase 2 of the program. As a precursor to this collaboration, proposals in response to this BAA must articulate a novel approach to the explainability challenge, addressing how their SoURCE CODE systems would furnish explanations to users allowing them to understand and verify system behavior. Two categories of an end user are envisioned: a non-expert analyst using the system explanations to understand system rationale, flag spurious attributions, gauge attribution confidence, etc.; and an expert with sophisticated reverse engineering and forensic, knowledge conducting highly nuanced attribution analysis of attacks. Offerors must be explicit in their proposals about how their explanatory techniques could be evaluated. This information will ensure that our testing methodology for explainability will properly capture any techniques that you are exploring.

With the aims of mitigating risk and investigating competing approaches, performers are encouraged to pursue several novel approaches under the same subtopic. Performers will compare and contrast various approaches using development datasets.

1.A.2.b Phase 2

The goal of phase 2 is for Performers to extend the capabilities developed in phase 1. Featurization, similarity matching, and demographic analytic systems in this phase should succeed at performing rank and retrieval and open search tasks using inputs of both source code and binary representations. Systems will be expected to query opposite representations (source code to binary, binary to source code) matches against a large corpus of software.

1.B Team Expertise

Collaborative efforts and teaming among Offerors are highly encouraged. It is anticipated that teams will be multidisciplinary and shall include expertise in one or more of the disciplines listed below. This list is included only to provide guidance for the Offerors; satisfying all the areas of technical expertise below is not a requirement for selection and unconventional or innovative team expertise may be needed based on the proposed research. Proposals shall include a description and the mix of skills and staffing that the

Offeror determines will be necessary to carry out the proposed research and achieve program metrics.

• Binary Analysis

• Compiler Design

• Software Engineering

• Programming Languages

• Authorship Attribution

• Threat Intelligence Analytics

• Cybersecurity

• Cyber-Forensics & Reverse Engineering

• Stylometry

• Artificial Intelligence

• Machine learning

Although the SoURCE CODE program is focused on automated approaches, teaming schemes that include meaningful collaborations with forensic experts, software reverse engineers, and practitioners of non-computational attribution methods are highly encouraged.

1.C Program Scope and Limitations

Proposals shall explicitly address all of the following:

• Underlying theory: Proposed strategies to meet program-specified metrics must have firm theoretical bases that are described with enough detail that reviewers will be able to assess the viability of the approaches. Proposals shall properly describe and reference previous work upon which their approach is founded.

• Research & Development approach: Proposals shall describe the technical approach to meeting program metrics.

• Technical risks: Proposals shall identify technical risks and proposed mitigation strategies for each identified risk.

• Software development: Proposals shall describe the approach to software architecture, modularization, and integration.

The following areas of research are out of scope for the SoURCE CODE program:

• Research that does not have strong theoretical and experimental foundations

• Approaches that rely on features outside of source code or binary executables (e.g. timestamps, network structure, multimedia, etc.).

• Approaches that rely solely on integration of existing cybersecurity solutions without experimental or scientific improvements.

• Approaches that focus solely on utilizing features from only the lexical and syntactic domains

• Approaches that rely solely on running the software is out of scope. But we are looking for innovative approaches to dynamic and static analysis.

• Improving natural language authorship attribution (i.e., for code comments). Use of these systems is encouraged in addition to the methods being explored, and modifications to existing systems is allowable only if required to improve code comment specific functionality – but improvement of natural language attribution techniques is out of scope.

• Development or Analysis of hardware systems

• Approaches that consist merely of integrating currently existing software systems.

• Developing new adversarial approaches to misattribute or cause errors in attribution or demographic systems.

Performers are permitted to use data outside of the standard data provided by T&E teams, however no proprietary data should be used. Please include any datasets that you wish to share within your proposal to the effort. Any additional data from proposers shall be distributed to T&E for dissemination to any and all selected performers on SoURCE CODE; datasets will not be made public beyond the program teams. We expect a diverse set of programming languages (such as C, C++, Rust, Python, and so on) and binaries

(ELF, EXE) but details about the final datasets will be shared at kickoff.

Delivered software will be evaluated by the T&E team on sequestered evaluation datasets. During each performance period, performers will build software systems that will be run and evaluated by the T&E team. Testing protocols do not allow for expert operators – systems must be fully automated. Systems or algorithms that have been trained using human-in-the-loop methods may be submitted, provided that they can be run autonomously.

1.D. Program Data

The SoURCE CODE program seeks to simulate the challenge of forensic similarity measures from real-world circumstances faced by threat intelligence analysts. Current academic datasets have a number of shortcomings that render them inadequate to understand the full scope of the challenge program, including:

• Understanding the impact of group codebases vs. a single individual’s code

• Understanding the impact and influences of project domains (e.g. Android vs. iOS vs. Windows

vs. Linux)

• Understanding the impacts and influences of development tools:

o Integrated Development Environments (IDEs) o Version Control Systems o Compiler choices o Build Environment o Deployment tools o Novel concepts proposed by performers

• And the impact of things like company style guides or project naming conventions.

SoURCE CODE Test and Evaluation teams will be seeking to fill some of these gaps through curation of datasets, annotating datasets, and/or building datasets to fill these gaps. Ground truth in these datasets will represent a diverse set of demographic data (with minimally country, group, author). Many types of malware will be considered within the datasets, which may include packed malware or malware with other obfuscation techniques. More details about the datasets will be provided at program kickoff.

The program will include two types of data – Development Data and Evaluation Data. Development data will be utilized by Performers, as they see fit, to conduct research, development, error analysis, and algorithm training. Evaluation data will be explicitly excluded from any algorithm training approaches and will be used by the T&E Team for performance evaluations of subcomponents, modules, and systems measured against the SoURCE CODE program metrics.

Offerors may include datasets that they may own if such data is not proprietary and can be shared among all selected performers.

1.D.1 Development Data

Development data will be utilized by Performers, as they see fit, to conduct internal research, development, error analysis, and algorithm training. Each Performer is anticipated to have a unique technical solution, and accordingly may require different development data depending on their approach.

Datasets will be collected, annotated, and curated by the T&E Team. Some of this data may be sequestered, and a portion will be provided to Performers for R&D and error analysis.

1.D.1.a Government Research and Evaluation Data

The T&E team will curate, annotate, and collect datasets to be made available to SoURCE CODE performers. Some data will be sequestered for the purpose of blind testing and evaluation. The government team will also work to augment these datasets with a collection effort to create human-authored ground-truthed source code and binary executable software and will provide some of this data upon anonymization and availability of the data in accordance with any appropriate Human Subject Research authorities.

1.D.1.b Performer Supplemented Data Sources

Performers may propose to utilize external datasets obtained by Performers that are available from third parties or that have been collected by the Performer outside of the SoURCE CODE program. An example of a third-party dataset would be an authorship attribution dataset, collected by a university under an IRB-approved HSR protocol, that has been approved by the dataset owner for release to the research community.

Data collected by a Performer under a different program are considered external data, even if the other program’s data collection was Government-sponsored.

All external datasets must be approved for use in the SoURCE CODE program by IARPA during contract performance, in accordance with HSR and applicable privacy policies, statutes, and regulations. As part of their proposal, Offeror’s shall prepare a SoURCE CODE Privacy Plan v1.0 that comprehensively describes the efforts the Offeror will take to protect personally identifiable information (PII) and United States

Persons (USP) information, and safeguard the security of any personal data collected and that of any devices, applications, networks, or services involved in collection, transmission, processing, and storage of such data.

This plan shall include but is not limited to: 1) All performers shall provide to the Government annual privacy training certificates of completion for all personnel on this effort to document and instill the importance of data protections, 2) creation of a data protection plan that addresses the performer’s safeguarding approach i.e, for encrypting the data at rest and in motion for any PII and/or USP data, and 3)

Notifying the IARPA Program Manager and Contracting Officer when there is any data spill or leak. Any claims that data are anonymous must be based on evidence and supported with sufficient information regarding how the data have been anonymized. Each Performer’s SoURCE CODE Privacy Plan shall cover all data categories described in this BAA; relevant information on GFI datasets will be provided by

SoURCE CODE T&E in advance of data release.

Performers may not use proprietary datasets unless these datasets are made available to all R&D performers and the T&E Team in the SoURCE CODE program. Public release of proprietary datasets is not a requirement; however, release for use within the SoURCE CODE program is required. Moreover, for any dataset not collected under the scope of the SoURCE CODE program, performers must provide the

Government with an accounting of all resources used and sources from which data are drawn and describe how the data will be used for development, testing, and training of algorithms.

1.E Testing & Evaluation

T&E will be conducted by an independent team of Government and contractor staff carrying out evaluation and analyses of performer research deliverables using program test datasets and protocols. The SoURCE

CODE data and test protocols (see 1.D.1.a Government Research and Evaluation Data) will be the primary mechanism by which the T&E team carries out their evaluations.

The SoURCE CODE program will pursue rigorous and comprehensive T&E to ensure that research outcomes are well characterized, deliverables are aligned with program objectives, and that the algorithm performance is measured across a full range of conditions. Such T&E activities will inform IARPA and

Government stakeholders on SoURCE CODE research progress and serve as invaluable feedback to performers to improve their research approaches, algorithmic training, and system development. The

SoURCE CODE program will work closely with government leaders in malware analysis and reverse engineering to refine and improve the T&E methodologies.

The government will provide performers with a uniform Evaluation Harness (EH) application programming interface (API) – to be provided at program kick-off - with relevant scripts to run program test protocols on program datasets. The design of the EH will be informed by prior work conducted by the T&E team and other government agencies. This evaluation environment will be the platform for independent government testing of performer deliverables.

Figure 5. Test and Evaluation. T&E team will provide and interface (e.g. Docker Containers / APIs) that will be used to automate the testing. Offerors will submit their feature space and similarity/demographic algorithms for independent testing and be tested for feature invariance or compared against baseline systems.

Performers will have specific deliverable milestones at which all subcomponent, system algorithms, and software will be delivered to IARPA and its designated T&E Team. The T&E Team will then conduct evaluations at the direction of the SoURCE CODE Program Manager (PM) and with the objective of characterizing the quality, functionality, and performance of the SoURCE CODE deliverables. In addition to quantitative measurements, T&E will be carried out to establish a thorough understanding of the progress, status, and limitations of the performer’s research.

T&E results and feedback will be provided to performers at regular intervals to keep them abreast of current independent performance measurements and to inform and improve their R&D approaches and methods.

T&E results from all performers will be shared with all teams to establish an understanding of the current state and progress of SoURCE CODE research; T&E results will also be shared with government stakeholders, including their contractors, for Government purposes. For example, a Principal Investigator

(PI) Review Meeting will be held annually to share research ideas, progress, and results across the SoURCE

CODE program (see 1.G.2 Program Milestones and Deliverable timelines).

IARPA may conduct other supplemental evaluations or measurements in its sole discretion to evaluate the

Performers’ research and Deliverables.

1.F Program Metrics

Achievement of metrics is a performance indicator under IARPA research programs. IARPA has defined

SoURCE CODE program metrics to evaluate effectiveness of the proposed solutions in achieving the stated program goal and objectives, and to determine whether satisfactory progress is being made. The metrics described in this BAA are shared with the intent to scope the effort, while affording maximum flexibility, creativity, and innovation to Offerors proposing solutions to the stated problem.

The final SoURCE CODE T&E protocols and evaluation methodology will be provided at program kickoff.

Program metrics may be refined during the various phases of the SoURCE CODE program; if metrics change, revised metrics will be communicated in a timely manner to Performers. The evaluation methodology may be revised by the Government at any time during the program lifecycle to better meet program needs.

SoURCE CODE program target metrics were selected based on three factors:

1. What is technically achievable but challenging based on current state-of-the-art in software similarity measurements;

2. What is statistically measurable based on the planned program evaluation data; and

3. What is useful to mission partners based on USG stakeholder needs and use cases.

The following is a description of primary program metrics and target scores. Score targets will increase for each successive program phase. SoURCE CODE performance metrics are designed to evaluate the software component deliverables for similarity and demographic tasks.

Performer software submissions for similarity and demographic estimation tasks are evaluated using standard machine learning classification metrics – along with standard biometric measures, such as d’. The recognition task tests whether a system can use a query document from a given author, group, country (or other demographic data) and retrieve documents from a set of documents produced by the same author or documents most likely from the same demographic class. At test time, the Performer system must generate feature encodings for a set of author query documents as well as documents from a retrieval database. Using

Performer system generated encodings, the T&E Team will retrieve a ranked list of nearest neighbors for each author query. The program does not place constraints on the properties of the feature space; however, teams must stipulate an encoding that allows calculation of similarity.

Specific metrics of interest include: Top-N accuracy (specifically Top-1 and Top-10) which are used to measure the success of the system in bubbling up documents from the demographic class being tests.

(Accuracy is chosen over Recall as proper classification of negative samples is also important).

Additionally, Equal Error Rate (EER) is to be calculated as a measure of the Receiver Operating Curve

(ROC) – a measure of the point where False accept and False Reject errors are equal. In a similar vein, measurements of the d-prime (d′) measure is necessary to determine the discriminability between the

“genuine” and “imposter” signals to assist in determining if feature representations are viable for these problems. Additional metrics may be calculated, to understand the challenge of the task and how well performer systems are performing on the tasks.

A summary of metric targets by phase is shown in Table 2 for similarity metrics and Table 3 for estimating demographic trait metrics; these metrics are subject to change over the course of the program. Accuracy targets are defined as aggregated statistical measures for a given phases’ evaluation datasets, which will contain binaries and source code from a broad range of software types, build systems / compilers, and authors with a diverse background in software development. These targets are estimates informed by current capabilities and use case requirements. To mitigate risk and better inform the Government regarding the appropriateness of phase-over-phase performance targets, the T&E Team will produce baseline systems for each task approximating current state-of-the-art. Further information about baseline models will be furnished to Performers at program kickoff.

Table 2. Metrics for similarity tasks in both phases of the program. Metrics shown are for 1,000 unique users and should scale to match cardinality of unique users. Additional numbers shown in brackets are for 100 and 10,000 authors respectively.

Phase I Phase II

Top-1 Accuracy [100/10,000] 85% [95/75] 90% [98/90]

Top-10 Accuracy [100/10,000] 95% [99/85] 96% [99/95]

EER Average 30% 20% d′ (sensitivity index) Measured increase 30% over Ph. 1 measurement

Avg. Processing Time on commodity hardware

(max per binary/source)

5 minutes 2 minutes

Attribution methods

(data permitting)

Source → Source, Binary →Binary

Source → Binary, Binary →Source

Table 3. Metrics for demographic tasks in both phases of the program. Metrics shown are for 1,000 unique users and should scale to match cardinality of unique users. Additional numbers shown in brackets are for 100 and 10,000 authors respectively.

Phase I Phase II

Top-1 Accuracy [100/10,000] 85% [95/75] 90% [98/90]

Top-10 Accuracy [100/10,000] 95% [99/85] 96% [99/95]

EER Average 20% 10%

Avg. Processing Time on commodity hardware

(max per binary/source)

5 minutes 2 minutes

# Groups/Demographics

(data permitting)

50 70+

Type of Set Closed Open

1.G.1 Waypoints

Waypoints are the means by which the Performer clearly demonstrates the quantitative and timely progress that must be made for the overall concept to meet end-of-phase Milestones. In other words, the intent of

Waypoints is to provide a clear measure of progress towards meeting the program Milestones so the PM and advisors can provide more effective guidance and assistance to the Performers. Performance against these Waypoints will be reviewed throughout the program, and the PM and advisors will use performance against the Waypoints to assess whether course corrections are needed to ensure program success.

The Government has defined Waypoints for all Performers that are included in the Program Milestones and

Deliverables (see 1.G.2 Program Milestones and Deliverable timelines). Progress against these Waypoints will be reviewed during site visits.

1.G.2 Program Milestones and Deliverable timelines

Table 4 shows the expected recurring events and deliverables that will occur across both phases of the program by all performers. Table 5 shows the expected events that will occur for all performers across both phases. Table 6 shows the expected deliverables for all performers across both phases of the program.

Offerors may add other waypoints or deliverables in addition to the minimum set listed in the tables below.

Table 4. Recurring Events and deliverables expected by all performers on SoURCE CODE across both phases.

Recurring Event / Deliverable Frequency Description

PI Virtual Meetings / Calls Monthly

Monthly teleconference with SoURCE CODE

PM, SETA, and T&E teams.

Monthly Status Reports

Monthly

Due on the 15th of each month; Technical and costs

DELIVERABLE: MSR documentation

Table 5. Events that are expected to occur during the life of the program. Locations are TBD; (please expect many events to occur in Washington, D.C.)

Events Month Description

Start Phase 1

Program Phase 1 Kickoff 0

Location: TBD; Attendance, presentation as requested by the PM, addressing performer questions as needed.

Presentation shall be shared with the PM 5 calendar days prior to the kickoff meeting. Information shared about system harness, APIs furnished to Performers; T&E present guidelines for developing protocol for explainability and utility metrics.

Site Visit 4, 12 At performer sites; Discussions about ongoing work with

PM, SETA, and T&E team.

PI Workshop 8, 16

Location: TBD; Attendance in person or virtual;

presentations about ongoing work to all performers and government team and possible government partners.

Star Phase 2

Program Phase 2 Kickoff 19

Location: TBD; Attendance, presentation as requested by the PM, addressing performer questions as needed.

Presentation should be shared with the PM 5 calendar days prior to the kickoff meeting.

Site Visit 24 At performer sites; Discussions about ongoing work with

PM, SETA, and T&E team.

Program Closeout 30

Location: TBD; Attendance in person or virtual;

presentations about work completed, lessons learned, and any other materials to close out the program. Audience will include all performers, government team, and government partners.

Table 6. Deliverables expected by performers across the life of the program.

Task/Deliverable Month Description

Start Phase 1

Performer Deliverables

(TA1: Feature Space)

System submission of Feature Space software deliverables to T&E team; Software shall be delivered at the beginning of evaluation. Updated software may be requested if bugs are identified during evaluation.

DELIVERABLE: Software component for Feature

Space– including source code.

Performer Deliverables

(TA1: Feature Space &

Similarity)

8,12,16

System submission of all subcomponents of software system to T&E team; Software shall be delivered at the beginning of evaluation. Updated software may be requested if bugs are identified during evaluation – deadline to update any bugs would be provided at time of request based on severity

DELIVERABLE: Software for all components of system– including source code.

Performer Phase 1 Final

Reports

Phase 1 Final Reports highlighting work completed, metrics achieved, challenges, opportunities for additional experimentation, and all documentation on software system.

DELIVERABLE: Internal Report for IARPA

Start Phase 2

Performer Deliverables

(TA1: Feature Space &

Similarity)

22,26,30

System submission of all subcomponents of software system to T&E team; Software shall be delivered at the beginning of evaluation. Updated software may be requested if bugs are identified during evaluation – deadline to update any bugs would be provided at time of request based on severity.

DELIVERABLE: Software for all components of system– including source code.

Performer Phase 2 Final

Reports

Phase 2 Final Reports highlighting work completed, metrics achieved, challenges, opportunities for additional experimentation, and all documentation on software system.

DELIVERABLE: Internal Report for IARPA

1.G.3 Software Deliverable Formatting

Performers will be required to provide algorithms, software deliverables, design documentation, and operating instructions to execute the automated software in a manner that conforms to a standardized industrial method or methods that will be provided at program Kickoff. To facilitate planning, offerors may assume that the standardized configuration will require the use of software containerization technology

(e.g., Docker and a REST API). This means that the entirety of a Performer’s system, including pre- and post-processing, must be included within the delivered software container. For models that require training, the expectation is for the initial model training to occur on Performer systems, with the ability for the T&E

Team to re-train and test the model with the same and/or other data. Performers shall also provide source code to the government and T&E team to enable modification of Performer containers to suit deployment constraints. Offeror teams that do not include the requisite expertise to conduct such software development shall include costs in their proposal to obtain software development support.

Each team is required to include among their key personnel a Lead System Integrator (LSI) who shall be responsible for preparing software Deliverable subcomponents, modules, and systems, performing quality control of Deliverable, and integrating key components into the primary SoURCE CODE system(s). The

LSI will also oversee communication and coordination across a Performer’s research teams including subcontractors, if applicable, to ensure research products are functional and following software coding best practices (e.g., inline comments, documentation). Additional team members and roles are dependent on the proposed research, as such, there is no predetermined or required skill mix.

1.G.4 Program API

The SoURCE CODE Program will utilize a…