SoURCE CODE Draft BAA (Tech Section Only)_C.pdf

Version Date: July 24, 2023

IARPA

DRAFT

BROAD AGENCY ANNOUNCEMENT

FUNDING OPPORTUNITY DESCRIPTION

IARPA-BAA-24-02

(U) SoURCE CODE

SECURING OUR UNDERLYING

RESOURCES IN CYBER

ENVIRONMENTS

IARPA-BAA-24-02

Release Date: September 19, 2023

INSTRUCTIONS

This Draft Broad Agency Announcement (BAA) is NOT a formal request for proposal.

Proposals are NOT being requested or evaluated at this time. A Final BAA will be released at a later date for which interested parties may submit proposals.

The Intelligence Advance Research Projects Activity (IARPA) is seeking interested parties to thoroughly review the Draft BAA Funding Opportunity Description and provide comments, questions, suggested changes, and feedback by October 11, 2023 in the format provided below.

IARPA does not anticipate posting responses to any comments, questions, suggested changes, and/or feedback received; however, all input will be considered in developing the Final BAA.

Additionally, the Draft BAA Funding Opportunity Description is subject to change in the development of the Final BAA as a result of IARPA’s consideration of the input received from interested parties in response to the Draft BAA.

Please submit all comments, questions, suggested changes, and feedback to the designated procurement email address: dni-iarpa-baa-24-02@iarpa.gov using the below format.

Interested parties will be given the opportunity to submit questions and comments in writing for IARPA response once the Final BAA is issued.

OVERVIEW

SECTION 1: FUNDING OPPORTUNITY DESCRIPTION

The Intelligence Advanced Research Projects Activity (IARPA) often selects its research efforts through the Broad Agency Announcement (BAA) process. The use of a BAA solicitation allows a wide range of innovative ideas and concepts. The BAA shall appear first under Contract

Opportunities on https://sam.gov, then the IARPA website at http://www.iarpa.gov/. The following information is for those wishing to respond to this Program BAA.

This BAA (IARPA-BAA-24-02) is for the Securing our Underlying Resources in Cyber

Environments (SoURCE CODE) Program. IARPA is seeking innovative solutions for the

SoURCE CODE Program in this BAA, which is envisioned to be a 30 month effort across all focus areas, beginning approximately July 2024. Technical Areas 2 and 3 are classified efforts that will be explored under a separate evaluation.

NOTE: If you have the appropriate facilities and clearances to conduct SECRET level research, please contact the IARPA contracts and the IARPA security team to request a classified addendum to this BAA for TA2 and TA3; this is not a requirement for the UNCLASSIFIED

BAA focus areas.

1.A. Program Overview

The volume, methods, and complexity of cyberattacks on companies and infrastructure has grown significantly and will continue to evolve over time. These cyberattacks – whether on government systems or private companies – pose a serious threat to national security. At the same time, there is a shortage of cyber expertise to fill all necessary cyber-focused positions in the commercial world. This challenge holds true with cyber-forensic experts, who play an important role in attributing these attacks to assist with informing companies and governments on the threats that are facing the U.S.

Attribution of these malicious attacks can work to disrupt criminal cyber capabilities and improve law enforcement and intelligence community responses to attacks. The Diamond Model of

Forensic attribution [1] is a common methodology used by forensic experts to attribute attacks on systems. The diamond model captures the process that forensic experts go through by looking at multiple aspects of the information available to include each point on the diamond (as shown in

Figure 1) – this includes the Victim of the attack, the capabilities used in the attack, and infrastructure used in the attack. With this information, the diamond model helps to infer the likely attacker. The SoURCE CODE program is not seeking to automate the entirety of attribution (the entirety of the diamond model of forensic attribution). Instead, SoURCE CODE will seek to automate and assist forensic experts on the Attacker-Capability edge (as shown in the green oval in Figure 1).

Figure 1. The Diamond Model of Forensic Attribution: showing the edge that the SoURCE CODE program is seeking to assist forensic experts.

SoURCE CODE will create novel methods and technologies that go beyond the common lexical and syntactic features that have been explored in prior literature; these features might include semantic and behavioral features that may become salient features for similarity and demographic traits. The Program will create scientifically validated forensic similarity and analytic technologies that allow for a measure of similarity of code and binaries as well as provide additional information or components of the code that may assist in analyzing hidden demographic information (groups, countries, or individuals where possible). SoURCE CODE systems shall provide evidence or explanations, beyond just a similarity score, to assist forensic experts in making a final attribution.

Figure 2 shows the expected tasks in this effort – including the expected outputs explored by performers.

Figure 2. The SoURCE CODE vision: Forensic systems capable of measuring similarity and providing additional information about the likely demographics of the author(s).

SoURCE CODE seeks to go beyond the limitations of existing academic literature in the space that utilizes limited subsets of hand-crafted features in largely the lexical and syntactic domains.

SoURCE CODE seeks to explore the semantic and behavioral features found within code to extract information and determine similarity. SoURCE CODE will also explore methods to provide evidence or explanations to the forensic experts to assist in their final determination.

Offerors to the SoURCE CODE program shall propose against the main goals of the similarity and demographic focus areas to ensure that their models will be able to meet the goals of the program.

Offerors shall propose methods to explore a fuller feature space (to include the possibility of learned features) to identify salient features that may be invariant across datasets. Using those feature sets, offerors will also propose similarity or demographic analytic algorithms for measuring source code to source code, binary to binary, and in later phases source code to binary and vice versa. Additionally, offerors shall propose methods in which significant evidence or explanations can be provided back to forensic experts to make a final determination.

A robust data collection initiative will be part of the SoURCE CODE Program to assemble diverse text types for use in Research and Development (R&D) for algorithm training, and in T&E for comparing the algorithm performance against ground truth. The T&E Team will conduct dataset development activities throughout the life of the program using Institutional Review Board (IRB)-approved Human Subject Research (HSR) protocols. A portion of this data will be made available to Performers for R&D. The Test and Evaluation cycle is described in more detail in 1.E Testing

& Evaluation. Additional details on program data can be found in 1.D. Program Data.

The SoURCE CODE Program will pursue rigorous and comprehensive independent T&E to ensure that research outcomes are well characterized, deliverables are aligned with program objectives, and performance is measured across the full range of data conditions. T&E activities will not only inform Government stakeholders on SoURCE CODE research progress but will also serve as valuable feedback to the Performers to improve their research approaches, algorithm training practices, and system development. The SoURCE CODE Program will work closely with

Government leaders in Reverse Engineering and Forensic attribution of software to continually refine and improve T&E methodologies.

1.A.1 Technical Challenges and Objectives

Offerors shall address three focus areas within the Technical Area 1 (TA1) to meet the goals of

SoURCE CODE to develop novel, human interpretable similarity and demographic algorithms.

Offerors must propose novel approaches to each of these focus areas, and if selected as a

Performer, Performers will be required to create an end-to-end technology with components from each focus area. These focus areas include:

Feature Space Generation and Extraction: The goal of this focus area is to research and develop an encoder that projects source code and binary representations of software into an optimized feature space wherein observations from the same classes are projected near one another, regardless of the representation of the software (binary or source code). Successful approaches must utilize features that are predictive of class similarity and/or demographic information

(country, group, individual). Deliverables will be evaluated by the T&E team using metrics outlined in 1.F Program Metrics.

Similarity and Demographic Analytic Algorithms: The goal of this focus area is to research and develop a system that will successfully identify similar binaries within a corpus using the stylistic characteristic of the software being analyzed. Systems must leverage representations from the feature space encoder to determine whether two or more software representations are similar or come from a specific author, group, or country. The software similarity and demographic analytic algorithms will be evaluated by the T&E team using metrics outlined in 1.F Program

Metrics.

System Explainability: Systems developed in TA1 must provide human-interpretable feedback providing evidence or explaining why specific documents likely are, or are not, similar or from a specific demographic group. Offerors shall propose novel evidentiary or explainability techniques to address this goal, and methods to evaluate this approach that are potentially generalizable to other performer approaches, including justifiable metrics. T&E Teams will finalize the protocol for evaluating explanations in Phase 1 and implement this protocol in evaluations during subsequent phases.

Figure 3 provides an example of the TA1 software components that will be expected under this program. T&E teams will design component interfaces to ensure that systems are modularized, allowing for addition or removal of components to meet government transition needs and additional evaluations.

Figure 3. Performer Outputs for TA1. Inputs will be source code and/or binary representations of software.

The performers will be expected to featurize and create similarity and/or demographic analytic systems with explainability components.

1.A.2 Program Phases

The SoURCE CODE program is anticipated to be a 2.5-year (30 month) effort, comprised of two

(2) phases. Both phases are being solicited under this BAA. Phase 1 will be 18 months in duration and phase 2 will be 12 months. The phases will encompass all of the TA1 focus areas described in previous sections (1.A.1 Technical Challenges and Objectives and 1.A.2 Program Phases).

Offerors are required to propose to both phases and all focus areas within TA1 of the Program under this BAA. Proposals that omit any of phases or focus areas will be considered non-compliant.

SoURCE CODE aims to create systems that can accomplish similarity and demographic analytics under multiple conditions encountered by Law Enforcement and U.S. Intelligence Community stakeholders. Program phases are designed to test Performer systems against increasingly challenging data scenarios. Differences in phases are shown in

Table 1.

Table 1. SoURCE CODE Program Phases and progression of challenges between both phases, data permitting.

Area Phase 1 (18 months) Phase 2 (12 months)

Attribution domains Source Code → Source Code

Binary → Binary

Source Code → Binary

Binary → Source Code

Number of groups and demographic attributes Fifty (50) Seventy or more (70+)

Type of Set Closed Set Open set

Each program phase will have multiple milestone deliverables at which performers will submit software deliverables to the T&E team for independent evaluation against datasets, to include hidden datasets. Deliverables will occur in 4 month increments and include featurization methods for testing, along with similarity and demographic analytic algorithms. Deliverables are outlined in Figure 4.

Figure 4. Mapping of TA milestones and activities to Phases. Also shows a schedule of key milestones and deliverables.

Evaluations will be performed on sequestered datasets using performer-submitted, containerized software components. Aspects of evaluation that involve human adjudication or feedback will be performed by the T&E Team, but performers may self-test using human evaluators.

In Phase 1, IARPA will provide development datasets and an evaluation harness developed by the

T&E team and provided at program kick-off, and any updates will be provided as they are released during the life of the program. The same harness will be used for testing and evaluation of

Performer-submitted software components at each subsequent evaluation milestone. The purpose of the evaluation harness is to enable performer-site training and testing for both the feature space, similarity, and demographic analytic algorithms. IARPA will also furnish documentation of the program Application Programming Interface (API) developed by the SoURCE CODE T&E Team.

This Government Furnished Information (GFI) will facilitate productive and useful research in a consistent and standardized manner.

Datasets developed by the T&E Team for each program phase will be partitioned into development and evaluation datasets. Although the T&E Team will deliver development data, Performers may seek to supplement their own development datasets aggregated from open source or self-generated and using creative techniques and heuristics to infer classes. These datasets must be submitted to the T&E Team to be combined into a single dataset, such that it is releasable to all Performers as discussed in 1.D.1.b Performer Supplemented Data Sources.

Evaluation of Performer progress towards program metrics will primarily utilize evaluation datasets that are sequestered. A portion of these datasets will be shared with Performers after each evaluation to enable Performers to conduct error analyses. The baseline API provided in Phase 1 will standardize system and subcomponent design and streamline comparative evaluations. More details on the datasets are available in 1.D. Program Data. and more details on the API are available in 1.G.4 Program API.

1.A.2.a Phase 1

The goal of Phase 1 is to develop new methods and explore the feature space between source code to source code and binary to binary representations of software. Featurization, similarity matching, and demographic analytic systems in this phase should succeed at performing rank and retrieval and open search tasks using inputs of both source code and binary representations. Systems will

Task # Mo. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 1 2 3 4 5 6 7 8 9 10 11 12

Kickoff -

TA1: Feature Space 30

TA1: Similarity 30

TA1: Explainability 30

PI Virtual Meetings / Calls

PI Workshop (in-person) -

Site Meetings -

Program Closeout -

Milestone Deliverables

Phase 1 Phase 2 only be expected to query like representations (source code to source code, binary to binary) matches against a large corpus of software.

Throughout the first phase of the program, Performer research will focus on foundational explorations and investigations into different approaches, theories, and concepts to establish the building blocks of their SoURCE CODE system(s). During this time, performers and the T&E team will collaborate on a protocol to evaluate human explainability. This protocol will be implemented towards the end of Phase 1 and carry into Phase 2 of the program. As a precursor to this collaboration, proposals in response to this BAA must articulate a novel approach to the explainability challenge, addressing how their SoURCE CODE systems would furnish explanations to users allowing them to understand and verify system behavior. Two categories of an end user are envisioned: a non-expert analyst using the system explanations to understand system rationale, flag spurious attributions, gauge attribution confidence, etc.; and an expert with sophisticated reverse engineering and forensic, knowledge conducting highly nuanced attribution analysis of attacks. Offerors must be explicit in their proposals about how their explanatory techniques could be evaluated. This information will ensure that our testing methodology for explainability will properly capture any techniques that you are exploring.

With the aims of mitigating risk and investigating competing approaches, performers are encouraged to pursue several novel approaches under the same subtopic. Performers will compare and contrast various approaches using development datasets.

1.A.2.b Phase 2

The goal of phase 2 is for Performers to extend the capabilities developed in phase 1.

Featurization, similarity matching, and demographic analytic systems in this phase should succeed at performing rank and retrieval and open search tasks using inputs of both source code and binary representations. Systems will be expected to query opposite representations (source code to binary, binary to source code) matches against a large corpus of software.

1.B Team Expertise

Collaborative efforts and teaming among Offerors are highly encouraged. It is anticipated that teams will be multidisciplinary and shall include expertise in one or more of the disciplines listed below. This list is included only to provide guidance for the Offerors; satisfying all the areas of technical expertise below is not a requirement for selection and unconventional or innovative team expertise may be needed based on the proposed research. Proposals shall include a description and the mix of skills and staffing that the Offeror determines will be necessary to carry out the proposed research and achieve program metrics.

• Binary Analysis

• Compiler Design

• Software Engineering

• Programming Languages

• Authorship Attribution

• Threat Intelligence Analytics

• Cybersecurity

• Cyber-Forensics & Reverse Engineering

• Stylometry

• Artificial Intelligence

• Machine learning

Although the SoURCE CODE program is focused on automated approaches, teaming schemes that include meaningful collaborations with forensic experts, software reverse engineers, and practitioners of non-computational attribution methods are highly encouraged.

1.C Program Scope and Limitations

Proposals shall explicitly address all of the following:

• Underlying theory: Proposed strategies to meet program-specified metrics must have firm theoretical bases that are described with enough detail that reviewers will be able to assess the viability of the approaches. Proposals shall properly describe and reference previous work upon which their approach is founded.

• Research & Development approach: Proposals shall describe the technical approach to meeting program metrics.

• Technical risks: Proposals shall identify technical risks and proposed mitigation strategies for each identified risk.

• Software development: Proposals shall describe the approach to software architecture, modularization, and integration.

The following areas of research are out of scope for the SoURCE CODE program:

• Research that does not have strong theoretical and experimental foundations

• Approaches that rely on features outside of source code or binary executables (e.g.

timestamps, network structure, multimedia, etc.).

• Approaches that rely solely on integration of existing cybersecurity solutions without experimental or scientific improvements.

• Approaches that focus solely on utilizing features from only the lexical and syntactic domains

• Approaches that rely heavily on dynamic analysis of the software – without identifying ways to statically represent features for safe analysis.

• Improving natural language authorship attribution (i.e., for code comments). Use of these systems is encouraged in addition to the methods being explored, and modifications to existing systems is allowable only if required to improve code comment specific functionality – but improvement of natural language attribution techniques is out of scope.

• Development of hardware systems

• Approaches that consist merely of integrating currently existing software systems.

• Adversarial approaches to misattribute or cause errors in attribution or demographic systems.

Performers are permitted to use data outside of the standard data provided by T&E teams, however no proprietary data should be used. Any additional data from proposers shall be distributed to

T&E for dissemination to any and all selected performers on SoURCE CODE.

Delivered software will be evaluated by the T&E team on sequestered evaluation datasets. During each performance period, performers will build software systems that will be run and evaluated by the T&E team. Testing protocols do not allow for expert operators – systems must be fully automated. Systems or algorithms that have been trained using human-in-the-loop methods may be submitted, provided that they can be run autonomously.

1.D. Program Data

The SoURCE CODE program seeks to simulate the challenge of forensic similarity measures from real-world circumstances faced by threat intelligence analysts. Current academic datasets have a number of shortcomings that render them inadequate to understand the full scope of the challenge program, including:

• Understanding the impact of group codebases vs. a single individual’s code

• Understanding the impact and influences of project domains (e.g. Android vs. iOS vs.

Windows vs. Linux)

• Understanding the impacts and influences of development tools:

o Integrated Development Environments (IDEs) o Version Control Systems o Compiler choices o Build Environment o Deployment tools o Novel concepts proposed by performers

• And the impact of things like company style guides or project naming conventions.

SoURCE CODE Test and Evaluation teams will be seeking to fill some of these gaps through curation of datasets, annotating datasets, and/or building datasets to fill these gaps. Ground truth in these datasets will represent a diverse set of demographic data.

The program will include two types of data – Development Data and Evaluation Data.

Development data will be utilized by Performers, as they see fit, to conduct research, development, error analysis, and algorithm training. Evaluation data will be explicitly excluded from any algorithm training approaches and will be used by the T&E Team for performance evaluations of subcomponents, modules, and systems measured against the SoURCE CODE program metrics.

Offerors may include datasets that they may own if such data is not proprietary and can be shared among all selected performers.

1.D.1 Development Data

Development data will be utilized by Performers, as they see fit, to conduct internal research, development, error analysis, and algorithm training. Each Performer is anticipated to have a unique technical solution, and accordingly may require different development data depending on their approach.

Datasets will be collected, annotated, and curated by the T&E Team. Some of this data may be sequestered, and a portion will be provided to Performers for R&D and error analysis.

1.D.1.a Government Research and Evaluation Data

The T&E team will curate, annotate, and collect datasets to be made available to SoURCE CODE performers. Some data will be sequestered for the purpose of blind testing and evaluation. The government team will also work to augment these datasets with a collection effort to create human-authored ground-truthed source code and binary executable software and will provide some of this data upon anonymization and availability of the data in accordance with any appropriate Human

Subject Research authorities.

1.D.1.b Performer Supplemented Data Sources

Performers may propose to utilize external datasets obtained by Performers that are available from third parties or that have been collected by the Performer outside of the SoURCE CODE program.

An example of a third-party dataset would be an authorship attribution dataset, collected by a university under an IRB-approved HSR protocol, that has been approved by the dataset owner for release to the research community. Data collected by a Performer under a different program are considered external data, even if the other program’s data collection was Government-sponsored.

All external datasets must be approved for use in the SoURCE CODE program by IARPA during contract performance, in accordance with HSR and applicable privacy policies, statutes, and regulations.

Performers may not use proprietary datasets unless these datasets are made available to all R&D performers and the T&E Team in the SoURCE CODE program. Public release of proprietary datasets is not a requirement; however, release for use within the SoURCE CODE program is required. Moreover, for any dataset not collected under the scope of the SoURCE CODE program, performers must provide the Government with an accounting of all resources used and sources from which data are drawn and describe how the data will be used for development, testing, and training of algorithms.

1.E Testing & Evaluation

T&E will be conducted by an independent team of Government and contractor staff carrying out evaluation and analyses of performer research deliverables using program test datasets and protocols. The SoURCE CODE data and test protocols (see 1.D.1.a Government Research and

Evaluation Data) will be the primary mechanism by which the T&E team carries out their evaluations.

The SoURCE CODE program will pursue rigorous and comprehensive T&E to ensure that research outcomes are well characterized, deliverables are aligned with program objectives, and that the algorithm performance is measured across a full range of conditions. Such T&E activities will inform IARPA and Government stakeholders on SoURCE CODE research progress and serve as invaluable feedback to performers to improve their research approaches, algorithmic training, and system development. The SoURCE CODE program will work closely with government leaders in malware analysis and reverse engineering to refine and improve the T&E methodologies.

The government will provide performers with a uniform Evaluation Harness (EH) application programming interface (API) – to be provided at program kick-off - with relevant scripts to run program test protocols on program datasets. The design of the EH will be informed by prior work conducted by the T&E team and other government agencies. This evaluation environment will be the platform for independent government testing of performer deliverables.

Figure 5. Test and Evaluation. T&E team will provide and interface (e.g. Docker Containers / APIs) that will be used to automate the testing. Offerors will submit their feature space and similarity/demographic algorithms for independent testing and be tested for feature invariance or compared against baseline systems.

Performers will have specific deliverable milestones at which all subcomponent, system algorithms, and software will be delivered to IARPA and its designated T&E Team. The T&E

Team will then conduct evaluations at the direction of the SoURCE CODE Program Manager

(PM) and with the objective of characterizing the quality, functionality, and performance of the

SoURCE CODE deliverables. In addition to quantitative measurements, T&E will be carried out to establish a thorough understanding of the progress, status, and limitations of the performer’s research.

T&E results and feedback will be provided to performers at regular intervals to keep them abreast of current independent performance measurements and to inform and improve their R&D approaches and methods. T&E results from all performers will be shared with all teams to establish an understanding of the current state and progress of SoURCE CODE research; T&E results will also be shared with government stakeholders, including their contractors, for Government purposes. For example, a Principal Investigator (PI) Review Meeting will be held annually to share research ideas, progress, and results across the SoURCE CODE program (see 1.G.2 Program

Milestones and Deliverable timelines).

IARPA may conduct other supplemental evaluations or measurements in its sole discretion to evaluate the Performers’ research and Deliverables.

1.F Program Metrics

Achievement of metrics is a performance indicator under IARPA research programs. IARPA has defined SoURCE CODE program metrics to evaluate effectiveness of the proposed solutions in achieving the stated program goal and objectives, and to determine whether satisfactory progress is being made. The metrics described in this BAA are shared with the intent to scope the effort, while affording maximum flexibility, creativity, and innovation to Offerors proposing solutions to the stated problem.

The final SoURCE CODE T&E protocols and evaluation methodology will be provided at program kickoff. Program metrics may be refined during the various phases of the SoURCE CODE program; if metrics change, revised metrics will be communicated in a timely manner to

Performers. The evaluation methodology may be revised by the Government at any time during the program lifecycle to better meet program needs.

SoURCE CODE program target metrics were selected based on three factors:

1. What is technically achievable but challenging based on current state-of-the-art in software similarity measurements;

2. What is statistically measurable based on the planned program evaluation data; and

3. What is useful to mission partners based on USG stakeholder needs and use cases.

The following is a description of primary program metrics and target scores. Score targets will increase for each successive program phase. SoURCE CODE performance metrics are designed to evaluate the software component deliverables for similarity and demographic tasks.

Performer software submissions for similarity and demographic estimation tasks are evaluated using standard machine learning and biometric measures for classification tasks. The recognition task tests whether a system can use a query document from a given author, group, country (or other demographic data) and retrieve documents from a set of documents produced by the same author or documents most likely from the same demographic class. At test time, the Performer system must generate feature encodings for a set of author query documents as well as documents from a retrieval database. Using Performer system generated encodings, the T&E Team will retrieve a ranked list of k-nearest neighbors for each author query. The program does not place constraints on the properties of the feature space; however, teams must stipulate an encoding that allows calculation of similarity.

Specific metrics of interest include: Top-N accuracy (specifically Top-1 and Top-10) which are used to measure the success of the system in bubbling up documents from the demographic class being tests. (Accuracy is chosen over Recall as proper classification of negative samples is also important). Additionally, Equal Error Rate (EER) is to be calculated as a measure of the Receiver

Operating Curve (ROC) – a measure of the point where False accept and False Reject errors are equal. In a similar vein, measurements of the d-prime (d′) measure is necessary to determine the discriminability between the “genuine” and “imposter” signals to assist in determining if feature representations are viable for these problems. Additional metrics may be calculated, to understand the challenge of the task and how well performer systems are performing on the tasks.

A summary of metric targets by phase is shown in Table 2 for similarity metrics and Table 3 for estimating demographic trait metrics; these metrics are subject to change over the course of the program. Accuracy targets are defined as aggregated statistical measures for a given phases’ evaluation datasets, which will contain binaries and source code from a broad range of software types, build systems / compilers, and authors with a diverse background in software development.

These targets are estimates informed by current capabilities and use case requirements. To mitigate risk and better inform the Government regarding the appropriateness of phase-over-phase performance targets, the T&E Team will produce baseline systems for each task approximating current state-of-the-art. Further information about baseline models will be furnished to Performers at program kickoff.

Table 2. Metrics for similarity tasks in both phases of the program. Metrics shown are for

1,000 unique users and should scale to match cardinality of unique users. Additional numbers shown in brackets are for 100 and 10,000 authors respectively.

Phase I Phase II

Top-1 Accuracy [100/10,000] 85% [95/75] 90% [98/90]

Top-10 Accuracy [100/10,000] 95% [99/85] 96% [99/95]

EER Average 30% 20% d′ (sensitivity index) Measured increase 30%

Processing Time

(max per binary/source) 5 minutes 2 minutes

Attribution methods

(data permitting)

Source → Source, Binary →Binary

Source → Binary, Binary →Source

Table 3. Metrics for demographic tasks in both phases of the program. Metrics shown are for 1,000 unique users and should scale to match cardinality of unique users. Additional numbers shown in brackets are for 100 and 10,000 authors respectively.

Phase I Phase II

Top-1 Accuracy [100/10,000] 85% [95/75] 90% [98/90]

Top-10 Accuracy [100/10,000] 95% [99/85] 96% [99/95]

EER Average 20% 10%

Processing Time

(max per binary/source) 5 minutes 2 minutes

# Groups/Demographics

(data permitting)

50 70+

Type of Set Closed Open

1.G.1 Waypoints

Waypoints are the means by which the Performer clearly demonstrates the quantitative and timely progress that must be made for the overall concept to meet end-of-phase Milestones. In other words, the intent of Waypoints is to provide a clear measure of progress towards meeting the program Milestones so the PM and advisors can provide more effective guidance and assistance to the Performers. Performance against these Waypoints will be reviewed throughout the program, and the PM and advisors will use performance against the Waypoints to assess whether course corrections are needed to ensure program success.

The Government has defined Waypoints for all Performers that are included in the Program

Milestones and Deliverables (see 1.G.2 Program Milestones and Deliverable timelines). Progress against these Waypoints will be reviewed during site visits.

1.G.2 Program Milestones and Deliverable timelines

Table 4 shows the expected recurring events and deliverables that will occur across both phases of the program by all performers. Table 5 shows the expected events that will occur for all performers across both phases.

Table 6 shows the expected deliverables for all performers across both phases of the program.

Offerors may add other waypoints or deliverables in addition to the minimum set listed in the tables below.

Table 4. Recurring Events and deliverables expected by all performers on SoURCE CODE across both phases.

Recurring Event /

Deliverable

Frequency Description

PI Virtual Meetings / Calls Monthly

Monthly teleconference with SoURCE CODE

PM, SETA, and T&E teams.

Monthly Status Reports

Monthly

Due on the 15th of each month; Technical and costs

DELIVERABLE: MSR documentation

Table 5. Events that are expected to occur during the life of the program. Locations are

TBD; (please expect many events to occur in Washington, D.C.)

Events Month Description

Start Phase 1

Events Month Description

Program Phase 1 Kickoff 0

Location: TBD; Attendance, presentation as requested by the PM, addressing performer questions as needed.

Presentation shall be shared with the PM 5 calendar days prior to the kickoff meeting. Information shared about system harness, APIs furnished to Performers; T&E present guidelines for developing protocol for explainability and utility metrics.

Site Visit 4, 12 At performer sites; Discussions about ongoing work with

PM, SETA, and T&E team.

PI Workshop 8, 16

Location: TBD; Attendance in person or virtual;

presentations about ongoing work to all performers and government team and possible government partners.

Star Phase 2

Program Phase 2 Kickoff 19

Location: TBD; Attendance, presentation as requested by the PM, addressing performer questions as needed.

Presentation should be shared with the PM 5 calendar days prior to the kickoff meeting.

Site Visit 24 At performer sites; Discussions about ongoing work with

PM, SETA, and T&E team.

Program Closeout 30

Location: TBD; Attendance in person or virtual;

presentations about work completed, lessons learned, and any other materials to close out the program. Audience will include all performers, government team, and government partners.

Table 6. Deliverables expected by performers across the life of the program.

Task/Deliverable Month Description

Start Phase 1

Performer Deliverables

(TA1: Feature Space)

System submission of Feature Space software deliverables to T&E team; Software shall be delivered at the beginning of evaluation. Updated software may be requested if bugs are identified during evaluation.

DELIVERABLE: Software component for Feature

Space– including source code.

Performer Deliverables

(TA1: Feature Space &

Similarity)

8,12,16

System submission of all subcomponents of software system to T&E team; Software shall be delivered at the beginning of evaluation. Updated software may be requested if bugs are identified during evaluation.

DELIVERABLE: Software for all components of system– including source code.

Performer Phase 1 Final

Reports

Phase 1 Final Reports highlighting work completed, metrics achieved, challenges, opportunities for additional experimentation, and all documentation on software system.

DELIVERABLE: Report

Task/Deliverable Month Description

Start Phase 2

Performer Deliverables

(TA1: Feature Space &

Similarity)

22,26,30

System submission of all subcomponents of software system to T&E team; Software shall be delivered at the beginning of evaluation. Updated software may be requested if bugs are identified during evaluation.

DELIVERABLE: Software for all components of system– including source code.

Performer Phase 2 Final

Reports

Phase 2 Final Reports highlighting work completed, metrics achieved, challenges, opportunities for additional experimentation, and all documentation on software system.

DELIVERABLE: Report

1.G.3 Software Deliverable Formatting

Performers will be required to provide algorithm, software Deliverables, design documentation, and operating instruction to execute the software in a manner that conforms to a standardized industrial method or methods that will be provided at program Kickoff. To facilitate planning, Offerors may assume that the standardized configuration will require the use of software containerization technology (e.g., Docker and a REST API). This means that the entirety of a

Performer’s system, including pre- and post-processing, must be included within the delivered software container. For models that require training, the expectation is for the initial model training to occur on Performer systems, with the ability for the T&E Team to re-train and test the model with the same and/or other data. Performers shall also provide source code to the government and

T&E team to enable modification of Performer containers to suit deployment constraints. Offeror teams that do not include the requisite expertise to conduct such software development shall include costs in their proposal to obtain software development support.

Each team is required to include among their key personnel a Lead System Integrator (LSI) who shall be responsible for preparing software Deliverable subcomponents, modules, and systems, performing quality control of Deliverable, and integrating key components into the primary

SoURCE CODE system(s). The LSI will also oversee communication and coordination across a

Performer’s research teams including subcontractors, if applicable, to ensure research products are functional and following software coding best practices (e.g., inline comments, documentation).

Additional team members and roles are dependent on the proposed research, as such, there is no predetermined or required skill mix.

1.G.4 Program API

The SoURCE CODE Program will utilize a standardized Application Programming Interface

(API) for all software Deliverables and evaluations. The first version of the SoURCE CODE API will be provided to Performers at the Phase 1 Kick-off Meeting and updated periodically thereafter.

The API will define function calls, data structures, and display creation and management for operating and evaluating SoURCE CODE software in a standardized manner. All Performer solutions must be compatible with the SoURCE CODE API.

1.H. Meeting and Travel Requirements

Offerors are expected to assume responsibility for administration of their projects and to comply with contractual and program requirements for reporting, attendance at program workshops, and availability for site visits. The following paragraphs describe typical expectations for meetings and travel for IARPA programs as well as the contemplated frequency and locations of such meetings.

In addition to ensuring that all necessary details of developed software, algorithm, and operational instructions are clear and complete, each Performer will be required to be available for questions and troubleshooting from the T&E Team in Performer status meetings.

1.H.1 Technical Exchange Meetings and Workshops

All Performer teams are expected to attend Technical Exchange meetings and workshops, including key personnel from prime and subcontractor organizations.

The SoURCE CODE program intends to hold a program Kick-off Meeting workshop in the first month of the program and the first month of each subsequent program phase. In addition, the program will hold a PI Review Meeting starting in Phase 1 and then similar workshops at least annually thereafter. Kick-off Meetings and PI Review Meetings may be combined for logistical convenience. The dates and locations of these meetings are to be specified at a later date by the

Government, but for planning purposes, Offerors should use the approximate times and locations listed in Table 5. Both types of meetings are likely be held in the Washington, D.C. metropolitan area, but IARPA may opt to co-locate the meeting with a relevant external conference or workshop to increase synergy with stakeholders. IARPA reserves the right to hold the meeting virtually for logistical or health and safety reasons.

Kick-off Meetings will typically be one day in duration and will focus on plans for the coming

Phase, Performer planned research, and internal program discussions. PI Review Meetings will typically be two days in duration and will have a greater focus on communicating program progress and plans to USG stakeholders. These meetings will include additional time allocated to presentation and discussion of research accomplishments as well as interactive system demonstrations for Government stakeholders.

In both cases, the workshops will focus on technical aspects of the program and on facilitating open technical exchanges, interaction, and sharing among the various program participants.

Program participants will be expected to present the technical status and progress of their projects to other participants and invited guests. Individual sessions for each performer team with the

SoURCE CODE PM and T&E Team may be scheduled to coincide with these workshops. Non-proprietary information will be shared by performers in the open meeting sessions; proprietary information sharing shall occur during individual breakout sessions with the SoURCE CODE PM and the T&E Team.

1.H.2. Site Visits

Site visits by the Government Team will generally take place semiannually during the life of the program. These visits will occur at the Performer’s facility. Reports on technical progress, details of successes and issues, contributions to the program goals, and technology demonstrations will be expected at such site visits. Performers shall participate and provide final meeting documents, to include captured action items, within 15 calendar days following the meeting. Draft materials, for any presentations, are due 5 workdays prior to the meeting. IARPA reserves the right to conduct additional site visits on an as-needed basis.

1.I. Period of Performance

The SoURCE CODE program is envisioned as a 30-month effort intended to kickoff in July 2024.

Phase 1 (Base Period) of the program will last 18 months; Phase 2 (Option Period 1) will last 12 months. Offerors are required to submit a proposal that addresses both phases of the program.

1.J. Place of Performance

Work for the SoURCE CODE program will be conducted at the Performers’ sites.

1.K. REFERENCES

[1] S. Caltagirone, A. Pendergast, and C. Betz, “The Diamond Model of Intrusion Analysis,”

Center for Threat Intelligence and Threat Research, Hanover, MD, Jul. 2013 [Online]. Available:

https://www.threatintel.academy/wp-content/uploads/2020/07/diamond-model.pdf. [Accessed:

Sep. 26, 2022]