BAA_20-03_SCISRS_Amendment 01.pdf

Securing Compartmented Information with Smart Radio Systems (SCISRS)

Program

IARPA-BAA-20-03

Release Date: September 28, 2020

Amendment 01 Date: October 23, 2020

BROAD AGENCY ANNOUCEMENT: IARPA-BAA-20-03

SCISRS PROGRAM

Table of Contents

GENERAL INFORMATION

SECTION 1: FUNDING OPPORTUNITY DESCRIPTION

1. A. Program Overview

1. A.1. Technical Challenges

1. A.2. Program Phases

1. B. Team Expertise

1. C. Program Scope and Limitations

1. D. Development Data

1. E. Test and Evaluation (T&E)

1. E.1. RF Testbeds

1. E.2. RF Characterization Hardware

1. E.3. RF Baseline Characterization

1. E.4. Introduction of Anomalous RF Signals

1. F. Program Metrics

1. G. Program Waypoints, Milestones, and Deliverables

1. G.1. Required Program Waypoints

1. G.2. Program Milestone Timeline and Deliverables

1. G. 3. Software Deliverable Formatting

1. H. Meeting and Travel Requirements

1. H.1. Workshops

1. H.2. Site Visits

1. H.3. Visits to Government Testbeds

1. H.4. Technical Status Meetings

1. I. Place of Performance

1. J. Period of Performance

SECTION 2: AWARD INFORMATION

SECTION 3: ELIGIBILITY INFORMATION

3. A. Eligible Applicants

3. A.1. Organizational Conflicts of Interest (OCI)

3. A.2. Multiple Submissions to the BAA

3. B. U.S. Academic Institutions

3. C. Other Eligibility Criteria

3. C.1. Collaboration Efforts

SECTION 4: PROPOSAL AND SUBMISSION INFORMATION

4. A. Proposal Information

4. B. Proposal Format and Content

4. B.1. Volume 1: Technical and Management Proposal

4. B.1.a. Section 1: Cover Sheet & Transmittal Letter

4. B.1.b. Section 2: Summary of Proposal

4. B.1.c. Section 3: Detailed Proposal Information

4.B.1.d. Section 4: Attachments

4. B.2. Volume 2: Cost Proposal (No Page Limit)

4. B.2.a. Section 1: Cover Sheet

4. B.2.b. Section 2: Estimated Cost Breakdown

4. B.2.c. Section 3: Supporting Information

4. C. Submission Details

4. C.1. Due Dates

4. C. 2. Proposal Delivery

4. D. Funding Restrictions

SECTION 5: PROPOSAL REVIEW INFORMATION

5.A. Technical and Funding Availability Evaluation Factors

5.A.1. Technical Evaluation Factor (technical criteria listed below)

5. A.1.a. Overall Scientific and Technical Merit

5. A.1.b. Effectiveness of Proposed Work Plan

5. A.1.c. Contribution and Relevance to the IARPA Mission and Program Goal

5. A.1.d. Relevant Experience and Expertise

5. A.1.e. Resource Realism

5. A.2. Funding Availability Factor

5. A.2.a. Budget Constraints

5. A.2.b. Program Balance

5. B. Method of Evaluation and Selection Process

5. C. Negotiation and Contract Award

5. D. Proposal Retention

SECTION 6: AWARD ADMINISTRATION INFORMATION

6. A. Award Notices

6. B. Administrative and National Policy Requirements

6. B.1. Proprietary Data

6. B.2. Intellectual Property

6. B.3. Human Use

6. B.4. Animal Use

6 .B.5. Publication Approval

6. B.6. Export Control

6. B.7. Subcontracting

6. B.8. Reporting

6. B.9. System for Award Management (SAM)

6. B.10. Representations and Certifications

6. B.11. Lawful Use and Privacy Protection Measures

6. B.12. Public Access to Results

6. B.13. Cloud Compatibility

Appendix A: Templates for Volume 1: Technical Proposal

A.1 Cover Sheet for Volume 1: Technical Proposal

Appendix A.2

Academic Institution Acknowledgment Letter

Appendix A.3

Intellectual Property Rights

Appendix A.4

Organizational Conflicts of Interest Certification Letter

Appendix A.5

Three Chart Summary of the Proposal

Appendix A.6

Research Data Management Plan (RDMP) BAA 20-03

Appendix B: Templates for Volume 2: Cost Proposal

Appendix B.1 Cover Sheet for Volume 2: Cost Proposal

Appendix B.2 Prime Contractor/Subcontractor Cost Element Sheet for Volume 2: Cost Proposal

Appendix B.3 - Software and IP Costs

Appendix B.4 – Travel Costs Trip breakdown

Appendix B.5 – Contract Deliverables Table

GENERAL INFORMATION

This notice constitutes a Broad Agency Announcement (BAA) and sets forth research of interest in the area of automated detection and characterization of radio frequency (RF) anomalies in complex RF environments. The solicitation process will follow Federal Acquisition Regulation (FAR) Part 35, Research and Development Contracting, as supplemented with additional information included in this notice. Awards based on responses to this BAA will be considered the result of full and open competition.

1. Federal Agency Name – Office of the Director of National Intelligence/Intelligence Advanced

Research Projects Activity (IARPA)

2. Funding Opportunity Title – Securing Compartmented Information with Smart Radio Systems

(SCISRS)

3. Announcement Type – Initial

4. Funding Opportunity Number – IARPA-BAA-20-03

5. Catalog of Federal Domestic Assistance Numbers (CFDA) – Not applicable Questions

Submit questions on administrative, technical, or contractual issues by email to dni-IARPA-BAA-20- 03@iarpa.gov. All requests must include the full name and affiliation of a point of contact. Do not send questions with proprietary content. A consolidated Question and Answer response will be posted on the https://beta.sam.gov/ for Contract Opportunities website (https://beta.sam.gov/) and linked from the IARPA website (https://www.iarpa.gov/index.php/research-programs/SCISRS). No answer will go directly to the submitter. IARPA will accept questions until October 9, 2020 @ 5:00 PM EST.

Dates

5.1. Posting Date: September 28, 2020

5.2. Proposal Due Date for Initial Round of Selections: November 13, 2020, 4:00 PM EST

5.3. BAA Closing Date: December 14, 2020, 4:00 PM EST

6. Anticipated individual awards – Multiple awards anticipated

7. Types of instruments that may be awarded – Procurement Contracts Agency Points of contact

ATTN: IARPA-BAA-20-03

Office of the Director of National Intelligence Intelligence Advanced Research Projects Activity Washington, DC 20511 Electronic mail: dni-IARPA-BAA-20-03@iarpa.gov

8. Program Manager (PM) – Paul Kolb, PhD

9. Program Website – http://www.iarpa.gov/index.php/research-programs/SCISRS

BAA Summary -The goal of the SCISRS program is to develop smart radio techniques to automatically detect and characterize RF anomalies in complex RF environments. The specific types of anomalies include low probability of intercept (LPI) signals, altered or mimicked signals, and abnormal unintended emissions.

SECTION 1: FUNDING OPPORTUNITY DESCRIPTION

IARPA often selects its research efforts through the BAA process. The use of a BAA solicitation allows a wide range of innovative ideas and concepts. The BAA will appear on https://beta.sam.gov/, the official U.S. Government website for federal awards, and a link to the BAA will appear on the IARPA website at http://www.iarpa.gov/. The following information is for those wishing to respond to this Program BAA.

This BAA (IARPA-BAA-20-03) is for the SCISRS program. IARPA is seeking innovative techniques to mailto:dni-IARPA-BAA-20-03@iarpa.gov mailto:dni-IARPA-BAA-20-03@iarpa.gov https://beta.sam.gov/ https://beta.sam.gov/ https://www.iarpa.gov/index.php/research-programs/SCISRS http://www.iarpa.gov/index.php/research-programs/SCISRS https://beta.sam.gov/ http://www.iarpa.gov/ automatically detect and characterize RF anomalies in complex RF environments. SCISRS is envisioned to be a 42-month effort, beginning approximately February 2021.

The SCISRS program has established working definitions of terms to provide clarity and consistency to the BAA. The reader is encouraged to refer to Table 1 below for these definitions. Hereafter, these terms (when used as nouns and sometimes adjectives) are capitalized throughout the text of this BAA unless the term is used in a different or more generic sense.

Table 1: SCISRS Definitions (nouns capitalized throughout the BAA):

BAA Term SCISRS Definitions

Anomaly An Anomaly (or “RF Anomaly”) is a Signal that differs from the RF Baseline or that is intended to hide in the ambient RF Baseline.

Background

The Background (or “RF Background”) is the superposition of all Emissions from (non-negligible) natural and manmade RF sources (and their interactions with the local physical environment) particular to a specific geographic location and time. To the extent that RF Emissions are directional, the RF Background is subject to the direction of RF Emission propagation at the location where the Background is measured. The RF Background is independent of the RF sources that will be installed inside each testbed. (Suggested Reference: International Telecommunications Union (ITU) Recommendation ITU-R P.372-13.)

Baseline

The Baseline (or “RF Baseline”) is the superposition of the RF Background, the Overt Baseline, and the Emanation Baseline in a SCISRS testbed. The constituents of the Baseline are defined as:

• Emanation Baseline: The Emanation Baseline refers to the contribution to the RF Baseline of a testbed from all unintended RF Emissions.

• Overt Baseline: The Overt Baseline refers to the contribution to the RF Baseline from all Overt Communication Signals originating from within each testbed.

• RF Baseline: The RF Baseline of a testbed is the superposition of the RF Background (including geographically specific components and components that may change over time) plus RF Emissions from all sources located within the testbed itself. The RF Baseline is composed of the RF Background, the Overt Baseline, and the Emanation Baseline.

characterize Describe the distinctive nature or features of, distinguish (one Signal from another).

Emanation An Emanation is an unintended RF Emission from microprocessors and/or other electronics. Emanations often carry information unintentionally. Emanations are Signals.

Emission An Emission (or “RF Emission”) is the electromagnetic wave radiation produced by a RF source in the frequency range from 20 kHz to 300 GHz that interacts with physical structures in the environment via reflection, refraction, and diffraction.

Overt

Apparent, not secret or hidden. An Overt Emission is one whose presence is readily detectable and whose purpose is reasonably discernable. Examples of Overt Signals include LTE, WCDMA, and OFDM waveforms emitted by cell phones and wireless routers. An example of an Overt Emission that is not a Signal is a broad band RF jamming Emission whose presence is obvious and whose purpose could be reasonably surmised.

Signal

A Signal is any RF Emission that transports information from a specific source; a Signal is not necessarily continuous in time or frequency. Types of Signals include:

• Altered Signal: An Altered Signal is a known Overt Signal (e.g., LTE, WCDMA, OFDM) that has been further modulated or otherwise altered to carry additional information; the resulting alteration is not Overt.

BAA Term SCISRS Definitions

• Communication Signal: A Communication Signal is a Signal that intentionally carries information.

• Emanation: A Signal whose transport of information is unintentional.

• Low Probability of Intercept (LPI) Signal: LPI Signals are low probability of intercept Communication Signals. Examples include DSSS, FHSS, burst, and snugglers (weak Signals hiding near strong ones); they are not Overt.

• Mimicked Signal (or “Mimic”): A Mimicked Signal is a Signal that resembles known Overt Signals (e.g., LTE, WCDMA, OFDM) in frequency, bandwidth, and pulse shape but are unrecognizable or misleading to the protocols established to receive them. Mimics are not Overt.

Transmission A Transmission is an individual instance of an uninterrupted Signal. Transmissions can vary in power and duration.

As additional clarification to the defined terms used in the SCISRS BAA, two Venn diagrams are shown in Figures 1 and 2. Figure 1 illustrates notional RF Emission types. Figure 2 illustrates the notional contribution of RF Emissions to the RF Baseline of a testbed by the Emission’s respective type and location.

These Venn diagrams do not include an exhaustive list of Emission types.

Figure 1: Notional RF Emission Types by Source according to SCISRS definitions.

Figure 2. Notional Contributors to the RF Baseline by Emission source type and location (shaded gray).

Anomalies that differ from the RF Baseline are shaded in red.

1. A. Program Overview

The Intelligence Community (IC) and the Department of Defense (DoD) missions require that information and data be generated, stored, used, transmitted, and received in secure facilities and “in the wild.”

Vigilance regarding the security of data is important regardless of where the data are being utilized.

Significant investments in infrastructure by the U.S. Government and the private sector have provided a high level of confidence in the security of data in facilities under the control of the data owner. In environments such as those illustrated in Figure 3, where there is potentially much less control, data security becomes more challenging.

Figure 3: Examples of environments where RF anomalies may indicate compromise of secure data.

One possible indicator of attempted data breach is unexpected RF Transmissions. The goal of the SCISRS program is to develop smart radio techniques to automatically detect and characterize these suspicious Signals and other RF Anomalies in complex RF environments. The specific types of Anomalies include LPI Signals, Altered or Mimicked Signals, and abnormal, unintended Emanations. Offerors should bear in mind that the techniques need to be scalable, computationally efficient, and adaptable to a range of radio hardware.

Among the reasons the SCISRS goal is challenging is that the relevant frequencies may range over several orders of magnitude and the data collection rates may approach terabytes per second (Tb/s), about 1 million times higher than the data rate for high definition video. Leveraging technological advances in software defined radios (SDRs) and computational methods may make the problem tractable. For example, the breadth of RF signal processing techniques that used to require a closet full of radio equipment (e.g., analog-to-digital converters (ADCs), receivers, mixers, filters, amplifiers, demodulators, detectors) can now be done with a laptop and a tablet-sized SDR at a reasonable cost. In particular, wide bandwidth ADCs efficiently convert a wide swath of the spectrum into a large effective number of bits (ENoB) with precise linearity and low noise, enabling clean digital surveys of a RF environment. Machine learning (ML) image recognition algorithms have become so efficient they can match video rates even with modest data processing. Moreover, the increasing ability of high-performance computing to accommodate the demands of ML techniques and digital signal processing may offer the potential to match 5G RF data rates.

Highly customized solutions based on specific hardware are not of interest. It is an objective of the SCISRS program that methods be adaptable to a wide variety of RF collection hardware and that they be capable of detecting and characterizing various kinds of Signals in environments cluttered with sources of noise and interference. Under the SCISRS program, Performers will be required to develop their detection and characterization systems to work with a different architecture to be specified in each phase of the SCISRS program. The basic framework of that architecture is described in SECTION 1. E.2. RF Characterization Hardware; IARPA requires that the solutions Performers will develop must work well with different hardware integrated into that framework. In order to test Performer methods, IARPA will establish two testbeds of cataloged RF emitters to which Anomalous Signals will be later introduced. Additionally, the SCISRS Test & Evaluation (T&E) teams will install standardized collection hardware and provide an application programming interface (API) to control the hardware and to format raw data for analysis.

Performers will be expected to demonstrate methods for command and control of the collection hardware and for automatic detection and characterization of all the ambient Signals that comprise the pre-existing RF Baseline of each testbed. Performers will be also be expected to demonstrate these methods to characterize and detect anomalous signals of various types added to the ambient RF Baseline. Additional details about the testing and the performance metrics can be found in SECTION 1. E. Test & Evaluation (T&E) and SECTION 1. F Metrics.

1. A.1. Technical Challenges

Offerors should expect that the RF Baselines of each testbed will be replete with Overt Signals commonly found in today’s wireless-enabled world (e.g., Emissions from internet-of-things (IoT) devices, other 5G waveforms, other Communication Signals, and unintended RF Emissions from microprocessors and other electronics). The RF Baselines may vary by band. The frequency bands of interest will range from 100 MHz to 6 GHz in the initial phase of the program, but it is anticipated that this frequency range will increase in subsequent phases. In addition to RF sources located inside each testbed, there will be unavoidable RF radiation propagating into the testbeds from the surrounding environment. The testbeds will be established in different geographical locations and, as a result, will likely present very different RF Backgrounds. The RF Background is independent of the RF sources that will be installed inside each testbed but inevitably contributes to the RF Baseline of each testbed. The kind of RF Background to be expected for each testbed will be provided in advance shortly after program kickoff and will conform to definitions found within the

International Telecommunications Union (ITU) Recommendation ITU-R P.372-13. The RF Background is expected to change over time as new sources of RF are introduced external to a testbed or as “old” sources turn off, move, or otherwise change. An example of this would be variations resulting from the Signals from a passing airplane; these Signals will likely propagate into a nearby testbed and represent a temporal and spatial fluctuation to the testbed RF Baseline. Offerors are referred to the Venn diagram in Figure 2 where each RF Emission source shaded gray (including natural sources) contributes to the RF Baseline.

The SCISRS T&E teams will intentionally make changes in each testbed, such as when the Anomalous Signals to be detected and characterized by Performers are introduced. According to the SCISRS definition, these Anomalies differ from the RF Baseline; they are excluded from the RF Baseline in the Venn diagram in Figure 1 and are shaded red.

Proposals must fully describe the Offeror’s technical approaches to addressing each of the following Technical Challenges (TCs).

1. TC-1 – Detection and Characterization of Overt Signals

The SCISRS objective for TC-1 is automated methods to efficiently detect and characterize the Overt Signals present in each IARPA established testbed.

2. TC-2 – Detection and Characterization of Anomalous Signals and Transmissions

The SCISRS objective for TC-2 is automated methods to efficiently detect and characterize RF Anomalies; i.e., RF Signals that differ from the RF Baseline or that are intended to hide in the RF Baseline.

3. TC-3 – Detection and Characterization of Unintended RF Emissions

The SCISRS objective for TC-3 is automated methods to efficiently detect and characterize the RF Emissions, sometimes referred to as Emanations, that are unintentionally emitted from microprocessors or other electronics present in each IARPA established testbed. In contrast to Communication Signals, which intentionally carry information, these Signals frequently transmit information unintentionally, i.e., not by design. A significant parametric change from the Emanation Baseline would be considered an Anomaly, specifically an Emanation Anomaly.

1. A.2. Program Phases

The SCISRS program will proceed in three phases. The phases are designed to drive Performers to develop solutions that are scalable, computationally efficient, and hardware-agnostic. Performers will be expected to address all three technical challenges in each phase, but the focus of each phase will differ. The expected performance, level of difficulty, and variety of Anomalies encountered will increase in each phase; more detail can be found in SECTION 1. F. Program Metrics. The following paragraphs introduce the phases.

Phase I – Focus on RF Baseline Characterization and Detection and Characterization of LPI Anomalies: Phase I will last 18 months and will focus on detection and characterization of Anomalous Signals that can be broadly characterized as LPI Signals (TC-2 for LPI). In order to identify Anomalies, Performers must first characterize the Overt Signals of the RF Baseline (TC-1) with the IARPA-provided hardware in each SCISRS testbed. Because these Signals are readily detectable of known modulation types, it is expected that TC-1 performance in Phase I will be very high. TC-2 for LPI will be more challenging.

For the purposes of the SCISRS program, the LPI signals to address include, but are not limited to, direct sequence spread spectrum (DSSS), frequency hopping spread spectrum (FHSS), snugglers, and burst.

Performers will be expected to store Signals with appropriate metadata to build a library of Signals against which future Signals can be compared. In Phase I, Performers will also conduct foundational research towards addressing TC-3 to drive towards target performance metrics for this category in Phases II and III.

At a certain time before testing begins, at its sole discretion, IARPA may specify the use of an alternative receiver (SDR) and/or other hardware for additional testing in one or both testbeds. The goal of introducing this “surprise hardware” is to discourage Performers from developing solutions that are customized to particular or limited hardware configurations. The newly specified SDRs will likely differ in performance, either diminished or enhanced, but the desired frequency range will remain unchanged. If IARPA specifies such “surprise” hardware will be used at the end of a phase, IARPA will provide those SDRs and other hardware to Performers as Government Furnished Equipment (GFE).

Exercise of the option to continue with the SCISRS program into Phase II will consider achievement of program metrics using all Phase I hardware, including “surprise hardware.”

Phase II – Focus on Altered & Mimicked Signal Anomalies: Phase II will last 12 months and will focus on detection and characterization of Anomalous Signals that can broadly be categorized as Mimicked or Altered Signals. Mimics are Signals that resemble known Overt Signals (e.g., LTE, WCDMA, OFDM) in frequency, bandwidth, and pulse shape but are either misleading or unrecognizable to the protocols established to receive them. As an example, a rogue device could emit a Signal that mimics a legitimate WiFi router. Altered Signals are known Overt Signals that have been further modulated or otherwise altered to carry additional information. Performers will also be expected to improve the methods they developed in Phase I to characterize the RF Baseline and to detect and characterize LPI Signals, in order to improve overall performance against any Anomalous Signal. The SDR and other collection hardware used in Phase II will be specified near the end of Phase I by IARPA. Similar to testing in Phase I, at any time during Phase II, IARPA may specify the use of an alternative SDR or other (“surprise”) hardware for additional testing in one or both testbeds.

Exercise of the option to continue to Phase III will consider performance against program metrics.

Phase III – Focus on Unintended Emissions: Phase III will last 12 months and will focus on detection and characterization of Anomalies in the Emanation Baseline arising from any microprocessors or other electronics. Focus on TC-2 in Phase III will be on significant parametric changes in these Emissions from previous RF Baselines. The SCISRS T&E teams will simulate changes that could arise from a variety of causes such as microprocessor malfunction or failure, loose or defective cabling, or from a deliberate act.

Performers will also be tasked with continuing to advance research into techniques to detect and characterize LPI, Altered, and Mimicked Signals and with improving their performance against TC-1.

Similar to the previous phase, the SDR and other collection hardware used in Phase III will be specified near the end of Phase II.

Similar to testing in Phases I and II, IARPA may specify the use of an alternative SDR or other (“surprise”) hardware for additional testing in one or both testbeds towards the end of Phase III.

1. B. Team Expertise

Collaborative efforts and teaming among Offerors are highly encouraged. It is anticipated that Performer teams will be multidisciplinary and may include expertise and experience in multiple fields related to the SCISRS program goals.

1. C. Program Scope and Limitations

Proposals shall explicitly address all of the following:

• Underlying theory: Proposed strategies to meet program-specified metrics must have firm theoretical bases that are described with sufficient detail that reviewers will be able to assess the viability of the approaches. Proposals shall properly reference previous work upon which their approach is founded.

• Development approach: Proposals shall describe the technical approach to meeting program metrics.

• Technical risks: Proposals shall identify technical risks and proposed mitigation strategies for each.

• Software development: Proposals shall describe the approach to software architecture, integration, and testing.

• Team and Program Management and Internal Testing and Evaluation: Proposals shall describe the approach to leveraging and managing the contributions of each member of the Performer team and the approach to testing and evaluation of the developed techniques independently from IARPA’s T&E.

The following areas of research are out of scope for the SCISRS program:

• Research that does not have strong theoretical and experimental foundations.

• Research that does not have plausible scientific support for the proposed results.

• Methods that are highly tuned for specific hardware and not readily extensible to multiple RF collection architectures.

During each performance period, Performers will build prototype algorithms for command and control, signal detection, and signal characterization that will be run and evaluated by IARPA and the T&E teams, at the T&E testbeds. Testing protocols do not allow for expert operators, although human-in-the-loop operation may be permitted at IARPA’s discretion.

1. D. Development Data

Achieving program metrics will require Performers to develop methods for 1) command and control of hardware for data collection, 2) signal detection, and 3) signal characterization. It is anticipated that Performers will need development data, specific to each testbed environment to achieve these objectives.

IARPA will provide development data appropriately collected from each testbed; i.e., RF Baseline data before the introduction of Anomalous Signals. Because Performer approaches may differ, particularly in hardware command and control, it is anticipated that Performers may need different development data, depending on their individual approaches. Development data requirements could include, for example, power versus time and frequency data from a particular node, phase data or data that represent node orchestration for task performance. Proposals must specify general development data requests (e.g., volumes, metadata) for the Offeror’s technical approach. Data requests may be refined after program kickoff when Performers will have visited the testbeds. IARPA will provide development data to each Performer but cannot guarantee all data requests can be accommodated. As an example, IARPA anticipates a typical data request will be for a recording of in-phase/quadrature (I/Q) or Fast Fourier Transform (FFT) data streams collected with SCISRS node hardware over a limited time (probably measured in minutes), potentially with novel collection strategies specified by the Performer. An example of a data request that likely could not be accommodated would be a RF measurement of individual testbed emitters isolated in an anechoic chamber. All development data collected by the T&E teams at the testbed sites will be made available to all Performers.

Performers may utilize additional training data beyond data provided by the T&E teams, as appropriate to their individual research approaches, but if they do, such data must be provided with at least Government Purpose Rights (GPR). In addition to the data, Performers must provide the Government with an accounting of all resources used and sources from which data were drawn and describe how the data will be used for development and training of algorithms. IARPA requires that Performer developed datasets be delivered with GPR. In the case of data generated by third parties, IARPA requires Performers to disclose the source of those data and such data must be freely available to all Performers. Offerors should anticipate that the data will be made available to all Performers for the purpose of work on the SCISRS program and that the Government shall retain the right to use the data for any future Government Purpose, which includes distribution of the data to contractors for Government Purposes. IARPA reserves the right to verify performance of Performer algorithms by independently training the algorithms; this would consist of starting with the initial, untrained algorithm and training it with data selected by the Government, following algorithm training instructions from the Performer.

1. E. Test and Evaluation (T&E)

IARPA research programs include rigorous, objective evaluations aimed at demonstrating achievement of carefully designed technical performance metrics. This section describes the testing and evaluation to which Performer deliverables will be subjected.

1. E.1. RF Testbeds

The SCISRS program will utilize two testbeds consisting of a controlled RF environment for the purpose of testing. The area of the testbeds will be approximately 1000 - 5000 square feet. The testbeds will be shielded and/or offset to attenuate impinging RF from external sources, initially targeting approximately 30 or 40 dB at 10 GHz. A minimum of 40 sources simultaneously emitting Overt Communication Signals at power levels consistent with Federal Communications Commission (FCC) standards will be resident in the testbeds. These sources will typically comprise commercially available telecommunication Signals, Emissions from IoT devices, or other similar Signals synthesized with a SDR-based system. A variety of electronic equipment will also be present to produce incidental, unintended RF Emissions, typically at much lower power levels than the Overt Signals.

1. E.2. RF Characterization Hardware

In part to compare test results objectively against metrics, IARPA will provide collection hardware to the Performers for their use at the time of testing. The specific collection hardware characteristics will be detailed early in each phase (or late in the previous phase, in the case of Phases II and III); a notional description of that hardware is provided here. Early in Phase I, Performers will have the opportunity to visit each SCISRS testbed. The hardware likely will consist of 4-8 collection nodes connected with precision timing cables to a timing network and a data network. Both the timing network and the data network will be connected to a fixed server via high capacity cables. It is anticipated that the timing network will be synchronized with sufficient precision for Performers to implement at least a rudimentary form of beam steering. Each node will consist of a commercial antenna, directional or omnidirectional, depending on the final configuration, feeding a particular Commercial Off-the-Shelf (COTS) SDR. The hardware configuration will likely be identical for all nodes. However, the node hardware, and in particular the SDR, may be different for each testbed and will likely change in later phases of the program. An API will be provided to control the hardware and to format raw data for analysis. Both I/Q and FFT data streams will be supported by the API. It is anticipated that Performers will build their own testbeds for the purpose of developing and testing their methods. IARPA does not intend that it be necessary for Performers to duplicate the SCISRS testbeds exactly for this purpose and therefore will not provide testbed design documents. Performers will be afforded the opportunity to ask questions of the T&E team during SCISRS testbed visits. Performers will have the opportunity to suggest changes to the node configuration, and potentially other hardware changes, in each testbed. Performers may choose to “volunteer” hardware (to be returned to Performers after test) for use in a testbed by the T&E teams and other Performers. To facilitate use of such hardware, Performers may be required to provide an API and other technical support.

Performers may be required to enter into and manage their own nondisclosure agreements (NDAs) to address any concerns about reverse engineering of proprietary hardware. The adoption of any Performer suggested hardware in a particular testbed is at the sole discretion of IARPA in consultation with the SCISRS T&E teams.

For Offeror reference, the server used in the Phase I testbeds is anticipated to have approximately 50 CPUs, 4 GPUs, ½ Tb of RAM, and cost approximately $20k. SDRs under consideration will have capabilities consistent with the Signal Hound BB60 and the Ettus 3Nxx series; IARPA offers this as information for Offerors’ reference for the purpose of estimating costs for any planned Offeror testbed build and for formulating algorithmic approaches. Reference to any particular hardware or manufacturer does not constitute a product endorsement. The precise “back-end” processing configuration used in Phase I for each testbed will be provided to Performers at program kickoff or early in Phase I. All data are to be analyzed with the IARPA provided servers, and the configurations in each testbed may not be identical.

1. E.3. RF Baseline Characterization

Although the vast majority of RF Baseline Signals originating within the testbed will be produced by Overt commercial devices, Offerors should be aware that there may be a small number of “pre-existing” Anomalies. Referring to the SCISRS BAA definition, these Signals are anomalous in the sense that they are custom Signals meant to hide in the ambient RF Baseline. They are meant to simulate suspicious Transmissions that would not be expected in a generic “clean” environment that would need to be flagged as such. Examples of such Signals might be a DSSS Signal trying to hide below the noise floor, a Signal spoofing a GPS satellite, or a Signal mimicking your own WiFi router. Such Signals would not belong in an expected, benign RF Baseline and would be excluded from the gray shaded area in the Venn diagram of Figure 2.

1. E.4. Introduction of Anomalous RF Signals

Prior to the final testing in each program phase, Anomalous Signals will be superimposed onto the RF Baseline by adding sources to each testbed. The types of Signals will vary by program phase as described in Section 1.A.2. During the final test at the end of each phase, the Performers will be expected to detect and characterize the Overt Communication Signals that in part comprise the pre-existing RF Baseline and the additional Anomalous Signals. Characterization must include specification of metadata, the details of which will be discussed in the next section.

1. F. Program Metrics

Achievement of metrics is a factor taken into account when determining whether to exercise options to continue performance under IARPA research contracts. IARPA has defined SCISRS program metrics to evaluate effectiveness of the proposed solutions in achieving the stated program goal and objectives, and to determine whether satisfactory progress is being made to warrant continued funding of the Performers.

The metrics described in this BAA are shared with the intent to scope the effort, while affording maximum flexibility, creativity, and innovation to Offerors proposing solutions to the stated problem. Proposals with a plan to exceed the defined metrics in one or more categories are desirable, provided that all of the other metrics are met, and provided that the proposals provide clear justification as to why the proposed approach will be able to meet or exceed the enhanced metric(s). Program metrics may be refined during the three phases of the SCISRS program; if metrics change, revised metrics will be communicated in a timely manner to Performers.

Beyond the content of Transmissions, Signals have interpretable features or metadata that distinguish them from one another. It is this metadata that must be reported by Performers and compared with ground truth in order for reported detections to be evaluated as false or positive; the true positive rates (TPR) of Transmission detection at a particular false positive rate (FPR)1 is the critical SCISRS metric. The specific metadata to be evaluated may be modified by IARPA at a later date. Representative examples of metadata include:

• Power

• Turn-on-time

• Duration

• Center frequencies

• Bandwidth

• Modulation type

• Device origination type

• Cyclostationary features

• Appropriate Emanation metadata

• Other interpretable features

SCISRS is focused on detecting and characterizing the Transmissions associated with Anomalous Signals.

Additionally, Performers will be required to report the metadata of pre-existing Overt Communication Signals originating from within the testbeds. It is not required to report turn-on times or durations of Transmissions associated with these Signals; they are presumed to be expected components of the RF Baseline. Conversely, determining and reporting the time of turn on, duration, and power of Anomalous Transmissions (including new Overt Communication Signals) are critical tasks and will be the principal means by which detection of an Anomalous Transmission will be validated. It may be possible for a Performer to receive credit for the detection of an Anomalous Transmission even if the turn-on time or duration (or other metadata) is incorrectly reported; in such a case, the presence of the Transmission would have to be reported sometime in the interval after the actual Transmission has turned on but before it has turned off. Anomalous Transmissions may overlap in time or frequency with other Transmissions; for such overlapping Transmissions, IARPA will consider the totality of Performer reported metadata to evaluate if the Anomalous Transmission was correctly reported or not. An example of one method for such a determination is taken from the Defense Advanced Research Projects Agency (DARPA) Radio Frequency Machine Learning Systems (RFMLS) program in which evaluation requires intersection over union (IoU) in time-frequency-category space > 0.5 for detection credit. Performers are required to group Anomalous Signals together that likely share a common origin. Either in their proposal or upon testing, Performers may propose or discover other metadata useful for the purpose of characterizing Signals that were not initially contemplated in this testing and evaluation protocol. Performers are encouraged to report these data.

A summary of the metric targets by phase is shown in Table 2. Metric targets provided in Table 2 are preliminary and subject to change over the course of the program. The targets get more challenging as the

1 TPR = TP/N; FPR = FP/N where TP = # of true positive detections, FP = # of false positives, FN = # false negatives, N = # actual transmissions = TP + FN program progresses through its three phases. As an example, the TPR for Overt Communication Signals goes up from 90% to 98% from Phase I to Phase III, as the FPR drops. The slightly reduced TPR for each Signal’s class of metadata of Communication Signals reflects the probability that Performers will not necessarily get every detail of each Signal correct. In Phase I, the LPI Transmissions will vary in difficulty such that the overall targeted TPR of the best Performers will be about 50%.

In Phase II, Performers are expected to improve their ability to detect LPI Transmissions from Phase I and develop algorithms to detect additional Transmission types introduced in Phase II. In Phase II, Performers’ target detection performance will be a TPR of 80% on LPI Transmissions comparable in difficulty to those encountered in Phase I. More difficult LPI Signals will be introduced in Phase II; the TPR target will be 50% for these. A new focus of Phase II is detection of Altered or Mimicked Signals. Metric targets for detection of these new Signals are modest in Phase II (50% TPR with 10% FPR) and will be more challenging in Phase III. Note that LPI are Communication Signals, and the TPR for their metadata applies accordingly. The number of types of intentional emitters will be carefully controlled in the RF testbeds but controlling the unintended RF Emissions (i.e., Emanations) will be much more difficult. By definition, the sources of these Emanations are not designed to emit, so controlling power, frequency, and other aspects of their Emissions will not be trivial. It will be the task of Performers to report and characterize all Signals they detect that are likely to be Emanations; the T&E teams will evaluate the Performers’ reports but may not validate each individual Signal. This is anticipated to be a very difficult task because Emanations are typically weak in signal strength. Prior to testing, IARPA may provide more detail as to the kinds of Emanations that are of interest. In Phase I, it is to the advantage of the Performer to report as many Emanations as possible because, as shown in Table 2, the metrics for Phases II and III will be set as multiples of the highest accurate number reported in Phase I. A hypothetical example is presented here to illustrate this point. In this example for Phase I, Performer A reports 200 validated Emanations, but Performer B reports only 100 validated Emanations. In Phase III then, the metric target for all Performers would be 400 (twice the highest number detected in Phase I); therefore, Performer A would have to report an additional 200 Emanations, but Performer B would have to report an additional 300 Emanations. In Phase III, where characterization of Emanations and related Anomalies becomes the focus of the program, the T&E teams may implement additional scoring rules that discourage guessing. In Phases I and II, there may be a small number of Emanation Anomalies that should be detectable by the Performers. By Phase III, the T&E teams will more precisely control these Anomalies, again targeting a 50% TPR and a 10% FPR.

Table 2 summarizes the target metrics planned for each of the program phases of the SCISRS program.

Table 2: SCISRS Program Metrics (see text above for additional details and clarification*):

Signal/Transmission Type

True Positive Rate (TPR), False Positive Rate (FPR)

Phase I Phase II Phase III

Overt Communication TPR = 90%, FPR = 5%

TPR = 94%,

FPR = 2%

TPR = 98%,

FPR =1%

LPI TPR = 50%,

FPR = 10%

TPR = 50%

(80% previous*), FPR =10%

TPR = 50%

(80% previous*), FPR = 10%

Altered/Mimicked - TPR = 50%, FPR = 10%

TPR=50%

(80% previous*), FPR=10%

Metadata for Comm.

Signals

TPR = 80%,

FPR = 10%

TPR = 88%,

FPR = 5%

TPR = 95%,

FPR = 2%

Emanations must report number* 125% Phase I best number*

200% Phase I best number*

TPR = 90%,

FPR = 10%

Anomalous Emanations TPR = 80%, FPR = 10%

TPR = 80%,

FPR =10%

TPR = 50%,

FPR = 10%

Because RF Emitters continuously transmit across the spectrum and back-end processors have finite storage and processing, near-real time analysis is implicit in the SCISRS problem formulation and execution. For consideration in the formulation of proposals, Offerors should anticipate the need to report Transmission metadata with a delay on the order of minutes or less between the metadata report and the termination of the actual Transmission. If an Offeror anticipates a tradeoff between report accuracy and reporting time, this should be described in the proposal. IARPA may provide additional constraints or metrics concerning reporting time for Phase I at kickoff. As with all metrics, there may be further refinement during later phases of the program.

1. G. Program Waypoints, Milestones, and Deliverables

Waypoints are the means by which the Performer clearly demonstrates the quantitative and timely progress that must be made for the overall concept to meet end-of-phase milestones. In other words, the intent of waypoints is to provide a clear measure of progress towards meeting the program milestones so the PM and advisors can provide more effective guidance and assistance to the Performers. Performance against these waypoints will be reviewed throughout the program, and the PM and non-government advisors will use performance against the waypoints to assess whether course corrections are needed to ensure program success.

1. G.1. Required Program Waypoints

The Government has identified the waypoints listed below for all Performers.

Waypoint 1: Preliminary Design Review (PDR). A PDR will occur during the first site visit, expected within the first four (4) months of contract award. The PDR must include documentation of the preliminary design and implementation plan, an accurate program schedule, and a mitigation plan for identified risks. The Performers shall describe their data collection strategies and algorithmic approaches at the PDR. Performers may choose to control the multiple SCISRS hardware nodes to act in novel, orchestrated ways. This will necessitate the Performer’s providing the command and control software to IARPA’s T&E teams to ensure proper functioning of the software in the IARPA testbeds and to enable the T&E teams to accommodate Performer requests for data. At the PDR, Performer progress, plans, proposed workflows and architectures will be presented by the Performer team and assessed by the Government with input from the T&E team and selected advisors. The PM will document any deficiencies in what was presented by the Performer at the PDR or within three (3) business days thereafter. The exit criteria for the Critical Design Review (CDR) will be discussed at the PDR. The PM may also refine the Exit Criteria for the CDR as part of the PDR documentation following the PDR.

Waypoint 2: Critical Design Review (CDR). A CDR will occur during a site visit to the Performer site within nine (9) months of the contract award. At the CDR, Performer progress, final plans, workflows, and architectures will be presented by the Performer and assessed by the Government PM, with input from the T&E team and selected advisors. The Performer will provide evidence that all CDR Exit Criteria have been met. The PM will document any deficiencies in what was presented by the Performer at the CDR or within three (3) business days thereafter. As a required deliverable for the CDR, Performers must provide a demonstration of their step-by-step process flow showing how data are to be analyzed as convincing evidence that their approach will result in an automated, highly functional, hardware-agnostic system. Performers should expect IARPA to elaborate on the specific requirements for CDR and the CDR Exit Criteria at the time of the PDR or before. Requirements may include live demonstration of Performer systems on or real-time analysis of IARPA provided data. In either case, Performers must demonstrate signal detection and characterization consistent with SCISRS end-of-phase metrics, or if unable to do so at the time of CDR, at least provide a compelling argument for how Performers will address deficiencies and achieve program metrics. Performers must also report the results of their internal testing at the CDR and describe how any deficiencies in performance will be addressed.

Offerors are encouraged to propose additional waypoints to quantify how their individual systems support the broader goals of the program. Well-constructed Offeror-defined waypoints provide task-driven intermediate steps towards meeting program technical metrics based on the Offeror’s approach.

Quantitative waypoints, reflected in the work plan and depicted on the schedule, help indicate progress toward milestones and reduce program risk by providing evidence that the technical and programmatic risks associated with the proposed approach are being addressed. A schedule of waypoint reviews must be included in proposals and shall include a rationale, definition, metrics, and an evaluation plan for each waypoint. Waypoint reviews may coincide with site visits, reviews at Government locations, and design reviews.

Government-defined program waypoints, milestones, and metrics may be refined during the various phases of SCISRS; changes will be communicated to Performers as quickly as possible.

1. G.2. Program Milestone Timeline and Deliverables

Table 3 shows a timeline for the program with defined waypoints, milestones, and deliverables.

Table 3: Program Waypoint, Milestone, and Deliverables Timeline:

Event Deliverables Months after Kick-off Comments Phase I Phase II Phase III Performers Visit T&E Testbeds

Initial visit must occur before end of month 2.

0-2 optional optional Helps inform Perfor-mers of strategy and technical approach.

PM Visits Performer Sites

Report on Action Items resolution from previous site visits and Action Item Closure Plans.

required:

4, 9, 15

PM

option:

6, 12, 18 required:

21 & 27

PM

option:

24 & 30 required:

33 & 39

PM

option:

36 & 42

Present program progress and results of internal testing.

Preliminary Design Review

(PDR)

Must include:

Preliminary design documentation and implementation plan.

Accurate program schedule.

Mitigation plan for identified risks.

4, coin-cides with first PM Visit

Present data collect-ion strategies and algorithmic approaches.

Showcase Performer testbeds or experimental setup.

PM to refine Exit Criteria for CDR.

Performers Request RF Baseline Data from T&E Team(s)

Requests must occur within given time frame and may require executable software consistent with T&E supplied API.

3-9 18-20 30-31 Not all requests may be accommodated.

T&E…