SCISRS Q.A Round One.pdf

Securing Compartmented Information with Smart Radio Systems (SCISRS)

BAA-20-03

Notice: IARPA will accept questions until October 9th

Industry Questions and Government Answers

Question number

GENERAL

INSTRUCTIONS Industry Questions Paragraph & Page # Government Responses

Question number

Section 1 -

TECHNICAL

PROGRAM

OVERVIEW

Government Responses question 1 round 1

Which GPU device would be available in the server? 1. E.2.

P. 14

The precise “back-end” processing configuration used in Phase I for each testbed will be provided to Performers at program kickoff or early in Phase I (as stated in the BAA on page 14). The SCISRS T&E teams are in the process of finalizing the GPU device which will be a standard commercial device.

question 2 round 1

T&E server configuration does not mention connection to data network capability. Would you clarify connection capability to the data network (speed, type, etc)? 1. E.1.

P. 13

The T&E teams will provide access to a local testbed network in the room used by Performers while they are on-site. This network will be a standard TCP/IP network with 1 Gbps copper connections and notionally 10-40 Gbps fiber connections.

question 3 round 1

Device origination type: Would you clarify this term as this can be broadly interpreted. Is geolocation of device required? 1. F.

P. 15

Please refer to Figures 1 and 2 for notional device types; this is not meant to be an exhaustive list. Additional detail may be provided at kickoff. Geolocation is not a requirement of the SCISRS program.

question 4 round 1

In subsequent phase, the metric will be "twice the highest number detected in Phase I" (or previous phase). What if the highest detection is false positive since not all detections would be validated by T&E team? 1. F.

P. 16

The T&E teams will validate as many Emanations as possible given finite time and resources to do so. The BAA points out that metrics may be refined as necessary throughout the program.

question 6 round 1

What would be the highest frequency range performers are expected to detect and analyze? P. 9 indicated, “The frequency bands of interest will range from 100 MHz to 6 GHz in the initial phase of the program, but it is anticipated that this frequency range will increase in subsequent phases” but did not specify the upper limit on band of interest.

1. A.1.

P. 9

The BAA purposefully does not specify a frequency range for later phases. One obvious consideration is the receiver capabilities ultimately chosen for use.

question 8 round 1

Please clarify that while the API will support both I/Q and FFT data streams, it will or will not support them both simultaneously from the same collection node. 1. E.2.

P. 14

The API is currently under development. It will not likely support both I/Q and FFT data streams. If the Offeror's approach requires both, this should be specified in their proposal.

question 13 round 1

Will SDR nodes be allowed to transmit? 1. E.1.

P. 13

SDR nodes will not be allowed to transmit, at least in Phase I. It should be noted that at the end of each phase, Performers must also report recommended changes to the SCISRS hardware for the next phase. This report enables a way for the PM and the T&E teams to more formally and deliberately consider hardware changes.

question 14 round 1

On page 15 of the BAA, Performers must report metadata that is matched against ground truth at a 90% TPR in phase 1. What is the exact list of metadata which must be correctly matched? (e.g. "Device Origination Type" may be quite challenging vs "Turn-on-time") How accurate must reported metadata be to get credit? (e.g. is it a failure to misdetect a modulation as QAM16 instead of QAM64? Or to measure a bandwidth at 250KHz instead of 100KHz?)

1. F.

P. 15-16

Similar to detection credit, the T&E teams will develop a plan based on principles like IoU; such plans may provide partial credit. The exact scoring criteria will be further detailed at kickoff and will be based in part on the capabilities promised in each Offeror's proposal.

question 15 round 1

Can you clarify what constitutes training data that must be provided to government as this seems very broad? For example, does this include all signal data generated in the performer developed test bed? 1. D.

P. 13

Machine Learning (ML) approaches typically rely on training data.

Only data used to train ML algorithms would be included. Testing data would not be included, unless it was used to re-train. Any data that is not recorded would not be included.

question 16 round 1

What is meant by “device origination type”? 1. F.

P. 15

Please refer to Figures 1 and 2 for notional device types.

question 17 round 1

The table of metrics, Table 2 states that the required metrics for Phases I and II classifications of anomalous unintended emanations are TPR = 80% and FPR = 10%. The text suggests the characterization of these emanations is not required until Phase III, with less aggressive metrics (TPR = 50% and FPR = 10%). Please clarify the timelines for the characterization of emanations relative to the phases of the program.

1. F.

P. 17

Emanation Anomalies are not the focus on Phases I and II, but a small number of evident ones may be presented by the T&E teams.

In order to receive credit, Performers must characterize these Anomalies by their metadata (e.g., turn-on time, center frequency, frequency harmonics). The T&E teams will dedicate more time and effort in Phase III to engineer a larger number of Emanation Anomalies over a broader range of detection difficulty.

question 18 round 1

Will metadata for detected signal types that is delivered to IARPA T&E team be shared with other performers? 1. F.

P. 14-16

Within IARPA programs, it is normal for Performer teams to know how each other did on evaluations from a performance metric perspective. It instills a good sense of competition and accomplishment. The exact scoring of each element typically isn't shared but discussing or highlighting specific examples or cases that certain approaches are having difficulty with can be of value.

The BAA stipulates that Performers may propose or discover other metadata useful for the purpose of characterizing Signals that were not initially contemplated in this testing and evaluation protocol. If Offerors deem that sharing such metadata will compromise their intellectual property, they must specify this in their proposal.

question 19 round 1

Will IARPA T&E evaluation of the algorithms/software for command and control, signal detection, and signal collection in the T&E testbed (p.12 of BAA) be considered a program metric for advancement? 1. C.

P. 12

Performer developed algorithms are a deliverable and not a metric.

Deliverables will be independently tested as described in the BAA, and results will be evaluated against the specified metrics in Table 2 (page 17). Offerors should note that program metrics are not the only factor considered for advancement to later phases.

question 20 round 1

Can you provide sample microprocessor types (e.g. architecture, multi-threaded?) that would be used in the IARPA T&E testbeds? 1. A.1.

P. 9 - 10

This questions appears to be asking about the kinds of microprocessors that would be used to generate Emanations. The BAA indicates that Offerors should expect that the RF Baselines of each testbed will be replete with Overt Signals commonly found in today’s wireless-enabled world including unintended RF Emissions from microprocessors and other electronics. Further detail may be provided at program kickoff and early in subsequent phases.

question 21 round 1

Will localization of the intended or unintended emission sources in the T&E testbeds be required for any program delivery? 1. A.2.

P. 11

Localization is not required.

question 22 round 1

Frequency range below 100MHz: can we safely ignore frequency contents (especially emanations) below this frequency, or should the setup be changed to cover that range as well? 1. A.1.

P. 9

The BAA purposefully does not specify a frequency range for later phases. One obvious consideration is the antenna capabilities ultimately chosen for use.

question 23 round 1

Do we have any control over or information on the types of applications (microprograms) being executed on the devices in the testbed, so that we can profile their emanations over time? 1. A.1.

P. 10

In real world scenarios that the testbeds seek to replicate, the applications that are running on microprocessors emitting Emanations are not necessarily known. In their proposal, the Offeror should estimate how much developmental data is required to profile these Emanations over time if this is critical to their approach.

question 24 round 1

Given that emanations are quite application-dependent, how is the “anomaly” defined? Do we know a set of applications that will be always executed on these devices (so that we can create a baseline, and report anomaly if they are different than them), or something else?

In other words, what types of anomalous behavior should we expect from each device in terms of emanations (e.g., are they intentionally transmitting information through emanations — i.e., they mimic the behavior of the normal emanation but actually transmitting sensitive information? or should we be worried that these emanations are unintentionally leaking information)

SECTION 1, Table 1 P. 6

The BAA definition of Anomaly still applies. In real world scenarios that the testbeds seek to replicate, the applications that are running on microprocessors emitting Emanations are not necessarily known.

The Offeror mentions intentional transmission of information through emanations; this would not strictly fit the BAA definition of Emanation Anomaly since the transmission of information is now intentional. If the mode of transmission is meant to hide, the Anomaly would be LPI. If the mode is meant to "look" like a normal Emanation, the Anomaly would be a Mimic as the Offeror points out. IARPA expects to further refine metrics for Emanation Anomalies in Phase III when this is a focus of the program.

question 26 round 1

We are confused about whether the Overt Signals from the ambient RF environment are included in the RF Baseline or not. In Table 1 we see:

RF Baseline: The RF Baseline of a testbed is the superposition of the RF Background (including geographically specific components and components that may change over time) plus RF Emissions from all sources located within the testbed itself. The RF Baseline is composed of the RF Background, the Overt Baseline, and the Emanation Baseline.

Then in subsequent sections:

electronics). The RF Baselines may vary by band. The frequency bands of interest will range from 100 MHz to 6 GHz in the initial phase of the program, but it is anticipated that this frequency range will increase in subsequent phases. In addition to RF sources located inside each testbed, there will be unavoidable RF radiation propagating into the testbeds from the surrounding environment. The testbeds will be established in different geographical locations and, as a result, will likely present very different RF Backgrounds. The RF Background is independent of the RF sources that will be installed inside each testbed but inevitably contributes to the RF Baseline of each testbed. The kind of RF Background to be expected for each testbed will be provided in advance shortly after program kickoff and will conform to definitions found within the and

International Telecommunications Union (ITU) Recommendation ITU-R P.372-13. The RF Background is expected to change over time as new sources of RF are introduced external to a testbed or as “old” sources turn off, move, or otherwise change. An example of this would be variations resulting from the Signals from a passing airplane; these Signals will likely propagate into a nearby testbed and represent a temporal and spatial fluctuation to the testbed RF Baseline. Offerors are referred to the Venn diagram in Figure 2

But then in the section on the RF test beds, we learn that there will be some kind of 50-dB shielding placed around the testbeds:

1. E.1. RF Testbeds The SCISRS program will utilize two testbeds consisting of a controlled RF environment for the purpose of testing. The area of the testbeds will be approximately 1000 - 5000 square feet. The testbeds will be shielded to attenuate impinging RF from external sources, initially targeting approximately 40 or 50 dB at 10 Ghz. A minimum of 40 sources simultaneously emitting Overt Communication Signals at power levels consistent with Federal Communications Commission (FCC) standards will be resident in the testbeds. These sources will typically comprise commercially available telecommunication Signals, Emissions from IoT devices, or other similar Signals synthesized with a SDR-based system. A variety of electronic equipment will also be present to produce incidental, unintended RF Emissions, typically at much lower power levels than the Overt Signals.

So are ambient RF (cellular, radar, TV, etc.) signals allowed to appear with reasonable strength at the antennas connected to the receiving SDRs or not?

If the testbeds are shielded, then the local RF variation won't matter to the program. But when a SCISRS device is to operate in the real world, at some later date, there won't be any shielding. It will have to deal with whatever is present at the antenna.

SECTION 1, Table 1 P. 6-7

1. A.1.

P. 9-10

1. E.1 P. 13

There will be some shielding in the testbeds as specified by the BAA. Please note that the full BAA supersedes anything stated in the technical draft of the BAA. It is anticipated that this shielding will not prevent all RF Background Signals from entering into the testbed at measurable strength. It is desirable from a T&E perspective to control the RF environment to the maximum extent possible. In the real world this is not always possible, and so it is consistent with SCISRS objectives to have some external Signals enter the testbed.

question 27 round 1

There seems to be an ambiguity between the RF Baseline and Anomalous Signals. Zigbee and WiFi are mentioned in the Venn Diagrams, so we assume that various ISM bands are of interest. Anybody can transmit in the band as long as they follow some simple power and spectral mask rules. So, are all New Wi-Fi signals anomalous?

SECTION 1, Figure 1, Figure 2 P. 7-8

The BAA anticipates that Performers will need development data, specific to each testbed environment to achieve these objectives.

IARPA will provide development data appropriately collected from each testbed; i.e., RF Baseline data before the introduction of Anomalous Signals. If new Signals appear that were not present when the developmental data was collected, they should be flagged as Anomalies.

question 28 round 1

Suppose we stare at some RF band of interest for a while, and detect and characterize the signals there. We see N signals, and we're pretty confident we understand them. Later, another signal turns on. Since the definition of Anomaly here is a "Signal that differs from the RF Baseline", it is an anomaly. On the other hand, "the RF Baseline of a testbed is the superposition of the RF Background (including geographically specific components and components that may change over time) plus RF Emissions from all sources located within the Testbed itself." So, Anomalies differ from the Baseline, which means they are not in the Baseline, which means they appear at some point in time. On the other hand, the Baseline is inherently time-varying.

So, is the idea that as new signals appear, they are flagged as potential Anomalies, characterized, and then added to the Baseline?

Also, how recent do they need to be, to be qualified as normal and not anomalous.

SECTION 1, Figure 1, Figure 2 P. 7-8

1. E.4 P. 14

The BAA anticipates that Performers will need development data, specific to each testbed environment to achieve these objectives.

IARPA will provide development data appropriately collected from each testbed; i.e., RF Baseline data before the introduction of Anomalous Signals. If new Signals appear that were not present when the developmental data was collected, they should be flagged as Anomalies.

Section 2 - AWARD

INFORMATION

Section 3 -

ELIGIBILITY

INFORMATION

question 30 round 1 Can innovative start-up companies which are small businesses propose to this project as a Prime? 3. A.

P. 23

Yes.

Section 4 - PROPOSAL

INFORMATON

question 5 round 1

Performers may choose to 'volunteer' hardware". Would it be possible for us to contribute to the T&E hardware at no cost to the Government? What would be the procedure for this? 1. E.2 P. 14

SECTION 1, Table 3 P. 19

The BAA does not lay out a specific procedure for how hardware would be volunteered. It does state that the PM will be in frequent communication, in person or by teleconference, with Performers including both Prime and Subcontractors. In particular, Performers will have the opportunity to visit the IARPA established testbeds early in each phase and suggest changes to the node configuration and potentially other hardware in each testbed. It should be noted that at the end of each phase, Performers must also report recommended changes to the SCISRS hardware for the next phase. This report enables the PM and the T&E teams to more formally and deliberately consider hardware changes.

question 7 round 1

The RDMP is listed as a component of the Detailed Proposal, Section 3 (see letter designators "L" and "M"), but is also listed as a required Attachment (Attachment 11). This could imply the government wants to see the RDMP twice in the proposal - please clarify.

4. B.1.c P. 32

Offerors are required to submit RDMP as an attachment only.

question 9 round 1

Our organization has highly applicable direct experience (gained through performance in a classified capacity) with a number of SCISRS technical topics. Inability to cite this experience puts our organization at a disadvantage. Would the Government accept a classified TS/SCI addendum to augment “4. B.1.c.”, subsection “G. Offeror’s previous accomplishments”?

4. B P. 26

Although classified proposals are not anticipated for this program, if an Offeror chooses to submit a classified proposal, the Offeror must first contact IARPA via dni-IARPA-BAA-20-03@IARPA.gov for detailed submittal instructions. In no case shall classified information be uploaded into IDEAS. Any proposal submitted, classified or unclassified must not exceed the page limit as described in 4. B. 1.

question 10 round 1

Our organization has significant direct experience (gained through performance in a classified capacity) transitioning technologies and capabilities to IC users as indicated to be of value in evaluation factor “5.

A.1.c. Contribution and Relevance to the IARPA Mission and Program Goal”. Inability to cite this experience puts our organization at a disadvantage. Would the Government accept a classified TS/SCI addendum to augment “4. B.1.c.”, subsection “G. Offeror’s previous accomplishments”?

4. B P. 26

Although classified proposals are not anticipated for this program, if an Offeror chooses to submit a classified proposal, the Offeror must first contact IARPA via dni-IARPA-BAA-20-03@IARPA.gov for detailed submittal instructions. In no case shall classified information be uploaded into IDEAS. Any proposal submitted, classified or unclassified must not exceed the page limit as described in 4. B. 1.

question 11 round 1

The SCISRS BAA does not directly indicate an IARPA desire for results to be published in peer-reviewed publications. To what degree are proposers encouraged to publish in peer-reviewed publications? 6. B.8.

P. 45

The BAA lays out clearly the goals of the program. Publication is encouraged but not required.

question 12 round 1

Please provide clarification on “4. B.1.c. Section 3: Detailed Proposal Information” subsection “L. Research Data Management Plan”. Are all offerors required to submit an RDMP, regardless of intent to publish in peer-reviewed publications?

4. B.1.c.

P. 32

If the Offeror has no intent to publish their data, they should indicate this on the RDMP attachment.

question 25 round 1

Can Customer Meetings be scheduled at Universities where the Performer Testbeds may be located? 1. H.1.

1. H.2.

P. 21

"Customer Meetings" appears to refer to site visits. Site visits by the Government Team including T&E team members and selected non-government advisors will generally take place up to four times yearly during the life of the program. These visits will occur at the Performer’s facility. IARPA Performers can be universities. The site to be visited may correspond either to the Prime or to a Subcontractor.

question 30 round 1

This section describes the supporting information required for each cost element for the proposal. Does detailed supporting information (i.e. quotes, indirect rate support, cost/price analysis) need to be included in the cost proposal submission or can it be submitted if the cost proposal is selected for cost or price analysis by the contracting officer?

4. B.2.b.

P. 37

The Cost Volume is only required if the Offeror's proposal has been selected for negotiation. The notification of selection for negotiation will be issued in writing by the CO and will include a request to submit the full Cost Volume within 10 business days or as otherwise authorized by the contracting officer.

question 31 round 1

Are contractors’ required to propose cost share for the program? 4. B.2.b.

P. 37

Cost Sharing is not required from Offerors and is not an evaluation criterion, but is encouraged where there is a reasonable probability of a potential commercial application related to the proposed research and development effort. If cost sharing is proposed, describe the nature and amount of cost-sharing, if any. Reference Resource Share from Section 4.B.1.c. Section 3 (J) of the BAA.

question 32 round 1 If an offerror chooses not to pursue publication in a peer-reviewed journal or conference and states their intent as such, is any Research Data produced? 4. B.1.c.

P. 32 If the Offeror has no intent to publish their data, they should indicate this on the RDMP attachment.

question 33 round 1 Is there a recommendation as to the length of the Summary? 4. B.

P. 26 The page limit for the Summary is five pages.

Section 5 - PROPOSAL

REVIEW

INFORMATION

Section 6 - AWARD

ADMINISTRATION

INFORMATION