FAQ_v3b.pdf

Faithful Integrated Reverse-engineering and Exploitation (FIRE)

HR001123S0025

Frequently Asked Questions (FAQs)

Question 1: For evaluation of CPVA tools, what additional artifacts/information (e.g. source code or system spec.), in addition to the subject CPS, will you provide to the performers?

Answer 1: We can’t comment on information that might be specific to approaches. Recommend the proposer please describe assumptions, risks, and mitigations in the abstract and/or proposal.

Question 2: Can you tell us which processor architectures you are interested in?

Answer 2: Please see Question 1.

Question 3: What kind of data will be provided for modeling purposes?

Answer 3: Please see Question 1.

Question 4: Test article composition will make a big difference in what we propose. A drone w/ an FPGA in it vs. a smart meter w/ a CPU are very different. When will we learn what the test will be?

Answer 4: Please see Question 1.

Question 5: Are we operating with design info like: manuals, schematics, source code, encryption keys?

Answer 5: Please see Question 1.

Question 6: FPGA tools and techniques like determining encryption key and extracting netlists from bitstreams have been invented and re-invented on numerous programs. For efficiency, may we; 1) assume we will get the netlist for any included FPGA or 2) assume we have access to such tools provided by the program?

Answer 6: Please see Question 1.

Question 7: CPVA may be highly domain-specific (e.g. drones vs. ICS). Do you expect each proposal to propose generic CPVA tools applicable to a wide range of CPS, or it’s OK to propose domain-specific

CPVA tools (e.g. for drones only)?

Answer 7: Please see Question 1.

Question 8: If a TA5 performer has ideas for a “design for analysis” CPS system that has a design goal, making FIRE evaluation easier, is R&D of that sort in scope?

Answer 8: Please see Question 1.

Question 9: Are you expecting a solution across FPGA, DSP, PLC, CPU or should we propose with deep expertise in one family?

Answer 9: Please see Question 1 and Section 1.B.1 “TA1 Modeling: Scaling Complexity in Models while Maintaining Accuracy “in the BAA.

Question 10: Would you be open to AI/ML as being part of our approach? (if it makes sense of course)

Answer 10: We are open to approaches that meets the program goals and metrics. Please see

Section 1.B “Program Description” of the BAA for the program goals and metrics. Also see

Question 1.

Frequently Asked Questions (FAQs)

Question 11: Are stochastic countermeasures and ML-based approaches in scope even if they may not eliminate an attack vector completely?

Answer 11: Please see Question 1.

Question 12: What protocols do we need to support?

Answer 12: Please see Question 1.

Question 13: What communications bus do we need to support? CAN? 1553? Ethernet?

Answer 13: Please see Question 1.

Question 14: Which TA is responsible for vulnerability discovery and patch generation (seems cross-cutting over TA1, TA2, TA4)?

Answer 14: Please see Section 1.B “Program Description” of the BAA for specific descriptions.

Question 15: The BAA describes a preference for integrated TA1-TA4 proposals. Does this imply the expected award for an integrated TA1-TA4 solution could be multiple times an individual TA award?

Answer 15: There is no implication. Please see Section V.A “Evaluation Criteria” in the BAA for evaluation criteria.

Question 16: With GOMAC being next week can the abstract deadline by extended by 1 week?

Answer 16: No, we cannot make special arrangements to accommodate specific proposers.

Question 17: A typical DARPA BAA has a 4 week period between BAA release and abstract submission.

FIRE only has 2 weeks and most of this community will be at GOMAC next week. Can the abstract due date be pushed back 1-2 weeks?

Answer 17: Please see Question 16.

Question 18: Can DARPA extend the abstract deadline by a week? The abstract for one TA is 4 pages if multiple Tas, it requires more pages. This is a complex program two-week timeline is too short.

Answer 18: Please see Question 16.

Question 19: Does the government plan to downselect performers even if they meet the metrics for each phase?

Answer 19: We will not speculate on future decisions.

Question 20: Does one need a clearance to participate and bid?

Answer 20: Please see Section 3.B.3 “Ability to support classified development” of the BAA for eligibility information.

Question 21: Will the lightning session slides be published?

Answer 21: No.

Frequently Asked Questions (FAQs)

Question 22: Will the CPSs include both OT and IT networks. Example: the medical pump includes the nurse’s station computer.

Answer 22: Please see Question 1.

Question 23: Would a TA-5 system need commercial equipment, or could synthetic but representative equipment be used?

Answer 23: Please see Question 1.

Question 24: Do we have to submit an abstract?

Answer 24: No. Abstracts can be used to obtain initial feedback which might inform proposal writing decisions and whether to continue with proposal submission.

Question 25: Will submitting an subtract improve our chances of getting a contract?

Answer 25: Please see Answer 24.

Question 26: can you clarify the scope of a software component?

– Image level “bitstream” “binary”

– library e.g. crypto library w/ multiple algorithms

– instance level e.g. single crypto algo from a library

Answer 26: A component is defined in Section 1.A “Background” of the BAA. A single “binary” may contain one or more components.

Question 27: can you clarify what constitutes a hardware component? When you say “up to 100” are you describing processing/sensing components or does that include passive (resistors, capacitors).

Answer 27: A component is defined in Section 1.A “Background” of the BAA. We can’t speculate on the purpose of resistor or capacitors for example.

Question 28: Is there any human-in-the-loop or the whole analysis should be automated? If there is a human, what competencies should one possess: SW/HW/RE engineering?

Answer 28: How you choose to use your 1-month metric period is up to your technical approach. Please describe assumptions, risks, and mitigations in the abstract and/or proposal.

Also see Question 1.

Question 29: Is the human operators and their perception of system health/status in scope for attack path modelling?

Answer 29: Please see Question 28.

Question 30: Other than “one month,” what resource utilization targets do you envision?

- Total staff-months during that month window

- Total reusable + non-reusable infrastructure costs (compute, instrumentation, space, power, etc.)

Frequently Asked Questions (FAQs)

Answer 30: Please see Question 28.

Question 31: Are there metrics for patch effects on benign/intend system behavior?

Answer 31: Please refer to the BAA on proposing patch metrics.

Question 32: Is survivability vs. complete exploit neutralization a valid FIRE outcome?

Answer 32: Please see Question 31.

Question 33: What patches are in scope? Examples: re-image firmware, update O/S or control s/w, reprogram FPGA, replace sensor, add ASIC or device to control board, re-architecture system to isolate unpatchable component, etc.

Answer 33: Please see Questions 31 and 1.

Question 34: Are patches that raise the bar for future attacks in scope? E.g.: reduce likelihood of success, raise the risk of attributability/detectability, raise the likelihood of an alarm presented to a human operator, etc.

Answer 34: Please see Questions 31 and 1.

Question 35: Some vulnerabilities can be hidden by other vulnerabilities. Detecting the second can only be done if the first is corrected. If patching is not measured, how will DARPA handle these scenarios.

Answer 35: Please see Question 31.

Question 36: Is a goal of this program automated exploitation and patching, or should we just be focusing on the environmental setup to be able to conduct this analysis?

Answer 36: Please refer to the program description and metrics in Section 1.B.” Program

Description” of the BAA.

Question 37: Can you explain more about TA3? What are you looking for? Why wouldn’t TA4 do the preparation for integration purposes?

Answer 37: Please refer to Section 1.B.” Program Description” of the BAA for the description.

Question 38: What is the motivation/rationale behind the 1 month schedule?

Answer 38: Please refer to Section 1.B.” Program Description” of the BAA for the program metrics description.

Question 39: TA5 is charged with building the test + evaluation models. Are TA1-TA3/TA4 performers expected or allowed to define the targets they use for the research phase?

Answer 39: Please refer to the BAA for IV&V descriptions. Also see Question 1.

Question 40: Which TA is responsible for defining a patch? That’s not an analysis problem.

Answer 40: Please see Question 31.

Question 41: How will the expected behavior(s) of an exploit be specified? E.g., A drone losing altitude by 30m will crash if it is flying at 20m. Is the exploit crash or the loss of altitude?

Frequently Asked Questions (FAQs)

Answer 41: Exploits are defined in Section 1.A. “Background” of the BAA.

Question 42: Isn’t defining relevant unexpected behavior part of the problem statement?

Answer 42: Please see the definitions of vulnerabilities and exploits within Section 1.a.

“Background” of the BAA.

Question 43: Is there a “weight” value to any TA within a combined proposal?

Answer 43: No. Evaluation criteria can be found in Section V.A “Evaluation Criteria” of the BAA.

Question 44: What are the expected interactions between TA3 and TA5? Will there be models provided?

How sophisticated will they be?

Answer 44: Please see Questions 1 and 39.

Question 45: Are we allowed to propose self-defined interim metrics to show progress for our solution?

Answer 45: Yes.

Question 46: Are we allowed to propose self-defined interim CPS to show progress for our solution?

Answer 46: Yes. Also see question 45.

Question 47: Can you please clarify what quantity of the systems will be needed for each specific deliverable for TA5?

Answer 47: This will be clarified in Section 1.D. “Schedule/Milestones” of Amendment 01 of the

BAA.

Question 48: Can you please clarify how long each of the QPRs should be for costing purposes?

Answer 48: This will be clarified in Section 1.D. “Schedule/Milestones” Amendment 01 of the

BAA.

Question 49: What does “interchangeable TA1, TA2, and TA3 approaches” in the BAA mean?

Answer 49: Interchangeable can mean the ability to replace a TA1, TA2 and/or TA3 solution with a better solution during program execution so as to meet the program goals. Please also see

Question 1.

Question 50: Would it still be useful if we provide an abstract on Monday, or would you recommend submitting what we have, albeit less complete, on Friday?

Answer 50: Please see section IV.C.1.a. “Abstract Due Date” of the BAA.

Question 51: Can you please provide a 4 TA (combined TA1 through TA4) Cost proposal sheet?

Answer 51: Yes, this has been uploaded to SAM.gov as an additional attachment 3.

Question 52: Would the government prefer or allow a Table of Contents (TOC) to be included between the Technical Proposal Volume I and Section II and would the TOC count against the total page limitation?

Frequently Asked Questions (FAQs)

Answer 52: Yes, the government would allow a Table of Contents. It will not count against the page limit.

Question 53: Are the hardware tool deliveries listed with the milestones and deliverables section expected to be primarily delivery of documentation and design documentation or does this include delivery of enhancements or upgrades applied to contractor owned hardware used to support data collection in TA3 and TA4 activities.

Answer 53: This only applies to hardware developed/procured using program dollars. Any enhancements or upgrades to hardware developed/procured using program dollars are expected to be delivered as well. Please also refer to data rights and intellectual property descriptions found throughout the BAA.

Question 54: BAA states that performers in TA4 must have facilities and personnel involved in demonstrations able to be accredited to TS-SCI. In the case of physical measurements supporting TA3 objectives without post-processing, may performers utilize existing controlled access facilities of a lower accreditation (e.g. CUI or Collateral Secret) along with a customer approved OPSEC plan and cleared personnel performing measurement tasks?

Answer 54: This is a case-by-case decision based on the system/object being observed. It is encouraged that proposals discuss any risks and/or mitigations.

Question 55: The RFP states that classified submission of addendums may be submitted one of two ways

(USPS and hand-carry). Would the government approve electronic submission via SIPR or SAV account?

Answer 55: Yes, but only if it is coordinated with the FIRE program security team prior to submission. Please note that submissions received past the deadline stated in the BAA may not be reviewed even for unforeseen circumstances such as network failure.

Question 56: Does the classified addendum portion count against the unclassified page count?

Answer 56: Yes, the classified addendum counts against the total page count. There is no unclassified page count that is separate from a classified page count. The total page count for both unclassified and classified addendums can be found in the BAA in Section IV.2. Full

Proposal Format.

Question 57: Does TA5 need to provide any instrumentation in addition to the test bench itself? (logic analyzers, disassembler hardware/software, etc.)

Answer 57: No and please see Question 1.

Question 58: For the TA5-A evaluation environment, does the team prefer a more expensive, high performance FPGAs as a potential CPS flight controller over a lower-cost open-source solution given the realism that it provides?

Answer 58: Please see Question 1.

Question 59: We are a little confused about the difference between plant and environment. Is it possible to go into slightly more detail on the difference?

Answer 59: The definitions can be found in Section I.A “Background.”

Frequently Asked Questions (FAQs)

Question 60: The cost template only has 3 TAs and 3 phases. 4 TAs and 4 phases are needed. When will a new cost template with these changes be made available?

Answer 60: Please see Question 51.

Question 61: Does the testbed platform A/B need to actually fly?

Answer 61: The testbed platform needs to be able to demonstrate observable effect(s) as stated in the BAA under Section I.A. “Background” and the program metrics.

Question 62: Is TA5 expected to deliver a vehicle dynamic system in addition to a test bed?

Answer 62: TA5 is expected to deliver physical systems for test and evaluation, and demonstration. Please see questions 1 and 57 with regards to whether a test bed (e.g., physics models/simulators, lab equipment) is also expected.

Question 63: Are unclassified proposers required to provide an intent to propose statement?

Answer 63: No.

Question 64: Can a performer submit both a TA1-TA4 proposal and a TA5 proposal?

Answer 64: Yes, however two separate proposals with firewalled plans are required. Additional details can be found in the BAA in Section III.D.2. “proposing to Multiple TAs.”

Question 65: Is an active FCL required to submit a proposal to TA5 including Track B, or is an in progress

FCL application adequate? Our company is going through the process of getting an FCL. At this point we have no reason to believe our FCL application will not be complete before final selection and/or a contract award. It is unlikely but not impossible that our FCL is granted before the proposal due date of

May 19th. We do have cleared employees and our proposal team will review the classified addendum.

Answer 65: No, but please see eligibility criteria as stated in the BAA section III.D.3. “Ability to support Classified Development.”

Question 66: Are TA5 performers also expected to create and insert vulnerabilities?

Answer 66: No, however TA5 is expected to work closely with the government IV&V team to create, insert, and verify vulnerabilities. It is encouraged that the proposal describe how they will work closely with the IV&V team to create, insert and verify vulnerabilities in support of program goals and phase metrics. Also please see Question 1.

Question 67: We already have a vulnerability analysis setup for ARM, x86 and Power PC, can we propose direct to Phase 2?

Answer 67: There is no direct to phase 2 option. Furthermore, the BAA states that the program is interested in more than general processing computing devices as stated in the TA1 section and the background section.

Question 68: Can you elaborate on how the program will track progress so we can properly address it in the proposal?

Frequently Asked Questions (FAQs)

Answer 68: Please refer to the program goals and program metrics. Also, it is encouraged that the proposal describes how the technical solution can scale to and constructively meet the program goals and metrics.

Question 69: We specialize in side channel analysis of FPGAs, which TA should be propose to?

Answer 69: As stated in the BAA, “The FIRE program is only interested in tools for cyber-physical vulnerabilities, tools that address ONLY cyber vulnerabilities or ONLY physical vulnerabilities are outside of the scope of the program.” Please also see Question 67.