SCEPTER_FAQ_02162022.pdf

SCEPTER

HR001122S0013

Frequently Asked Questions Last Updated: 2/16/2022

Q1. Will it be possible to get access to a video of the session via SAFE later, if we cannot get in, or a copy of the slides?

A1. No. Slides will not be provided. No video will be provided.

Q2. About identifying the best courses of actions, can we use human in the loop learning to select courses of actions? or can it be a manually selected?

A2. Yes. The user interface (UI) and user experience (UX) should be thoughtfully considered.

Q3. Will you be sharing contact information for teaming purposes?

A3. Yes, the industry day registration list will be shared for the purposes of teaming.

Q4. Are there any KPIs that should be addressed other than those that were briefed today? Can you provide KPIs for each spiral?

A4. Performance metrics are very important to the program and should be specifically addressed in proposals. Additional metrics may be proposed especially if they more clearly demonstrate progress against the program goals.

Q5. Can you provide realistic scenarios for FA1 so we can formulate our proposed architecture for this effort?

A5. Proposers may use historical scenarios or proposer defined scenarios for the purposes of proposing.

Q6. Do you have a preferred simulator that performers should select? You listed several options.

A6. No. Proposers are encouraged to select the simulation environment that they feel are most likely to achieve the program metrics.

Q7. Can you say what the scenario in Phase 1 will be?

A7. See response to Question #5.

Q8. Could you clarify what is a "unit" for purposes of FA2 metrics? A military formation is a unit with potentially many entities. Are you measuring directly controlled formations or total # of entities?

A8. Solutions that can accommodate flexible unit definition to explore various scenario resolutions are desirable. By way of example, in some scenarios a unit may represent a single aircraft. In other scenarios a squadron or a wing may be represented as a single ‘unit’.

Q9. In the evaluation of the COAs, there is a qualitative eval of whether the COAs are human-interpretable (e.g. "looks like" an ambush). There is a risk when training in sims that the AI has learned to exploit something that is simulation-specific (e.g. breaking physics). Is the role of the trusted sims (e.g. OneSAF) to provide an environment where we can safely assume nothing like that is possible, or should we include metrics/justification in the evaluation portion of our proposal to validate/verify that the AI has learned a genuine, generalizable strategy vs. sim-specific strategy?

A9. We recognize that trusted, “high-fidelity” simulators are not perfect. Proposers should demonstrate an understanding of the strengths and weaknesses of the simulator proposed. Proposers may propose improvements as necessary to overcome any limitations that prevent the SCEPTER validation as described in the BAA.

Q10. Assume simulations will incorporate denied/noisy/partial information and that ISR will be important aspect (along with comms and logistics). Correct?

A10. Correct.

Q11. Section 1.1.5 (last paragraph) indicates that proposals should define a UI/UX. Summary bullets indicate that a Phase 1 Objective is to design a UI/UX. Please clarify what constitutes a sufficiently detailed design to satisfy deliverable requirements? Please verify that building the UI/UX is not a Phase 1 goal. When and by whom will the UI/UX be built?

A11. The BAA outlines program metrics and specifically does not define solution requirements. The Phase 1 objectives include executing the SCEPTER capability. The UI/UX is considered part of the SCEPTER capability.

Q12: Developing a COA for one side in a conflict necessarily involves reasoning about the possible COAs contemplated by the other side. Will the Government provide the adversarial COAs as part of the scenario? Or are performers expected to develop adversarial COAs concurrently with COAs for the side whose COAs we are developing?

A12: The first two paragraphs of section 1.1.5 describe the envisioned progression. See also the bulleted summary objectives for Phase 1 in the same section.

Q13: Will the proposal slide summary template be provided in PPT format? When trying to export out of a pdf, the slides are images and cannot be altered.

A13: A PPT template will not be provided. Proposers are free to use the style and format that best conveys the intended message. The images in Appendix 1 are provided to communicate the content that should be included in 3-slide summary.

Q14: The BAA does not indicate any specific travel/locations during program performance. Can DARPA provide any guidance w.r.t travel?

A14: Table 2 and the text immediately above Table 2 in Section 1.1.5 provides guidance.

Q15: Is it permissible to include CUI in the SOW?

A15: Yes.

Q16: (U) What are the blue arrows in Figure 2 (Page 9) of BAA?

Q16: Scenarios will be provided to performers as indicated by the blue diamonds for use in the Spirals identified by the blue arrows.

Q17: (U) With reference to Figure 2 (Page 9) of BAA, what do the scenario drops entail? What artifacts will be included in the government provided scenario?

A17: The “human defined” plans will be informed by the Government team and provided with the scenarios in a human readable format. Performers are responsible to encode the “human-defined” plans in the simulator proposed.

Q18: (CUI??) With reference to Table 3 (Page 13 of BAA), what is the definition of “COA Resiliency” contained in the Description of COA Quality in Table 3?

A18: The “COA Quality” metric is defined in section 1.2 under the heading of “COA Quality”. COA resiliency was offered as an example of a possible COA Quality metric. If a proposer elects to use this example, proposers are responsible to reify the definition sufficiently to unambiguously measure progress against this metric.

Q19: (U) The last bullet at the bottom of page 10 of BAA says “Develop teaming methods that enable modeled entities to execute distributed kill-chains in communications-denied environments.” Is DARPA referring to autonomy teaming between agents or human-machine teaming?

A19: This bullet refers to teaming between modeled entities.

Q20: Does DARPA have specific instructions for how subcontractors should email their proprietary proposals pursuant to section 4.2.1.2 “Volume II, Cost Proposal”?

A20: See section 4 of the BAA, specifically Section 4.2.3.1. Unclassified proposals, including proposals containing Controlled Unclassified Information and proprietary information may be submitted via DARPA’s BAA Website (https://baa.darpa.mil)

Q21: Is there any interest in APIs which allow for interacting with existing planning systems?

A21: Proposed solutions which enable the SCEPTER capability to interact with existing planning systems would be favorably received. Proposals should assume that reviewers have no knowledge of existing planning systems and should describe the merits of such systems and how these merits relate to SCEPTER if proposed.

Q22: May we include a Table of Contents and an acronym list that do not count against page limits of the Volume I, Technical and Management Proposal?

A22: Proposal format instructions are given in Section 4.2.1 of the BAA.

Q23: Can you clarify what the 3 parts are referring to in the first paragraph of section 4.2.1 (U) Proposal Format (“ Volume I, Technical and Management Proposal (composed of 3 parts)”)?

A23: Volume I should follow the guidance given in section 4.2.1.1. There are 6 sections under this heading, not 3. The parenthetical “composed of 3 parts” is in error.

Q24: Volume I, Technical and Management Proposal, Section VI: Additional Information, lists “no more than 12 pages” and also allows responders to include three relevant papers. Do the three papers all need to be less than 12 pages total including the bibliography or are the three papers outside of the 12 page limit?

A24: The entirety of Section VI, including any relevant papers, should not exceed 12 pages.

Q25: Is there a procedure to share the announcement for Scepter with non-US citizens who do not have a work visa or other form of residency?

A25: See Section 4.1.1 of the BAA.

https://baa.darpa.mil/

Q26: We incurred a delay on accessing program materials, will the proposal due date be extended to compensate for this lost time?

A26: According to our records, all requests for the CUI addendum and CUI guide were processed within the timeframe outlined in section 4.1.1 of the BAA when the requestors promptly responded to requests for proof of U.S. person documentation. We will not extend the proposal due date at this time.

Q27: Does each SCEPTER Proposal have to cover both FA-1 and FA-2 or is DARPA’s intent to allow FA-1 and FA-2 to be Proposed and executed separately?

A27: Compliant Phase 1 proposals will address both focus areas.

Q28: Several of the suggested simulation frameworks are not open-source. Does DARPA plan to enable and/or provide access to source code for all the cited simulation frameworks? What Government Furnished Information (GFI), Government Off The Shelf (GOTS) equipment, facility access, or similar agent-based M&S programs will provided with this effort?

A28: Proposers are expected to gain access to the simulation frameworks proposed. DARPA is willing to provide this sponsorship if required. Requests for all GFI/GFE should be documented in Volume I, Section IV.F of the proposals (see 4.2.1.1 of the BAA).

Q29: Will DARPA provide a copy of a style guide for detailed formatting requirements?

A29: No. Proposal format instructions are given in 4.2.1 of the BAA.

Q30: For pricing purposes, what is the anticipated start date for Phase 1?

A30: Proposers may assume program commencement in Q4 of FY22 (see Figure 2, Section 1.1.4).

Q31: The detailed cost breakdown is required to include itemization of major subcontracts. What is the definition of a major subcontract?

A31: A subcontract in excess of $750,000 is considered a major subcontract.

Q32: Is this more 6.2 level work or 6.3 level work?

A32: This program will be funded with 6.3 dollars.

Q33: Should we consider decentralized planning or just centralized planning? How many users does SCEPTER need to support? Would a simulation built at an aggregate level of resolution be acceptable for the program? Is resource allocation considered in-scope for the program? What analytical and visualization tools are expected / desired?

A33: Proposers are free to choose the architecture that they feel is most responsive to need described in the BAA, and the most compelling to the future potential user community.

Q34: Is there a human domain aspect to SCEPTER or is it all kinetic/military?

A34: Phase I scenarios will be focused on maneuver and electromagnetic warfare. See section 1.1.5.

Q35: Will scenarios be modified to be within the scope of the selected simulator or should we expect to update the simulation when it cannot cover the scenario scope?

A35: Scenarios will be designed to explore realistic needs of the warfighter and should not be significantly constrained by the limitations of a simulator. Existing simulators may need to be improved or enhanced to meet this need (See Section 1.1.5)

Q36: The simulations mentioned (e.g. NGTS, AFSIM) in the BAA require both tactical mission plans and tactic level behavior rules to execute a simulation. Will the government evaluation team convert plans developed by the SCEPTER agents into the mission & tactical plans needed by the simulations?

A36: The Government team will not convert performer generated plans into the simulation environment. Proposers are responsible for identifying military simulator environment(s) and supplementing their capabilities, if necessary, to ensure they are suitable to address all of SCEPTER’s needs.

Q37: How will tournament play be conducted in Phase 2? This is relevant to Phase 1 proposals because performers may want to select their Phase 1 simulators based on what will be used in Phase 2.

A37: All Phase 2 performers will be asked to support all simulation environments that successfully demonstrated SCEPTER metrics in Phase 1. In this way, all SCEPTER solutions will naturally be expanded to support multiple simulation environments in Phase 2.

Q38: The BAA states: "Develop teaming methods that enable modeled entities to execute distributed kill-chains in communications-denied environments." Can you please elaborate as to what more specifically is expected / desired here?

A38: State-of-art multi-entity teaming schemes rely heavily on explicit communications amongst team members for tasking, situational awareness, etc. SCEPTER scenarios will include adversarial electromagnetic attacks on communication. Proposers will need to clearly articulate how their solution will enable the discovery of teaming behaviors (COAs) in the presence of these attacks.

Q39: Can you elaborate how the teammate joint attention metric should be measured?

A39: The measurement of this and other metrics may depend on the solutions proposed, however it is acceptable to use logs or other explicit “after-the-fact” means to measure joint attention.

Q40: Where does the "unplanned objective" come from and when?

A40: An example of on an “unplanned objective” would be a target of opportunity.

Q41: The BAA states that it is anticipated that the performers will need HPC, and that the cost of the computational resources needed should be separated in the budget. The amount of computing needed depends on the three scenarios. How should we deal with that dependence given that the scenarios have not yet been provided?

A41: Proposers should size their computing needs to demonstrate scenarios consistent with the program metrics. See Table 3.

Q42: Is the cost spreadsheet required even for OTAs that are 100% milestone based and do not charge any dollars based on hours?

A42: Yes. Additionally, if an OT is used please provide a schedule for milestone payments that is in line with your proposed approach.

Q43: The BAA states that the total amount anticipated to be awarded is "$39M (Phases 1 and 2)".

Does this mean that the sum of the two phases is $39M or that each phase is $39M?

A43: The sum of the two phases.

Q44: Will computing be provided for the training of machine learning algorithms for our solution, or should they be included in the cost proposal?

A44: Proposals should detail the computing needs required for the proposed approach and include the costs/schedule of securing this environment. Because it may be mutually beneficial to share computing resources amongst SCEPTER performers and the Government team, computing and development platform costs should be costed separately to facilitate this potential resource sharing.

Q45: What domains (such as ground, aerial, aquatic, electronic) will the government's simulation software cover and evaluate?

A45: SCEPTER solutions should be able to address a wide variety of military scenarios, including a variety of domains (i.e. undersea, sea, land, air, and space), settings, terrain, force compositions and adversary objectives (known or unknown).

Q46: How will external evaluation of our solution be conducted?

A46: The Government team will conduct both quantitative and qualitative assessments. Code drops to the Government team will be used to evaluate the software usability and its performance. Military planning experts will evaluate the SCEPTER recommended COAs.