HR001120S0028-Amendment-01.pdf

Frequently Asked Questions (FAQ) Air Combat Evolution (ACE) Technical Area 1 (TA-1) Broad Agency Announcement (BAA) HR001120S0028

Amendment 01 4/15/2020

The purpose of this amendment is to publish answers to Frequently Asked Questions (FAQs) submitted against the Defense Advanced Research Projects Agency (DARPA) BAA HR001120S0028, Air Combat Evolution (ACE) Technical Area 1. Answers will also be posted on the DARPA Opportunities webpage for this solicitation found at: http://www.darpa.mil/work-with-us/opportunities. This amendment also incorporates Appendix 5, titled “ASSOCIATE CONTRACTOR AGREEMENT (ACA)” previously omitted from the original posting of

HR001120S0028.

Programmatic Questions

Program Schedule

Q1: The Figure 8 shows a total duration of 40-41 months depending on how the partial quarters are read. The text gives a total duration of 44 months. How many months should we plan for?

A1: Figure 8 is intended to help visualize the overall program flow and the text should be used to inform the proposal. Proposers should plan for a duration of 44 months for the three program phases in total.

Q2: When should we plan on the effort starting?

A2: For estimation purposes only, please assume a start date of October 1, 2020.

Q3: Does the contract start date of other TA performers impact the start date of TA-1?

A3: No, the contract start date of other TA performers will not impact the start date of TA-1 activities.

Q4: Why is there a one-month gap in between Phases 1 and 2 in the BAA? Is this a mistake or part of the plan?

A4: There is not expected to be a break in work between Phases 1 and 2.

Program Funding

Q5: Lockheed Martin received an award on ACE for approximately $10M. Was the $10M award on HR001119S0051 part of the $25M budget for ACE-TA-1? We are determining our budget for the effort and want to estimate correctly.

A5: No. Awards made to TA-2, TA-3, and TA-4 performers are not part of the $25M budget for

ACE TA-1.

Q6: The solicitation states "Total amount anticipated to be awarded – Up to $25 million".

Assuming this statement is in reference to TA-1 alone, do you have an idea as to the approximate range of expected individual TA-1 award sizes?

http://www.darpa.mil/work-with-us/opportunities http://www.darpa.mil/work-with-us/opportunities

A6: As stated in the BAA in the Part I: Overview Information section, there are multiple TA-1 awards anticipated, and the total amount to be awarded is up to $25 million. The total number of TA-1 awards and level of funding for each TA-1 award will be dependent on the proposals received.

Q7: Are DARPA’s payments to the prime contractor based on completion of proposed deliverables to DARPA, based on the monthly financial status reports or something different?

A7: Payment terms are dependent upon award instrument type. For this procurement, the award will be either a procurement contract or other transaction.

Program Milestones and Deliverables

Q8: Does DARPA plan to down-select the number of performers after the scheduled evaluations?

A8: Selected proposals for negotiation of a potential award will be awarded Base contracts for Phase 1 and Options for Phases 2 and 3 respectively. The Government will have the unilateral right to exercise or not exercise options.

Q9: To inform travel estimates, what are the expected locations and durations of Technical Data Reviews (TDRs), Quarterly Program Reviews (QPRs), integration quarterlies, evaluation events, and flight tests?

A9: Each of these events is estimated to last no more than one week, and most events will take place at DARPA, Johns Hopkins University/Applied Physics Laboratory (JHU/APL), or military test ranges.

Q10: What is the expected length of the Integration Quarterlies in each phase? Where are they to be held? Do the Integration Quarterlies occur together with the Experimentation Integration Team (EIT) and the respective TA awardees, or do we have separate meetings with each?

A10: The duration, location, and participation at each Integration Quarterly will vary based on specific objectives. Duration is expected to range from one work day to one work week. Some will be virtual, while others will be hosted at JHU/APL or a performer site. Some will involve only the TA-1 performer and the EIT, while others will involve other performer organizations from one or more TA.

Q11: What is a QPR? Where are the QPRs held? Who attends the QPR? What are the expected inputs and outputs of the QPR?

A11: Quarterly Program Reviews will occur periodically throughout the three program phases and will be structured as evaluation events in which each TA-1 performer’s algorithm will be competed against a combination of EIT-developed algorithms, algorithms developed by other performers, and human pilots. QPRs are likely to be held in the Washington, DC area, and attendees will include representatives from the ACE performer teams, DARPA, and the EIT.

Q12: Please explain what the TDR is and who will be expected to prepare the TDR information?

A12: Each Technical Data Review (TDR) is “designed to provide the necessary evidence to support evaluation of progress against program metrics and transition into the next phase”. This data package will be prepared primarily by the TA-1 performer and presented to DARPA.

Q13: What is the duration of the “Modeling & Simulation (M&S) Demonstration Event” and where will it be held?

A13: The M&S Demonstration Event during Phase 1 will likely be held at JHU/APL in Laurel, MD and is expected to last approximately one week.

Q14: What is the duration of the “Phase 2 Eval” and “Subscale Demonstration Events” events and where will they be held?

A14: The Phase 2 evaluation events will likely be held at a military test range to be disclosed at a later date, and the duration of each will be approximately one week.

Q15: What is the duration of each of the “Phase 3 Eval” events, including the “Final Demonstration” event and where will they be held?

A15: The Phase 3 evaluation events will likely be held at a military test range to be disclosed at a later date, and the duration of each will be approximately one week.

Q16: The BAA states that travel is anticipated for the Kickoff meeting and regular program-wide PI meetings. Are these PI meetings in addition to the Integration and Evaluation events, or synchronous with them? Should we anticipate all travel to the Washington, DC area or to other locations for pricing purposes? How many days are the evaluation events anticipated to last?

A16: PI meetings are expected to occur synchronous with integration and evaluation events. For cost estimating purposes, proposers should assume that each evaluation event will last approximately one week and take place at DARPA, JHU/APL, or military test ranges, as applicable.

Q17: Our interpretation of Figure 8 is that Phase 2 includes one sub-scale integration and competition event (1v1) for and Phase 3 includes another two sub-scale integration events (2v1, 2v2), plus the three full-scale integration and competition events, for a total of six events. Is that correct?

A17: Proposers should develop a preliminary test plan as a part of their proposal which should include test objectives and then number and sequence of required events.

Q18: In reference to the statement “Define all deliverables (reporting, data, reports, software, etc.) to be provided to the Government in support of the proposed research tasks/activities”, would the Government please provide a specific list of deliverables that vendors should provide?

A18: Proposers should define what they are offering to the Government as a part of their proposal, including a statement of data rights associated with the deliverables.

Q19: Will performers’ agents compete directly against one another at any of the demonstration events (i.e., will one TA-1 performer’s 1v1 agent fight a dogfight against another performer’s agent)? If so, at what events and in what environments?

A19: Yes, TA-1 performers should be prepared to compete their algorithms against a combination of algorithms developed by the EIT, algorithms developed by other TA-1 performers, and human pilots at any of the evaluation events during each of the three program phases.

Proposal Content, Length, and Format

Q20: Does the 40-page limit include Volume I (Technical and Management Proposal) and Volume II (Cost Proposal)?

A20: The 40-page limit includes only Volume I (Technical and Management Proposal) see section IV.B.1 on page 26.

Technical Volume

Q21: What is the recommended page length for each section? The BAA instructions state "Bracketed numbers before each section denote recommended page limits", however no brackets appear in the suggested proposal outline.

A21: The total length of Volume I, the Technical and Management Proposal, should not exceed 40 pages (see section IV.B.1 on page 26). Proposers should use their judgement in determining how many pages to allot for each required section of the Technical Volume without exceeding the total page limit. There is no page limitation for Volume II, the Cost Proposal.

Q22: If teaming agreements are required, will the Government allow teaming agreements to be excluded from the page count?

A22: Teaming agreements are included within the page count.

Q23: Are references to published papers allowed? Are they included in the page count?

A23: Yes, references to published papers are allowed and the references will be included in the page count if they appear in Volume I.

Q24: Can we submit videos as part of our submission, and if so, what is the preferred method? Do videos have to be submitted via web link where videos must be approved for public release?

A24: Proposals are encouraged to include whatever is relevant to make the technical point that is being addressed including figures, tables, and video (if desired it may must be included as a link embedded in the proposal and does not have to be approved for public release). Furthermore, proposers can choose to submit videos if they desire, however, any supplemental information shall not take the place of the required written proposal. Proposers are responsible for ensuring that any videos, links etc. meet all DoD required standards for security and are uploaded properly.

Q25: Please indicate which, if any, of the pages in the technical volume are excluded from the page count; Are the cover page, letter of commitment, and SOW included in the page count?

(The exclusion of the summary slides and the forms for intellectual property are indicated).

A25: All content besides the summary slides and intellectual property forms are included in the page count, including the cover sheet, letter of commitment, and SOW (see section IV.B.1 on page 26).

Q26: Please provide more detail on the difference between the two parts at the end of Volume I, Section III – Part I: Description of milestone cost and accomplishments and Part J: Cost schedules and measurable milestones for the proposed research.

A26: Part I is intended to include the costs specifically associated with the completion of program milestone events, as well as the accomplishments associated with each milestone. Part J is intended to include a summary-level cost schedule for the full work package.

Q27: Please clarify what information is required for Volume I, Section III, Part I: Description of milestone cost and accomplishments. How does this information differ from the information requested in the Volume II, Cost Proposal? Why is the Government requiring that cost information be provided in Volume I? Does this break down the SOW tasks by cost or are these higher level suggested payment milestones?

A27: Part I is intended to include the costs specifically associated with the completion of program milestone events, as well as the accomplishments associated with each milestone.

Cost Volume

Q28: What is expected in the Volume II, Section 2a response that is different from the detailed cost breakdown (Section 2b)? Can more explicit instructions be provided?

A28: Volume II, Section 2a is intended to include information (e.g., Basis of Estimates (BOE)) that substantiates the cost breakdown information provided in Section 2b. See Section IV.B.1.b).(2).(a) on page 30.

Q29: For Cost Volume, cover sheet, item(s): What information is required for sub-awardees?

A29: On the cover sheet of the Cost Proposal, Sub-awardee Information should include the names of any sub-awardee organizations and references to any sub-awardee Cost Proposal content not included directly in the prime contractor’s submission (see Appendix 3 page 44).

Q30: Can references to an appended spreadsheet be used in Cost Volume, or does information need to be in-line with the narrative?

A30: References to an appended spreadsheet can be used in the Cost Volume.

Subcontracts

Q31: Can subcontractors submit their proprietary cost volume directly through the DARPA portal?

A31: Yes, separate, direct cost volume submission from a subcontractor is allowed as long as the volume is conspicuously associated with the prime contractor’s submission (see Section IV.B.3 on page 35).

Q32: While the primary contract will not be fundamental research, is it allowable to include subcontract effort that would be considered fundamental research if this work is properly shielded from disclosure of military systems information? Reference: “Fundamental Research”, page 23. “As of the date of publication of this BAA, the Government expects that program goals as described herein either cannot be met by proposers intending to perform fundamental research or the proposed research is anticipated to present a high likelihood of disclosing performance characteristics of military systems or manufacturing technologies that are unique and critical to defense.”

A32: Any proposed compartmentalization of fundamental research for performance by a subcontractor will be reviewed on a case-by-case basis. Based on our initial assessment and IAW the BAA we cannot guarantee that such an arrangement will be permissible on this program without better understanding of the specific request (see Section II.B on page 23).

Proposer’s Day

Q33: Will the Government please provide a list of the Proposer’s Day attendees?

A33: Yes, the list of attendees and their contact information has now been made available.

Q34: The Proposer’s Day presentations contain a lot of valuable information that will help us focus our proposal. Will these be made available to us?

A34: The proposer’s day slides have been posted to the DARPA website and can be found here but are provided to augment the BAA which should be considered the primary reference document for all proposals:

http://www.darpa.mil/attachments/20200326_ACE_ProposersDayTA-1v2.pdf Please note that if there is a discrepancy between the slides and the BAA, the BAA takes precedence.

Program Security

Q35: Do you anticipate all phases of the effort TA-1 to present a high likelihood of disclosing performance characteristics of military systems or manufacturing technologies that are unique and critical to defense, or only later phases? Do you anticipate each phase to fall under International Traffic in Arms Regulations (ITAR)?

A35: TA-1 proposers should plan for all program phases to generate ITAR-restricted content, and that no program phase will exclusively contain content characterized by Distribution Statement A.

Technical Questions

Aircraft Model and Simulation Environment

Q36: Will JSBSim with FlightGear for visualization be the only simulation environment used on the program? In addition, will there be the possible use of Advanced Framework for Simulation, Integration, and Modeling (AFSIM) or Next-Generation Threat System (NGTS)?

A36: JSBSim is expected to be the primary simulation environment for TA-1 activities. Other simulation toolsets will serve other ACE activities and may ultimately be integrated with TA-1 algorithms and/or the TA-1 simulation environment.

Q37: Should we anticipate/plan for a stochastic environment modeled within JSBSim (e.g., variable wind conditions, cloud cover)?

A37: Stochastic elements may be introduced to the simulation environment to either improve its representation of the conditions expected to be encountered in live flight test (e.g., variable winds) or to implement effects required to achieve particular program objectives (e.g., algorithm robustness to partial observability).

http://www.darpa.mil/attachments/20200326_ACE_ProposersDayTA1v2.pdf

Q38: Will the model of the vehicle include variability in the vehicle parameters (aerodynamic, mass properties, propulsion, etc.)

A38: There will be multiple aircraft models featured in the TA-1 simulation environment, one representing each of the sub- and full-scale live aircraft to be tested. There are currently no plans to introduce variation in these aircraft models.

Q39: How will vehicle attributes be characterized for insertion into autonomy system? Should changes to external stores & fuel burn be taken into account?

A39: Aircraft mass properties will be represented in each aircraft model. Mass properties will vary with time as fuel is burned and as external stores separate from the aircraft.

Q40: How will weapon effects be handled in JSBSim? Standard weapons to be provided by EIT, implicitly through Gym reward function?

A40: Weapons effects will be implicitly modeled using unclassified weapons employment zones, which will be the basis for the TA-1 scoring implementation.

Q41: Signature management could be important in policy development for guided weapon cases.

Will sensor modeling be necessary/possible?

A41: We do not currently plan to explicitly model aircraft signatures but may achieve the effect of this by varying track quality with relative aircraft positions.

Q42: How much pilot representation is needed? Is weapon and target selection conducted by the pilot, then the AI takes over fully? Must weapon release be confirmed by the operator at time of launch?

A42: TA-1 performers should assume that a pilot is not required for either weapon/target pairing or weapon release confirmation during TA-1 evaluation events. Trust Assessment Evaluations will require bidirectional control handoffs between AI and human pilot, such that the pilot under test will be able to take control of the aircraft from the TA-1 algorithm and pass it back at their discretion.

Q43: Is the common sim environment expected to provide simulations that emulate the actual interfaces of the aircraft or simply model the logical interfaces?

A43: At a minimum, the TA-1 simulation environment will emulate the live aircraft’s logical interfaces. Other attributes of the aircraft interfaces, including format and network implementation, may also be emulated as required.

Q44: The agent interface described in the BAA employs a synchronous control mechanism where the agent calls the simulation with an action and receives a new state in response. It is unclear how this will be extended to the team (2v1 & 2v2) case where there are two agents. Can you please provide information describing how the interface will be modified to accommodate teaming scenarios?

A44: The agent interface described in the Modeling and Simulation Environment Overview will serve 1v1, 2v1, and 2v2 scenarios in Phase 1. At each time step, each of the “Red” and “Blue” agents will resolve an action specifying the control inputs to the one or two aircraft within their force structure, and these control inputs will be applied to the corresponding aircraft model(s) prior to the simulation stepping each of the flight dynamics models.

Scenario Configurations and State Data

Q45: What are the terminal/win conditions for 1v1, 2v1, and 2v2?

A45: Each engagement will end when the mission objective is achieved (e.g., total attrition of either the Red or Blue force structure), a boundary condition is violated (e.g., exceeding a maximum permissible range separation between aircraft) or when a timeout condition is met.

Achieving total attrition of the opposing force qualifies as a “win” condition, and a scoring implementation will enable capture of more resolute performance metrics.

Q46: In 2 v 1 or 2 v 2 scenarios, should we expect each side consists of the same aircraft type?

A46: Yes, while engagements may include a mix of live, virtual, and constructive assets, each engagement is expected to be homogeneous in aircraft type.

Q47: Will we have accurate identification of all other aircraft in the scenario, for example IFF information?

A47: Yes, TA-1 performers should expect accurate identification information but imperfect track information.

Q48: In the 2v1 and 2v2 scenarios, should we assume that both friendly aircraft are operating under autonomous control?

A48: Each scenario configuration featuring a multi-agent force structure will specify whether each aircraft is manned or unmanned.

Q49: In the teaming scenarios (2v1 & 2v2) during sub-scale and full-scale demonstrations will the agents on the two UAV’s receive identical state data, or will noise or sensor models lead to the two agents receiving non-identical state vectors?

A49: There is a possibility that the two aircraft in live flight teaming scenarios will receive non-identical state information.

Q49: What are the communication links/implementation between the two agents in the 2v1 and 2v2 scenarios when these are running on separate aircraft?

A49: The two agents will share information via an Radio Frequency (RF) communications link.

The details of this communications implementation will be resolved by the TA-4 performer.

Q50: The BAA mentions artificial sensor degradation such that certain properties are not available or noisy. Will these sensor outputs already be filtered or should that filtering be performed by the TA-1 performer? If it is the former, will we have access to covariance state information? If the latter, will the sensor outputs be modeled using radar limitations such as field of view and Doppler-only velocity outputs or will it just be Gaussian noise? If using radar limitations, will the radar providing sensing be a ground or air based radar?

A50: Aircraft sensors will be modeled at an effects level, and the processes used to simulate those effects will be made available to TA-1 performers.

Q51: What type of noise model should we expect for state data? Is the model intended to reflect and be representative of platform sensors including accuracy/error models, field of view, and probability of detect? If so, what types of sensors are being modeled?

A51: Aircraft sensors will be modeled at an effects level, and the processes used to simulate those effects will be made available to TA-1 performers.

Q52: Will state data include measures of uncertainty, such as covariance or track quality?

A52: There will be some indication of detection or track quality included in the state vector in scenarios where state information is imperfect.

Hardware and Interface Specifications

Q53: Will the common sim environment provided by EIT to the TA-1 performer include mission computing hardware representative of the sub-scale and full-scale platforms? If not, will the TA- 4 performer provide computer hardware to the TA-1 performer?

A53: TA-1 performers should expect the EIT and TA-4 performer to provide specification for the mission computing hardware in the sub- and full-scale aircraft, respectively. The EIT and TA-4 performer will maintain sub- and full-scale aircraft mission computing hardware at their sites in support of integration events but will not be distributing hardware to TA-1 performers.

Q54: What operating environment or environments (e.g., hardware, operating system) will the agents run in? It would be helpful to be as specific as possible including versions, etc. in order to minimize integration risk.

A54: Hardware and software configurations will be provided to selected TA-1 performers by program phase and for each evaluation event. Proposers should expect to use Ubuntu 18.04 for Phase 1 algorithm integration and evaluations.

Q55: Are any agent-to-system (e.g., Flight Control System (FCS)) interface definitions (e.g., Robot Operating System (ROS), Future Airborne Capability Environment (FACE), Standardization Agreement (STANAG), Ethernet) nailed down?

A55: This information will be made available to selected TA-1 performers during program execution.

Q56: Will the state space be defined using a standard protocol (e.g., Open Mission Systems (OMS), Universal Command and Control Interface (UCI), STANAG 485)?

A56: This information will be made available to selected TA-1 performers during program execution.

Technical Approach

Q57: Will there be an Machine Learning (ML)-specific focus in the ACE program as Artificial Intelligence (AI) algorithms or is it open to other AI paradigms that can also achieve the objectives in TA-1 and ACE in general?

A57: The ACE TA-1 solicitation is open to all technical approaches that proposers believe will meet the TA-1 and ACE program objectives.

Q58: What are the Phase 2 & Phase 3 inference hardware constraints? L-39 & F-16 hardware is very limited compared to today’s ML training hardware, same with sub-scale. Is this defined by TA-4 performer?

A58: The flight hardware specifications for Phases 2 and 3 will be resolved by the EIT and the TA-4 performer, respectively, and will be made available after TA-1 contracts are awarded.

Q59: What is the major difference between the approaches in the AlphaDogfight Trials and TA- 1?

A59: The AlphaDogfight Trials feature exclusively 1v1 constructive engagements with full-scale aircraft models and perfect state information. TA-1 will build upon the AlphaDogfight Trials by extending scenario configurations to 2v1 and 2v2 force structures, introducing imperfect state information, and deploying air combat maneuvering algorithms on live aircraft.

Algorithm Verification and Safety Assurance

Q60: Will we be given explicit constraints (such as stalling/leaving flight envelope) that the AI is expected to not violate?

A60: Yes, such constraints will be determined during program execution in collaboration with the EIT and TA-4 performer.

Q61: Will the sub-scale and/or full scale aircraft be provided with any safety monitoring system such as Safe Testing of Autonomy in Complex, Interactive Environments (TACE)? If so, what requirements will that levy on the AI Agents to maintain safety?

A61: Yes, both the sub-scale and full-scale aircraft will be instrumented with safety monitoring systems. These safety monitoring systems are expected to be implementations of the Test Resource Management Center’s Safe Testing of Autonomy in Complex, Interactive Environments (TACE). The requirements levied on AI agents in order to integrate with TACE and maintain safe operations will be resolved during program execution.

Q62: Will DARPA or the EIT assist with developing formal specifications that the algorithms should be tested against, or black swan verification scenarios to prove robustness to rare scenarios – or is the TA-1 provider intended to demonstrate these capabilities independently as much as possible within budget?

A62: TA-1 performers are expected to demonstrate algorithm robustness and performance capabilities in rare scenarios as much as possible.

Q63: What are the TA-1 Verification & Validation (V&V)-type testing and model validation responsibilities for all three phases?

A63: TA-1 performers are responsible for delivering algorithms that integrate with the applicable experimentation infrastructure (e.g., M&S environment, sub-scale aircraft testbed, full-scale aircraft testbed) and meet the requirements for maintaining safe test operations. These requirements will be provided during program execution. The TA-1 performer is responsible for determining the extent of V&V testing and model validation that must be performed to meet these requirements.

Cross-TA Interactions

Q64: Can we contact, collaborate and meet with other TA awardees (TA 2, 3, 4) directly and outside of the Integration Quarterlies, or are we only permitted to communicate with them via EIT at the Integration Quarterlies?

A64: Yes, direct collaboration with other TA performers outside of the Integration Quarterlies may be required and should be executed in accordance with the terms of the Associate Contractor Agreement.

Q65: Would the Government please provide Appendix 5? (Reference: “To ensure this close collaboration across the TAs, each contractor will be required to execute associate contractor agreements as defined in Appendix 5.”

A65: Appendix 5 is incorporated with this amendment to HR001120S0028.

Q66: Are any of the other TA performers on contract yet? Knowing who those are could be very useful.

A66: TA-3 contracts have been awarded to Lockheed Martin and Dynetics. TA-2 and TA-4 contracts have not yet been awarded.

Q67: Given the importance of TA-2-4 to TA-1, can we get more information about the TA-2-4 approaches?

A67: HR001120S0028 Appendix 4 contains additional information on the TA-2-4 efforts. The details of each performer’s approach will be made available following TA-1 contract awards, as applicable.

TA-2 Interactions

Q68: If a TA-1 provider’s specific AI approach has unique benefits with respect to Human- Machine Interface (HMI) capabilities, will there be any ability to influence the TA-2 provider?

Or, will the interaction between TA-1 and TA-2 mostly be specific requirements sent from TA-2 to TA-1?

A68: TA-1 performers are expected to have some opportunity to influence HMI design and implementation activities led by the TA-2 performer. The EIT will act as a mediator and consider the equities of all performers involved in this collaboration.

Q69: In considering the interface between our autonomy and the human pilot, what control or influence do we have over the specific modes of interface, for example auditory vs Heads-Up Display (HUD)? Can you provide more details on the interface with the TA-2 performer in this technical area?

A69: TA-1 performers are expected to have the opportunity to present the merits of particular interface modalities to the TA-2 performer and the EIT. TA-1 and TA-2 performers will collaborate to design and implement an interface between the TA-1-developed combat autonomy and TA-2-devleoped HMI and trust assessment infrastructure that: (1) provides the human pilot with any information believed to be pertinent, (2) provides the trust assessment infrastructure with information required to measure trust, and (3) enables bidirectional control handoffs between the combat autonomy and the human pilot at the pilot’s discretion.

Q70: What type of information are you expecting TA-1 performers to supply to TA-2 performers (especially during flights)? This seems like a key interface for safety and pilot situational awareness.

A70: Proposers should specify what information they believe should be made available to the pilot to enable him or her to maintain situational awareness and determine the suitability of the algorithm for particular combat situations. The EIT will facilitate interactions between TA-1 and TA-2 performers once on contract.

TA-4 Interactions

Q71: Will TA-4 provide a high-fidelity sim against which TA-1 can validate the robustness and transferability of its agents?

A71: Additional information regarding high-fidelity simulation environments for the full-scale live aircraft to be flown in Phase 3 will be made available to TA-1 performers during program execution.

Q72: Does the TA-4 performer allow us to have agents that run on different threads?

A72: TA-1 performers can expect full-scale aircraft hardware specification information from the TA-4 performer during program execution.

Q73: Will the TA-4 performer provide throughput and memory allocation for our agents?

A73: TA-1 performers can expect full-scale aircraft hardware specification information from the TA-4 performer during program execution.

Aircraft Integration and Test Operations

Q74: Will live/real sensor data be provided to the agents in the sub-scale and/or full-scale exercises?

A74: The current expectation for state information to be provided to the agents during sub-scale and full-scale live flight exercises is tabulated in HR001120S0028 Table 1.

Q75: The agent interface described in the BAA employs a synchronous control mechanism where the agent calls the simulation with an action and receives a new state in response. This does not sound viable as an interface for live aircraft where such synchronization cannot be guaranteed. What changes in the interface are anticipated for the sub-scale and full-scale aircraft integration?

A75: The synchronous control mechanism described in the Modeling and Simulation Environment Overview refers to the synchronization of stepping each of the aircraft’s flight dynamics models forward in time. This synchronization is required to ensure that simulation time increments at the same rate in each of the flight dynamics models and that each agent has the opportunity to submit a set of control inputs at the same frequency. In live aircraft implementations, simulation time will align with real time (i.e., wall clock time) and hence need not be explicitly managed to increment synchronously for all aircraft, but there remains a requirement to limit the frequency at which each agent is allowed to submit a set of control inputs to the aircraft. There will be a target update rate for state information fed to the agents and a maximum frequency at which actions can be submitted to the aircraft control system.

Q76: How many practice flight test events will the TA-1 performer be permitted prior to each competition event? For cost estimating purposes, what are the guidelines for number of trips we should assume including durations for Phase 2 and 3?

A76: Proposers should assume approximately one test per month during Phases 2 and 3, with each test event expected to last approximately one week.

Sub-Scale Aircraft

Q77: What is your anticipated work breakdown for sub-scale testing between EIT and the TA-1 Performer? Is the TA-1 performer primarily responsible for integration of the TA-1 algorithms onto the sub-scale platform? Or is the TA-1 performer handing off software to EIT, with EIT primarily responsible for integration of the algorithms and performing in-flight testing?

A77: The EIT is responsible for procuring the sub-scale aircraft and flight hardware, distributing the flight hardware specification and agent-system interface specification to performers, and coordinating sub-scale aircraft flight test activities. TA-1 performers will collaborate with the EIT to integrate their algorithms with the sub-scale aircraft flight hardware and are responsible for ensuring that their algorithms operate as intended during all sub-scale flight test events.

Q78: How many sub-scale aircraft will be made available for each Phase 2 performer? Will the communications between aircraft also be provided?

A78: There will be multiple aircraft made available for sub-scale flight test operations and for hardware-in-the-loop ground testing at JHU/APL. The exact number of aircraft to be made available is not known at this time. A description of the inter-aircraft communications implementation will be made available to TA-1 performers during program execution.

Q79: What types of data reports and data rates can the TA-1 performers expect from the sub-scale aircraft sensor(s)? For example, will sensor(s) report detected vehicles in absolute positions (i.e. latitude/longitude/altitude) or via relative angular measurements (azimuth/elevation/range)?

A79: Refer to Table 1 of the BAA for the current expectation of the state information to be provided to TA-1 algorithms during sub-scale aircraft experimentation. Additional information on sub-scale aircraft instrumentation such as update rates for state information will be made available to TA-1 performers during program execution.

Q80: Will software be provided for control and reporting from the sub-scale sensors? If so, what form – source code (for re-hosting on different operating systems) or compiled executables?

A80: We do not currently plan to provide software for sensor control and reporting.

Q81: Which group (EIT or which TA# performer) is responsible for providing the sensor(s) for sub-scale aircraft?

A81: The EIT is responsible for sensory instrumentation of the sub-scale aircraft.

Full-Scale Aircraft

Q82: What is your anticipated work breakdown for full-scale testing between the TA-4 performer and the TA-1 Performer? Is the TA-1 performer primarily responsible for integration of the TA-1 algorithms onto the full-scale platform? Or is the TA-1 performer handing off software to TA-4, with TA-4 primarily responsible for integration of the algorithms and performing in-flight testing?

A82: The TA-4 performer is responsible for procuring the full-scale aircraft and flight hardware, distributing the flight hardware specification and agent-system interface specification to performers, and coordinating full-scale aircraft flight test activities. The EIT will facilitate collaboration between the TA-1 and TA-4 performers to integrate their algorithms with the full-scale aircraft flight hardware and are responsible for ensuring that their algorithms operate as intended during all full-scale flight test events.

Q83: What types of data reports and data rates can the TA-1 performers expect from the full-scale aircraft sensor(s)? For example, will sensor(s) report detected vehicles in absolute positions (i.e. latitude/longitude/height) or via relative angular measurements (azimuth/elevation/range)?

A83: Refer to Table 1 of the BAA for the current expectation of the state information to be provided to TA-1 algorithms during full-scale aircraft experimentation. Additional information on full-scale aircraft instrumentation such as update rates for state information will be made available to TA-1 performers during program execution.

Q84: Will software be provided for control and reporting from the full-scale sensors? If so, what form - source code (for re-hosting on different operating systems) or compiled executables?

A84: We do not currently plan to provide software for sensor control and reporting.

Q85: Which group (EIT or which TA# performer) is responsible for providing the sensor(s) for full-scale aircraft?

A85: The TA-4 performer is responsible for sensory instrumentation of the full-scale aircraft.

Performance Metrics and Evaluation

Q86: What is the MVP (minimum viable product) for TA-1? What is the selection criteria for advancement to the next phase?

A86: Current expectations for algorithm performance are described in Section I.B of the BAA, “Program Metrics”. There are no selection criteria for advancement into the next phase rather they are options that may be executed.

Q87: Can additional simulation capability expansion occur during Phases 2 & 3?

A87: Yes, proposers are can expect to continue advancing the performance of their algorithms throughout the program including if the options on their contract or executed.

Q88: Will there be program metrics beyond the ones listed in Figure 10 (Pwin)? For example, are the items in the technical description list ("Evaluation Events" Section 5.B) program requirements or nice-to-haves?

A88: Section I.B of the BAA, “Program Metrics”, identifies a primary performance metric for TA-1 activities.

Q89: What is relative value of explainability versus performance?

A89: We expect proposers to present a solution for balancing explainability and performance in a manner that accomplishes the program objectives and builds operator trust.

XIII. APPENDIX 5: ASSOCIATE CONTRACTOR AGREEMENT (ACA)

Language similar to that below will be included in contract and agreement awards against HR001120S0028.

(a) It is recognized that success of the ACE research effort depends in part upon the open exchange of information between the various Associate Contractors involved in the effort.

This language is intended to ensure that there will be appropriate coordination and integration of work by the Associate Contractors to achieve complete compatibility and to prevent unnecessary duplication of effort. By executing this contract, the Contractor assumes the responsibilities of an Associate Contractor. For the purpose of this ACA, the term Contractor includes subsidiaries, affiliates, and organizations under the control of the contractor (e.g.

subcontractors).

(b) Work under this contract may involve access to proprietary or confidential data from an Associate Contractor. To the extent that such data is received by the Contractor from any Associate Contractor for the performance of this contract, the Contractor hereby agrees that any proprietary information received shall remain the property of the Associate Contractor and shall be used solely for the purpose of the ACE research effort. Only that information which is received from another contractor in writing and which is clearly identified as proprietary or confidential shall be protected in accordance with this provision. The obligation to retain such information in confidence will be satisfied if the Contractor receiving such information utilizes the same controls as it employs to avoid disclosure, publication, or dissemination of its own proprietary information. The receiving Contractor agrees to hold such information in confidence as provided herein so long as such information is of a proprietary/confidential or limited rights nature.

(c) The Contractor hereby agrees to closely cooperate as an Associate Contractor with the other Associate Contractors on this research effort. This involves as a minimum: (1) maintenance of a close liaison and working relationship; (2) maintenance of a free and open information network with all Government-identified associate Contractors; (3) delineation of detailed interface responsibilities; (4) entering into a written agreement with the other Associate Contractors setting forth the substance and procedures relating to the foregoing, and promptly providing the Agreements Officer/Procuring Contracting Officer with a copy of same; and, (5) receipt of proprietary information from the Associate Contractor and transmittal of Contractor proprietary information to the Associate Contractors subject to any applicable proprietary information exchange agreements between associate contractors when, in either case, those actions are necessary for the performance of either.

(d) In the event that the Contractor and the Associate Contractor are unable to agree upon any such interface matter of substance, or if the technical data identified is not provided as scheduled, the Contractor shall promptly notify the DARPA ACE Program Manager. The Government will determine the appropriate corrective action and will issue guidance to the affected Contractor.

(e) The Contractor agrees to insert in all subcontracts hereunder which require access to proprietary information belonging to the Associate Contractor, a provision which shall conform substantially to the language of this ACA, including this paragraph (e).

(f) Associate Contractors for the ACE research effort include:

Contractor Technical Area