HR001119S0051-Amendment-03.pdf

Amendment 03 to DARPA BAA HR001119S0051, Air Combat Evolution (ACE)

The purpose of this amendment is to publish answers to Frequently Asked Questions (FAQs) submitted against DARPA BAA HR001119S0051, Air Combat Evolution (ACE). Answers will also be posted on the Opportunities page for this solicitation on DARPA.mil (http://www.darpa.mil/work-with-us/opportunities).

Acronyms:

ACE: Air Combat Evolution ACMI: Air Combat Maneuvering Instrumentation AGCAS: Automatic Ground-Collision Avoidance System API: Application Programming Interface ARCNet: Autonomy Research Collaboration Network ASR: Aircraft Selection Review BAA: Broad Agency Announcement CMO: Contracts Management Office DARPA: Defense Advanced Research Projects Agency DOT: Dual Operational Task EIT: Engineering Integration Team FAA: Federal Aviation Association GFE: Government Furnished Equipment GFI: Government Furnished Information HMI: Human Machine Interface HRPO: Human Research Protection Office HSR: Human Subjects Research ICD: Interface Control Document IRB: Institutional Review Board ITAR: International Traffic in Arms Regulations LFE: Large Force Exercise LVC: Live Virtual Constructive MC: Mission Commander M&S: Modeling and Simulation MUMT: Manned-Unmanned Teaming QPR: Quarterly Program Review RPP: Request for Project Proposal SOW: Statement of Work STO: Strategic Technology Office TA: Technical Area TACE: Safe Testing of Autonomy in Complex Interactive Environments TSPI: Time Space Position Information TAE: Trust Assessment Event UAS: Unmanned Aerial System UAV: Unmanned Aerial Vehicle WEZ: Weapon Engagement Zone

WVR: Within Visual Range

General

1. Q: What is the overall Budget for the TA234 BAA?

A: 63.6 million dollars.

2. Q: Is it safe to assume that approximately half of the TA234 BAA budget will go to TA4?

A: Given that TA4 involves building/modifying full scale aircraft, it is safe to assume that a significant portion of the budget will be designated for that TA.

3. Q: The last sentence on Page 40 of the BAA says, “See Section IV.B.4.b. of this BAA”.

Should that instead say, “See Section IV.B.3.a. of this BAA”?

A: Yes, the last sentence on page 40 of the BAA should read, “See Section IV.B.3.a. of this BAA”.

4. Q: In the latest BAA Amendment, under “Section II. Summary of Proposal” (page 36) there is a parenthetical that states that “this section should be supplemented by a more detailed plan in Section III of the Technical and Management Proposal”. In reviewing the Section III outline, it is unclear where this information best belongs. Many STO BAA’s have a specific subsection in Section III for a detailed technical approach to enhance and expand the information in the Summary section, but that doesn’t seem to be included.

Can you please clarify which section should include the detailed technical approach and plan?

A: This discussion belongs in Section III B. See page 38 of the BAA.

5. Q: Please provide estimated delivery dates for the following GFE/GFI:

a. WVR air combat simulation environment (TA1) and mission- level M&S environment (TA3)

b. Access, licensing, etc. for the selected TA1 and TA3 M&S environments

c. TA3 data set associated with BAA page 15, bullet 1

d. ICDs associated with TA1 algorithms to be provided to other TAs

A: Initial versions of the above GFE/GFI are estimated to be delivered at program kickoff. Subsequent versions of the software, data sets, and documentation may also be provided during program execution.

6. Q: Please provide estimated delivery dates for releases of TA1 algorithms to be provided to other TAs.

A: Assuming this refers to TA1 performers delivering algorithms to the EIT and other TAs for integration and testing purposes, code drops from TA1 will occur approximately every three months throughout the program around the time of each Quarterly Program Review (QPR).

7. Q: Per section 3a on page 48 of the BAA, "proposals may be received and evaluated up to six months (180 days) from the date of posting on FedBizOpps". Is it DARPA's intent for proposers to have offers VALID for 6 months (180 days)?

A: “While proposers are not necessarily required to have their proposals valid for the full 180 days, the proposals must remain valid for long enough to encompass the proposal evaluation and contract negotiation processes.”

8. Q: What is the anticipated security classification of the TA3 data sets to be provided, and will this classification bleed over into the rest of the TAs?

A: The classification is anticipated at SECRET-Collateral; however, there is potential for higher classification. It is not anticipated that the classification of the TA3 datasets would be exposed to the other TAs.

9. Q: When will DD254s be issued?

A: DD254s will be issued for any contracts involving access to classified information at the time of award. For all other contracts, DD254s will be issued during the program as needed.

10. Q: If a proposer is not currently cleared to handle classified information, will that be an issue?

A: Per III.A.1.c)(2) “When proposing to perform classified work, applicants will ensure all industrial, personnel, and information systems processing security requirements are in place at the appropriate level (e.g., Facility Clearance Level (FCL), Automated Information Security (AIS), Certification and Accreditation (C&A), and any Foreign Ownership Control and Influence (FOCI) issues are mitigated prior to contract award.”

11. Q: Is the full-scale Tactical Autopilot being implemented by TA1 or TA4?

A: TA4 will implement the TA1 algorithms on full-scale aircraft.

12. Q: Do you expect the MC scenarios to require manual control for all actions to be taken, or for the MC scenarios to be AI-aided?

A: Any simulated unmanned aerial systems will have realistic levels of autonomy.

However, the MC scenarios will have no blue battle management intelligent agent.

TA2 and the EIT will work closely with TA3 to ensure that the MC scenarios for the DOT will have appropriate demands on pilot attention to enable trust assessment (e.g, the TA2 performer may dictate some constraints for the AI within the DOT MC scenarios to support the experimental design of the trust assessment experimentation). Regardless of agent intelligence, within the context of the ACE program, all actions involving legal, moral, and ethical concerns will be driven by the human pilot in the loop. Beyond that, TA3 performers are encouraged to propose any level of intelligence deemed appropriate and achievable for both the DOT paradigm MC scenarios and the broader TA3 technical objectives.

13. Q: Do you expect to run simultaneous scenario runs?

A: Only in M&S. We do not anticipate simultaneous scenario runs for sub-scale and full-scale testing. If performers would like to propose running simultaneous scenarios, they should provide a description for how that would be executed and a justification for the added value.

14. Q: Will the same simulation environment be used for TA2 and TA3?

A: Not necessarily. The EIT will specify the simulation environment for the Mission Commander (MC) simulation-based task to be used by TA2 in the DOT paradigm for trust assessments as well as the simulation environment to be used for TA3 AlphaMosaic algorithm development and assessment. The selected environments for both will be determined early in the program and may be the same. If performers have a preference for either or both, that should be stated and justified in their proposals.

15. Q: Will the DOT interact with TA1?

A: TA1 algorithms will be used within the DOT for the dogfight task. TA2 proposers should describe and justify how they will integrate the TA1 algorithms into the DOT, including the HMI. The EIT will facilitate interactions between TA1 and TA2.

16. Q: Will red (adversary) AI be provided by the EIT?

A: Yes. The EIT will provide red air for both training and evaluation of TA1 and TA3 algorithms. TA1 and TA3 performers are also welcome to propose to develop their own red air for training purposes.

17. Q: Will AI be scalable, from automatic to manual?

A: Assuming this is in reference to TA1 AI, the human pilot in the manned aircraft will have the ability to turn the AI off and on. When the AI is switched on, the autonomous behaviors of the dogfight AI are dictated by TA1.

18. Q: Will 1v1 and 2v2 flights occur simultaneously?

A: Assuming this question is in reference to sub-scale and full-scale flights, no.

19. Q: What is the division of responsibilities between TA1 and TA4?

A: TA1 algorithms dictate combat maneuvering (i.e., flight path and end point) of the full-scale aircraft in 3 dimensions. TA4 dictates how those algorithms are instantiated and executed on the aircraft.

20. Q: What should TA3 performers expect from TA1 algorithms?

A: Algorithms from TA1 are not required to be directly transferred to or leveraged by TA3. The EIT will manage the interface between TA1 and TA3 to ensure that TA3 performers are provided with sufficient data to support integration of TA1 individual aircraft combat maneuvering (dogfight) behaviors into TA3 training data and algorithms.

TA3 proposals should indicate how TA1 outputs would be leveraged to meet TA3 objectives.

21. Q: Will any GFE hardware be provided?

A: The EIT will provide an environment for the M&S and sub-scale TAEs, as well as the sub-scale aircraft for use in Phase 2. These resources will also be made available to performers for development and internal testing outside of the TAEs. For any additional GFE hardware, that is up to the proposers to identify and justify.

22. Q: What is the relationship between TA2 and TA3?

A: TA2 provides the front-end (HMI) to the TA3 back-end MC scenarios to be used in the DOT paradigm for the trust assessments.

23. Q: In Section IV. B. 1. c. Subawardee Proposals on page 40 of the BAA, "Subawardee proposals submitted to the Government by the proposed awardee should be submitted in a sealed envelope that the proposed awardee will not be allowed to view." Since proposals are to be submitted digitally via DARPA's BAA Website (https://baa.darpa.mil) in a single zip file (page 49), are subawardees required to submit their proposals as sealed hardcopies per page 40 or digitally via the DARPA website per page 49?

A: Subawardees may submit their cost proposals through the DARPA BAA Tool (BAAT) and should identify the prime awardee when submitting. Please refer to Section IV.B.3.a. of the BAA for information on submitting through BAAT.

24. Q: Are the Table of Contents, List of Tables, List of Figures, and Acronym List counted against the 40-page limitation of Volume 1, Technical and Management Proposal.

A: Yes.

25. Q: Does the SOW count against the page count?

A: Yes.

26. Q: Can the SOW be included as an Appendix?

A: No.

27. Q: Who were the performers on the seedling for the program?

A: There was no seedling effort for the ACE program. The seedling mentioned in the industry day slides was a seedling under another STO Program Manager, and simply served to illustrate one potential method for scaling algorithms.

28. Q: What is the Funding Type (6.1, 6.2, 6.3) for the Program?

A: ACE is a 6.3 program.

29. Q: Are there any specific section page limits for responses or just an overall page count limit?

A: Just an overall page count limit.

30. Q: Can SAV be used for proposal submissions?

A: No. All classified submissions have to follow the submission requirements in IV.B.2.b.2 of the BAA.

31. Q: Who will be the contracting agent for the program (agency & office/division)?

A: The DARPA Contracts Management Office (CMO) will be the agent for TA1, TA3, and TA4. The Air Force Research Laboratory (AFRL) at Wright-Patterson Air Force Base (AFB) will be the agent for TA2.

32. Q: How will interactions and requirements exchanges between TA1 and TA2 be handled? Will requirements be unilateral or bilateral?

A: The EIT will facilitate bilateral communication between the TA2 performer and each of the TA1 performers to accommodate both TA1 and TA2 requirements and constraints. Workshops and regular working meetings will be held to support these interactions, particularly early in the program during requirements development.

33. Q: How much time delay should be built in for the human in the loop?

A: Assuming this question is referring to building in additional time between the TA1 algorithms determining the next combat maneuver to execute and the initiation of that maneuver, presumably in order to allow for information to be conveyed to the pilot via the HMI and/or for the pilot to intervene; this is something that should be considered and proposed with a justification by both TA2 and TA4 performers. These types of requirements will be defined after program kickoff, and will be facilitated via working groups led by the EIT.

34. Q: Are the TAEs the same as the Phase 3 flight testing events?

A: Yes.

35. Q: Can performers propose to do internal LVC to mitigate risk.

A: Yes. Include a detailed plan and justification.

36. Q: Will the TA1 algorithms ever take control away from the pilot in the way that an AGCAS would?

A: No.

37. Q: Please provide background information on the TACE processor.

A: The Test Resource Management Center funded the TACE program to develop a set of airborne and ground-based utilities to enable an autonomous control algorithm to safely issue commands to the autopilot onboard a small UAS. The airborne utilities serve to pass through commands from the autonomous controller under normal operations, disregard commands that serve to violate a safety condition, and invoke autonomous remediation to safe flight as required. TACE software and documentation will be provided to ACE performers as GFE/GFI.

38. Q: Are any of the data to be provided/collected ITAR restricted?

A: The use of ITAR-restricted data is not within the current scope of the program. This is subject to change.

39. Q: Is it possible for a prime to submit two proposals to a single TA if the proposals being developed are from different divisions of the company and comprise distinct approaches?

A: No. Only one proposal will be reviewed per prime per TA. Primes are encouraged to consolidate ideas and team members into a single proposal.

40. Q: Can an extension be obtained for the proposal due date?

A: As indicated in IV.B.3 of the BAA, proposal submissions received after the proposal due date and time may not be reviewed.

41. Q: Can performers propose to use algorithms that are enhanced by human expertise?

A: Yes.

42. Q: Are you interested in causality explanation?

A: Yes. A case should be made for any approach that could contribute to building well-calibrated trust in combat autonomy and appropriate unbounding of AI performance capabilities.

43. Q: Who is responsible for HMI development for full-scale aircraft?

A: TA4 will be responsible for developing and integrating the HMI designed by TA2.

44. Q: Is there a limit on G loading?

A: This will be determined by TA4 and it must be ≤ the red aircraft capability.

45. Q: Will the pilot take control of the UAV(s) during 2v1 or 2v2 dogfights?

A: No.

46. Q: Can proposers provide their own Air Combat Maneuver Instrumentation (ACMI) pods and data acquisition approaches?

A: Yes. The government intends to provide TSPI data for all real and simulated aircraft;

however, proposers are welcome to describe an approach and justification for obtaining using their own aircraft ACMI/TSPI data.

47. Q: What are the types of tasks that pilots will perform while the Artificial Intelligence (AI) is taking over the flight/dogfight operations? Are these tasks part of the ACE program? Are there other types of tasks that can be proposed?

A: The secondary mission command tasks and corresponding scenarios will be part of the ACE program, particularly the TA2 trust assessments, and are envisioned to be representative of TA3 (AlphaMosaic) mission types and scenarios; however other tasks may be proposed by TA2. TA2 proposals should provide recommendations on tasks that stimulate the operator in ways that will support the Dual Operational Task (DOT) paradigm to be used for trust evaluations. TA3 proposed tasks, scenarios, and solutions should provide requirements/opportunities for the human pilot to manage or control elements of the TA3 (AlphaMosaic) mission. Naturally the pilots will still retain the tasks associated with system health monitoring and other aspects of safe aircraft operations.

48. Q: How might the Explainable Artificial Intelligence (XAI) program feed into ACE?

A: Performers are encouraged to leverage outcomes of XAI or any previous/ongoing related efforts, for example, to develop HMIs that support the creation of well-calibrated trust.

49. Q: Is predictability a liability? There seems to be a lot of debate in this area.

A: Performers in all TAs should consider what levels of predictability are necessary over time in order to support well-calibrated trust while still retaining novelty required to effectively manage the dynamics of the problem posed by an adversary. Introducing the ability to change the level of predictability based on individual operator trust levels could help co-evolve the tactics and technology.

50. Q: What is the funding level for the individual TAs?

A: The overall Budget for the TA2/3/4 BAA is $63.6 million. There is no prescribed funding level for the individual TAs.

51. Q: How can a very small company participate?

A: For TA1, the AlphaDogfight Trials provide an ideal mechanism for individuals and small teams to participate and posture themselves for teaming opportunities on the TA1 BAA. Any individuals and small organizations can also participate on any TA either as a prime performer or by teaming with other organizations or put forth an independent proposal.

52. Q: Who provides the UAV assets for Phase 2?

A: The government-led EIT will be purchasing, instrumenting, and delivering the sub-scale aircraft.

53. Q: Who “owns” moving the TA3 large scale data analytics and sense-making and the TA1 combat autonomy from the M&S environment to the TA4 aircraft?

A: TA1 performers develop the combat autonomy algorithms. TA4 integrates the TA1 combat autonomy on the full-scale aircraft. TA2 performers design the HMIs for interaction with the aircraft during dogfights and the HMI for the TA3 (AlphaMosaic) scenarios to be used in the DOT paradigm for trust assessments. TA4 will integrate both HMIs onto the full-scale aircraft. The government-led EIT will facilitate cross-TA interactions, and no direct interaction is anticipated between TA3 and TA4 performers without EIT involvement. Working groups will also be held at the beginning of the TA234 BAA Phase 1 and the TA1 BAA Phase 1 to work out inter-TA interactions.

54. Q: Do you envision the same prime to be awarded multiple TAs?

A: There are no restrictions on primes bidding or being awarded prime contracts for multiple TAs. However, a prime can only submit 1 proposal per TA.

55. Q: Why is the TA1 BAA later and separate?

A: The government aims to cast a wide net for TA1 algorithm developers, to include non-traditional DoD performers. This will be accomplished using the AlphaDogfight Trials, which will be initiated prior to the formal DARPA TA1 BAA to stimulate the network of potential performers.

56. Q: How is ACE different from the Air Force Research Laboratory (AFRL) Skyborg program?

A: ACE intends to develop technologies that could be transitioned to programs like Skyborg and integrated into both manned and unmanned department of Defense (DoD) aircraft. ACE is specifically focused on developing trusted, scalable, high-performing air combat autonomy technology, while Skyborg is seeking to drive toward a fully deployable system by 2023.

57. Q: Will there be an opportunity for one-on-one meetings with the ACE Program Manager after the BAA is released?

A: Interested parties were able to sign up for one One-on-one meetings with the PM that were held on June 18 and 21. No further one-one-one meetings will be held with respect to this BAA.

TA1

58. Q: Will TA1 have its own Proposers Day?

A: Yes.

59. Q: Can we let a pilot try to execute the algorithm-generated commands before letting the autopilot?

A: Yes. Describe and justify your proposed approach.

60. Q: Will TA1 get feedback from TA2 to advise the development of trusted algorithms?

A: The provision of real-time feedback to the TA1 algorithms is contingent upon the TA2 performer approach, therefore should not be assumed as being provided/available;

however, if performers think that real-time or post hoc feedback is an important aspect of TA1 and/or TA2, that should be described and justified in the proposed approaches.

61. Q: Can a TA1 performer also be a sub on TA2 in order to help define interface needs for/to TA1?

A: Yes.

62. Q: What sensors are assumed for the red and blue aircraft? Is DARPA going to specify/provide the sensors, the features they measure, performance, etc.?

A: Assume perfect knowledge of TSPI for both red and blue aircraft tracks will be provided. DARPA reserves the right to add degradation and different sensor types of information if successful with TSPI data alone.

63. Q: What is considered “winning” in the dogfight (e.g., valid shots/saddled into control zone for X number of seconds/hard deck/ kill ratio)?

A: The win conditions will be specified in the TA1 BAA and will include some of these elements.

64. Q: In TA1, will our algorithms be fighting humans or automated aircraft?

A: Both; this will be specified in the TA1 BAA.

65. Q: For TA1, can performers provide their own simulation environment or must they use the provided simulation tools?

A: The simulation environment will be provided by the government-led EIT as the environment must be standard across all performers for evaluation purposes. However, for internal development performers are welcome to perform risk reduction and initial algorithm testing on their own environment as long as it can be successfully converted into that provided by the EIT during the evaluations.

66. Q: Regarding the transparency of the AI, do you intend the AI to act as a human would (but faster) or act in some optimal way? The system that won the Go championship used new/non-standard strategies that confused experts. Such algorithms could result in low trust but high performance.

A: We are looking for solutions that provide performance advancements while capably engendering appropriate trust. Low-performing algorithms are not useful, nor are high performing algorithms that can’t be trusted. Performers should consider a plan to perform a bounded expansion of trust that once achieved can permit an unbounded expansion in performance.

TA2

67. Q: Will all HMI take place in the cockpit?

A: Yes. A human pilot will interface with both the dogfight task and the Mission Commander (MC) task from a simulated or full-scale live aircraft cockpit.

68. Q: How long are Trust Assessment Events (TAEs)?

A: Each TAE is expected to last 5 days, potentially with additional days dedicated to set-up, testing, and breakdown.

69. Q: How many HMIs need to be built?

A: TA2 is responsible for designing and prototyping the HMI (or HMIs) to support both the dogfight role and the MC role. This could consist of 2 distinct HMIs, a single HMI, or a hybrid design; proposers should indicate if these will be separate interfaces and justify their approach. TA4 will implement the final HMI(s) on the full-scale aircraft.

70. Q: Will the AI ever be intentionally turned on and off by the experimenters, rather than the pilot or the safety pilot?

A: It is up to the TA2 proposers to submit and justify an experimental methodology for measuring and modeling trust.

71. Q: Will the same pilots act as participants for the TAEs across the entire program?

A: It is expected that pilots will be “borrowed” from active duty, so it would be very difficult to ensure their availability for the entire length of the program. Performers should address how their approach will accommodate the likelihood of different pilots participating in TAEs.

72. Q: Should the HMI be a handheld tablet or integrated into the pilot-vehicle interface

(PVI)?

A: That is up to the proposers and may include other alternative interfaces.

73. Q: How will eye tracking measure trust?

A: It is up to the TA2 proposer to develop a methodology for trust measurement and calibration, to include the specification of data collection (such as eye tracking) and how such measured data will be processed to determine the trust assessment.

74. Q: Is it envisioned that trust measurement in the cockpit might be used operationally?

For example, to determine if pilots are trusting the technology once it’s fielded or to use real-time trust measurements to adjust AI behaviors?

A: This is not within the scope of the current BAA.

75. Q: Is it reasonable to propose the use of sensors to provide contextual information regarding status of the aircraft, environmental conditions, or the pilot for use in flight? Is it reasonable to propose physiological sensors for use in flight?

A: Yes. It is up to the proposer to develop and justify a methodology for trust measurement and calibration, to include the specification of data collection (such as eye tracking) and how such measured data will be processed to determine the trust assessment.

76. Q: What is meant by representative HMIs?

A: Assuming this refers to representative HMIs for Manned-Unmanned Teaming (MUMT) battle management on Page 11 of the BAA, the TA2 performer should focus HMI development and optimization on the dogfight task as the primary purpose of the MC task is to divide the pilot’s attention in order to measure trust.

77. Q: How do you see trust and human machine interaction scaling from simple to complex behaviors and contexts (e.g., single to multiple platforms)?

A: We are looking to TA2 performers to model what that looks like. TA2 proposals should describe how that scaling would be measured and modeled.

78. Q: Will pilots interact with the MC simulation environment and TA3-developed scenarios in all phases of the program, including in full-scale aircraft?

A: Yes. TA2 will be conducting trust assessments using the DOT paradigm in all three Phases.

79. Q: Can HMIs include helmet displays?

A: Yes. Describe and justify the value.

80. Q: Will the TA2 performers have access to the test environments/platforms/pilots outside of the TAEs for internal experimentation and testing? Can TA2 performers propose to use proxy tasks and/or subjects for internal testing?

A: Yes. Detail the proposed experimental approach and justification in your proposal.

Additionally, the EIT will be making replicas of the M&S and subscale experimentation environments available to performers for interim internal development/testing at locations TBD based on proposal selection results.

81. Q: Is there interest in exploration of trust relationships with the AI that is part of the MC task and scenarios to be used within the DOT paradigm for the TAEs, or is the focus solely on assessing trust in the dogfight AI?

A: The focus is solely on assessing trust in the dogfight AI.

82. Q: In the 2v1 and 2v2 dogfighting scenarios pilot trust in the dogfight AI might look very different than in the case of 1v1. It might also be difficult to assess trust in the AI if it is flying both the aircraft the pilot is in and an unmanned wingman while the pilot is also conducting the MC task as part of the dual task paradigm. Is there a vision for how TA2 should address this potential confounding factor?

A: TA2 proposals should detail and justify an experimental methodology and approach for modeling and measuring trust in the TA1 algorithms, both on the manned aircraft and on an unmanned wingman in a 2v1 or 2v2 dogfight. The proposed methodology should provide a means for eliminating potential confounding factors.

83. Q: For TA2 proposals, does the required IRB draft protocol for HSR count towards the technical proposal page limit?

A: No. The required (TA2 only) IRB draft protocol for HSR does not count towards the technical proposal page limit. It should be included as an Appendix to the Volume I Technical and Management Proposal.

84. Q: Does TA2 dictate the distractor (MC) task for the DOT paradigm and also develop the HMI for that task?

A: TA2 can provide guidelines to TA3 for the MC task and task scenarios to be used as part of the DOT paradigm in order to ensure that the task serves its purpose in the trust assessments by requiring the appropriate amount of attention from the pilot in order to perform well on the task. TA3 will design and develop the MC task scenarios for the DOT paradigm. TA2 will develop the HMI for the MC task and scenarios to be used in the DOT paradigm for the trust assessments. The EIT will facilitate interactions between TA2 and TA3 to support this.

85. Q: For 2v1 and 2v2 dogfight engagements, would the autonomy only be in use on the unmanned wingman or both on the manned platform and the unmanned wingman?

A: TA2 proposals should detail and justify an experimental methodology and approach for modeling and measuring trust in the TA1 algorithms both on the manned aircraft and on an unmanned wingman in a 2v1 or 2v2 dogfight. This could include experimental scenarios in which the AI is only implemented on the unmanned wingman and/or the scenarios in which the MC task is not used to divide pilot attention.

86. Q: Will the EIT have IRB coverage that extends to the TA2 performers and/or provide oversight of TA2 HSR?

A: No. TA2 performers will be required to obtain IRB approval from their own IRB (local) as well as secondary approval from HRPO for all HSR.

87. Q: Will HSR involve active duty military, which constitutes a protected population?

A: Yes. The TAEs will involve active duty military.

88. Q: Can TA2 proposals include a menu item of options similar to the encouraged cost options for TA4?

A: Yes. Providing DARPA with options for scope and cost is allowed. TA2 proposals must provide a detailed description of the value added and cost of all options. Additionally, TA2 proposals must adhere, at a minimum, to the Base Period (Phase 1), and two Option Period (Phases 2 and 3) structure outlined in the BAA. Additional options may be proposed, but each must be supported and costed independently and to the same level of detail as that required by the BAA for the Phase 1-3 periods.

89. Q: Can we assume a mounted tablet for HMI interface purposes?

A: Proposers should describe their approach and justify its value.

90. Q: Will the AI algorithm be limited in the type of maneuver?

A: There will be a safety layer both manual and auto. This safety layer is configurable.

91. Q: Is it in scope to add sensors to the program for the functioning of the algorithms?

A: No, TSPI data required for algorithm function will be provided; but if additional data/sensors would improve your approach, explain the approach and expected value in your proposal.

92. Q: There are tradeoffs between explainability and accuracy in models as novel solutions might not be predictable?

A: Agreed, DARPA is interested in bounded expansion of trust leading to an unbounded expansion in performance.

93. Q: Are on-body sensors okay?

A: Yes, if not invasive and are IRB approved and compliant. Describe and justify value added.

94. Q: Are there constraints on HMIs or interaction methods?

A: No, proposers should describe their approach and justify its value.

95. Q: Is the generation of a mathematical model of trust part of the task?

A: Yes.

96. Q: Is there any interest in HMIs/algorithms to “increase” trust, or only to measure/model trust?

A: Yes. We expect the HMI itself to contribute to appropriate trust building. For example, features such as path projection and decision explanation could support well-calibrated trust and serve to increase trust. Modeling and measuring trust are required elements of “increasing” trust.

97. Q: In the trust evaluation, will the pilots be given time to develop (or not) trust in the autonomous system?

A: Yes. TA2 proposals should include a description of the methodology to be used to train pilots to use the system and to develop positive or negative trust in the system.

Each week long trust assessment event will include dedicated training time at the beginning to provide familiarity with the system.

98. Q: Are Turing-like tests a possibility (meaning sometimes the AI is flying the dogfight and sometimes another human is flying the dogfight, and the pilot doesn’t know)?

A: Possibly, if a performer feels this would be helpful they should provide a detailed description of how this type of paradigm supports the ACE program goals.

99. Q: Are you looking for measures that can be implemented in the platform itself in real-time?

A: While measures of trust must be proposed across M&S, sub-scale, and full-scale aircraft to enable post flight analysis, real-time assessment is not a requirement. Any proposed real-time assessment and/or closed loop solutions should be described and justified.

100. Q: To what degree do you distinguish measures of trust from other cognitive constructs (e.g., workload, communications efficiency)?

A: Addressing this question should be part of the proposed approach and associated justification.

101.Q: How do you account for trusting too much (overreliance on automation)?

A: TA2 proposals should describe and justify an approach that will develop well-calibrated trust, not over or under reliance.

TA3

102.Q: Should the TA3 game be more like a simulation or an arcade game?

A: Simulation.

103.Q: How many scenarios is the TA3 expected to develop?

A: 3-5 scenarios.

104.Q: Can a TA3 develop a test harness to collect data and deconflict with the EIT?

A: Yes.

105.Q: For the first quarter delivery, what is expected of the Data Model Design?

A: Explain your plan, your process/methodology, and schedule and provide any early drafts of your design.

106.Q: What algorithms are expected for the datasets?

A: Need algorithms which can understand tactics and recognize new strategies.

107.Q: Will TA3 assess trust with a Human In The Loop (HIYL)?

A: TA3 is not responsible for any trust assessments; all trust assessments will be conducted under TA2.

108.Q: Will the campaign data set include simulation runs and is there a size limit (i.e., number of sorties limit)?

A: Yes, the proposer needs to be able to analyze multiple data sets. Any limits to the scoping and size of the data sets should be addressed in the proposal.

109.Q: For TA3, how complete will be the EIT mission commander scenario?

A: The EIT will provide red air for the MC scenarios. The complexity and pilot interactions with the scenarios will also be partially guided by requirements from TA2 for the trust assessment methodology using the DOT paradigm. These interactions will be facilitated by the EIT.

110.Q: Do TA3 teams have to develop DOT scenarios for both AFSIM and NGTS?

A: No. TA3 performers will only be developing scenarios in one environment; however, given that the environment has not yet been selected, TA3 proposers should indicate which environment they are more familiar with and/or which they would recommend for TA3 development. If recommending one environment over the other, proposers should provide a justification. TA3 proposers recommending one environment over the other and proposing to develop in the recommended environment should also include an explanation for how they would complete development in the other environment if required to do so, how much longer it would take to complete development in the other environment, and any other potential considerations that reviewers should consider.

111.Q: If the TA3 DOT scenarios are in AFSIM, would the rest of TA3 development also be in

AFSIM?

A: Yes.

112.Q: Will the red air for TA3 be provided by the EIT?

A: Yes. The EIT will provide red air for training and assessment purposes for the MC scenarios to be used in the DOT paradigm for the TAEs, and will also provide red air for assessment of the TA3 AlphaMosaic algorithms. TA3 performers will also have access to LFE data that includes red air. However, this does not prevent TA3 performers from proposing to develop their own red air for algorithm training/development purposes.

If proposing to develop red air in order to increase blue air capabilities/performance (e.g., training via self-play) TA3 proposers should include this approach in their proposals along with a justification.

113.Q: If the intelligent agent within the TA3-generated MC scenarios for the DOT consistently performs well (i.e., is trustworthy), this could impact the DOT paradigm experimental results. How intelligent should that agent be?

A: See answer to Question 12 above.

114.Q: Will TA3 performers need to contribute to writing the ICD with the EIT?

A: The EIT will host a series of workshops after the program kickoff in order to reach consensus on the simulation environments to be used and details of the APIs/ICDs.

115.Q: Is the TA1 intelligent agent the same as the one to be used in TA3?

A: TA3 proposals should detail and justify how potential performers would go about leveraging TA1 algorithms and/or developing their own.

116.Q: Current Large Force Exercise data likely doesn’t include a lot of Mosaic warfare tactics and strategies. Is it assumed that the data provided would be supplemented with virtual data?

A: Yes. Under this BAA we want to explore beyond just the way things are currently being done, including use of future weapons, sensors, etc., which will have to be simulated under ACE.

117.Q: What is the size of the envisioned campaign for TA3 testing (e.g., sorties per day)?

A: TA3 proposers should specify and justify the size of the envisioned campaign consistent with the scenarios provided in the BAA (DCA, OCA, SEAD, CSAR, and TBM hunt).

118.Q: What does TA3 develop versus the EIT for the MC scenarios for the DOT paradigm?

A: The EIT will provide the red (adversary) force for the scenarios. TA3 does the rest.

119.Q: Will the MC scenarios for the DOT paradigm be at the classified level?

A: The primary point of the MC scenarios for the DOT paradigm is to provide a distractionary task to require the pilot to divide attentional resources. The government expects this can be accomplished representatively at the unclassified level. If proposing an approach in which MC scenarios for the DOT paradigm be at the classified level proposers should justify the rationale/value.

120.Q: Page 26 of the BAA states that, “The TA3 performers will extend WVR combat algorithms to campaign levels. The goal will be to increase performance for heterogeneous multi-aircraft campaigns of increased complexity to include high fidelity versions of all mission scenarios (DCA, OCA, SEAD, CSAR, and TBM hunt). The TA3 performers will conduct experimentation and demonstration to explore and identify new strategies and tactics based on data analytics from large force exercises. These new strategies and tactics should be instantiated within the DOT paradigm task scenarios for use in the full-scale trust assessments conducted by the TA2 performer.”

What is meant by high fidelity?

A: That refers to the potential for having higher level electromagnetic (EM) information (e.g., beyond Link 16 data).

121.Q: Is TA3 expected to design for pilot trust or conduct Human Subjects Research (HSR)?

A: No. All HSR will be conducted under TA2, and the TA2 trust assessments will focus on pilot trust in the dogfight (TA1) AI.

122.Q: What is the envisioned/anticipated/desired size and diversity of engagements for TA3? What are the measures of interest/goodness (e.g., timescale, scalability)?

A: TA3 proposals should provide suggested figures of merit and justification for what constitutes high performance from a Mosaic Warfare perspective.

123.Q: Will the various TA3 performers be required to develop similar MC tasks and task scenarios to enable the TA2 performers to create just one HMI for the MC task and scenarios to be used in the DOT paradigm for trust assessments?

A: TA2 proposals should detail and justify an approach to both accommodating and constraining multiple TA3 MC task/scenario designs. TA3 proposals should detail and justify an approach to both accommodating and constraining requirements from TA2 for the MC task and scenarios to be used in the DOT paradigm for trust assessments.

The EIT will also facilitate bilateral communication between the TA2 performer and each of the TA3 performers to support negotiation of any requirements on behalf of both TA3 and TA2. Workshops and regular working meetings will be held to support these interactions, particularly early in the program during requirements development.

124.Q: Page 16 of the BAA (2nd paragraph) states: “Proposals should include details on how their model will be implemented on both the Advanced Framework for Simulation, Integration, and Modeling (AFSIM) and Next Generation Threat System (NGTS) to facilitate interaction across TAs within the ACE program. The relative merits of each simulation environment for their proposal must be explicitly described.” Page 18 of the BAA (2nd paragraph) states: “Scenario definitions will be delivered in the selected simulation environment” (AFSIM or NGTS). Is the EIT choosing between AFSIM and NGTS for the Mission-level M&S environment--and potentially other mission-level simulations proposed in BAA responses (i.e., both AFSIM and NGTS will not be used in the EIT M&S environment)?

A: Yes, one environment will be selected by the EIT. The government is interested in leveraging the experience of industry in making this decision.

125.Q: Page 15 of the BAA states: “TA3 performers will develop Mission Commander DOT scenarios for use in the TAEs”…and implement in simulation”. Page 18 of the BAA states: “Each government-furnished Mission Commander scenario will include red forces of different levels of capability and will be delivered in order of increasing complexity.” Will the TA3 team be provided with government-furnished mission commander scenarios as a starting point for their scenario development or is this a typo? If initial scenarios are provided, what will be their classification level?

A: Page 18 should read, “Government-furnished red forces of varied levels of capability will be provided for each Mission Commander scenario, and will be delivered in order of increasing complexity.” The TA3 teams will be responsible for developing the MC scenarios based on the GFE/GFI red air. It is desired that the Mission Commander scenarios for use in the DOT be developed at the unclassified level.

126.Q: Are performers permitted to add additional domain expertise to the TA3 solutions?

A: Yes.

127.Q: With respect to the Dual Operational Task (DOT), what are TA3 performers providing?

A: The DOT is a means to facilitate the measurement of trust in TA2 via 3-5 air combat scenarios to be used to divide pilot attention. TA3 performers are providing the Mission Commander scenarios the pilot will interact with.

128.Q: Is the focus of TA3 on providing MC scenarios for the dogfight trust assessments or on developing larger scale AlphaMosaic algorithms?

A: TA3 is focused on scaling from local to global missions. The MC scenarios are being developed by TA3 simply to provide TA2 with an operationally relevant distractionary task for the DOT paradigm to be used in the TAEs.

129.Q: If we use learning techniques in TA3 will it conflict with TA2?

A: No.

130.Q: What do TA3 performers get from TA1 performers?

A: See answer to Question 20.

131.Q: The three different scenarios in TA3, are they all the same type?

A: No. See answer to Question 125.

132.Q: Is the TA3 performer or the EIT developing the (AlphaMosaic) mission scenarios?

A: TA3 performers will develop these scenarios and work with the EIT to verify compliance with TA2 evaluations and the greater M&S framework. The EIT will also provide high-level scenario categories such as (cruise missile defense or suppression of enemy air defenses) to help shape the specifics of the TA3 products. These mission scenarios will also be informed by government provided live large force exercise data.

133.Q: What is the product from TA3?

A: Mosaic, campaign-level data analytics tools, mission commander scenarios for use during the trust assessment events, algorithms capable of performing battle management for simulated missions, and quantification of the relationship between local and global behavior performance metrics.

134.Q: When you say “large force exercise data analytics”, are you actually providing a data set from a large force exercise or just referring to that as an example? Can you clarify the data that will be provided? Will the data sets provided be classified?

A: Data from instrumented large scale training exercises will be provided to TA3 performers. This will minimally include TSPI and shot logs or other content. It is safe to assume that these data sets will be classified. It should also be noted that TA3 performers are encouraged to propose and utilize any additional relevant data sets that they have access to.

135.Q: Can TA3 performers start with data from game-like simulations?

A: Yes. Provide a description of the types of data and the approach for leveraging simulation-based data to support development of algorithms for real data sets.

136.Q: How much data do you have to provide to TA3 performers?

A: This has yet to be determined.

TA4

137.Q: Please define the demonstration periods (i.e., length of time) involving full-scale aircraft.

A: See Table 3 for a schedule of full-scale demonstrations and tests (including the full-scale TAE). Each demonstration event will take place over the course of 5 days.

138.Q: Please provide an estimated number of flight hours for the full-scale demonstrations A: This has yet to be determined and will depend in part on proposed approaches for other TAs. TA4 proposals should provide detailed pricing options, including costs per flight hour for each proposed full-scale aircraft option.

139.Q: Do the full-scale aircraft need to be exclusively dedicated to the ACE program?

A: No. The aircraft will have to be available for use during the scheduled air worthiness tests and full-scale demonstrations. TA4 teams will be expected to manage the availability of the full-scale aircraft for these events and all internal development and testing.

140.Q: Is having a low-performance flyable aircraft as a software-integration-lab before adapting the final test aircraft allowable?

A: Yes. Include all costs and justification in your proposal.

141.Q: What is the level of control you will have of the aircraft? Will it be the same inputs a pilot has or can you have control of the flight control surfaces directly?

A: There are several acceptable approaches to aircraft control. Describe and justify your proposed approach.

142.Q: Who provides the tactical data link?

A: You can presume the tactical data link between aircraft will be government furnished equipment (GFE).

143.Q: Is there emulation of a sensor?

A: No.

144.Q: If the TA4 performer is providing the autopilot, what does the EIT provide?

A: Quarterly interface meetings to define and refine the ICDs and APIs.

145.Q: Is there a radar or other sensor onboard?

A: No. Algorithms will have access to real-time TSPI data for friendly/adversary aircraft.

146.Q. Do TA4 aircraft require 2 seats?

A. Yes, for the blue aircraft; optional for the red aircraft.

147.Q: Who is the airworthiness authority for TA4? Please describe the DARPA airworthiness process.

A: DARPA has an appointed airworthiness authority. The DARPA airworthiness process is detailed in the DARPA airworthiness policy (DARPA Instruction-69). The selected TA4 performer will be provided with all information necessary to obtain an airworthiness certification. Instruction will be provided to TA-4 performers once under contract. TA4 performers should specify in their proposals the airworthiness certification they intend to obtain (e.g., FAA, DARPA), including cost.

148.Q: What is expected from TA4 at the first QPR1? Should an initial design be completed by the end of Month 3?

A: Just the initial design is expected from TA4 by the end of Month 3 for QPR1. The design is expected to evolve from there. Performers will also conduct an Aircraft Selection Review (ASR) within 3 months of contract award to provide a detailed analysis justifying aircraft selection prior to proceeding with any aircraft or long-lead acquisition.

149.Q: How are trade-offs between representative 5th Generation Fighters (increased transition potential) and representative 4th Generation Fighters (reduced cost/easier access) prioritized in the program vision?

A: Aircraft must be real enough to build trust and should be representative of a 4th generation fighter in terms of combat maneuvering capabilities. A total of four aircraft are desired to enable full-scale 2v2 engagements. However, cost/performance tradeoff options are sought if well justified. Proposals are encouraged that include multiple aircraft options with associated pricing for both blue and red aircraft. For example, a proposal might include the cost of renting aircraft by the hour to provide adversaries for evaluation events or options for two different kinds of blue aircraft could be included to help design an affordable program plan.

150.Q: Will the Dogfight engagements and win conditions involve guns only?

A: Yes. The win conditions will be provided to performers and will simply require blue or red air to acquire and maintain a specified position of advantage (based on TSPI data) over the opponent consistent with a gun WEZ for a specified period of time.

151.Q: Is it possible/desirable for TA4 performers to propose a combination of dissimilar aircraft (e.g., parity between red and blue aircraft)?

A: It is possible, but should be detailed and well justified in TA4 proposals. Proposals are encouraged that include multiple aircraft options with associated pricing and justifications of value added for both blue and red aircraft.

152.Q: How much novelty should be enabled on red TA4 aircraft?

A: This has yet to be determined and will be dictated in part by TA1 performance achievements and algorithms developed by the EIT.

153.Q: Can TA4 teams propose to use their own ACMI pods for collecting/communicating TSPI data?

A: DARPA is planning to provide TSPI…