HR001121S0030-Amendment-01.pdf

HR001121S0030 EDGE 1

Broad Agency Announcement Enhancing Design for Graceful Extensibility (EDGE)

Defense Sciences Office

HR001121S0030

Amendment 1

July 16, 2021

HR001121S0030 EDGE 2

Table of Contents I. Funding Opportunity Description

A. Introduction B. Background C. Program Description/Scope D. Program Structure E. Technical Area Descriptions F. Schedule/Milestones G. Deliverables H. Government-furnished Property/Equipment/Information I. Other Program Objectives and Considerations

II. Award Information A. General Award Information B. Fundamental Research

III. Eligibility Information A. Eligible Applicants B. Organizational Conflicts of Interest C. Cost Sharing/Matching D. Ability to Receive Awards in Multiple Technical Areas - Conflicts of Interest

IV. Application and Submission Information A. Address to Request Application Package B. Content and Form of Application Submission C. Submission Dates and Times D. Funding Restrictions E. Other Submission Requirements

V. Application Review Information A. Evaluation Criteria B. Review and Selection Process C. Federal Awardee Performance and Integrity Information (FAPIIS)

VI. Award Administration Information A. Selection Notices B. Administrative and National Policy Requirements C. Reporting

VII. Agency Contacts VIII. Other Information

A. Proposers Day B. Frequently Asked Questions (FAQs) C. Collaborative Efforts/Teaming D. Sample ACA Clause

BAA Attachments:

Attachment A: ABSTRACT SUMMARY SLIDE TEMPLATE Attachment B: ABSTRACT TEMPLATE Attachment C: PROPOSAL SUMMARY SLIDE TEMPLATE Attachment D: PROPOSAL TEMPLATE VOLUME 1: TECHNICAL & MANAGEMENT Attachment E: PROPOSAL TEMPLATE VOLUME 2: COST Attachment F: MS ExcelTM DARPA COST PROPOSAL SPREADSHEET Attachment G: PROPOSAL TEMPLATE VOLUME 3: ADMINISTRATIVE & NATIONAL POLICY REQUIREMENTS

HR001121S0030 EDGE 3

PART I: OVERVIEW INFORMATION

Federal Agency Name: Defense Advanced Research Projects Agency (DARPA), Defense Sciences Office (DSO) Funding Opportunity Title: Enhancing Design for Graceful Extensibility (EDGE) Announcement Type: Amendment Funding Opportunity Number: HR001121S0030 Catalog of Federal Domestic Assistance (CFDA) Number(s): 12.910 Research and

Technology Development Dates (All times listed herein are Eastern Time.)

o Posting Date: July 16, 2021 o Proposers Day: June 1, 2021. See Section VIII.A.

o Abstract Due Date: June 9, 2021, 4:00 p.m.

o FAQ Submission Deadline: July 9, 2021, 4:00 p.m. See Section VIII.B.

o Full Proposal Due Date: July 29, 2021, 4:00 p.m.

Anticipated Individual Awards: DARPA anticipates multiple awards for Technical Areas 1, 2, and 3 and a single award for Testing and Evaluation Simulation Engine.

Types of Instruments that May be Awarded: Procurement contracts, cooperative agreements or Other Transactions. Award instruments will be limited to procurement contracts and Other Transactions for proposers whose proposed solution includes Controlled Unclassified Information (CUI)

Agency contacts Technical POC: Bartlett Russell, Program Manager, DARPA/DSO BAA Email: EDGE@darpa.mil BAA Mailing Address:

DARPA/DSO

ATTN: HR001121S0030

675 North Randolph Street Arlington, VA 22203-2114

DARPA/DSO Opportunities Website: http://www.darpa.mil/work-with-us/opportunities

Teaming Information: See Section VIII.C for information on teaming opportunities.

Frequently Asked Questions (FAQ): FAQs for this solicitation may be viewed on the

DARPA/DSO Opportunities Website. See Section VIII.B for further information.

Security: EDGE is an UNCLASSIFIED program. If proposers would like to work with

Controlled Unclassified Information (CUI) or classified information please specify so in the abstract and proposal and refer to section IV.B.4.

mailto:EDGE@darpa.mil https://www.darpa.mil/work-with-us/opportunities?oFilter=DSO https://www.darpa.mil/work-with-us/opportunities?oFilter=DSO

HR001121S0030 EDGE 4

PART II: FULL TEXT OF ANNOUNCEMENT

I. Funding Opportunity Description

This Broad Agency Announcement (BAA) constitutes a public notice of a competitive funding opportunity as described in Federal Acquisition Regulation (FAR) 6.102(d)(2) and 35.016 as well as 2 C.F.R. § 200.203. Any resultant negotiations and/or awards will follow all laws and regulations applicable to the specific award instrument(s) available under this BAA, e.g., FAR

15.4 for procurement contracts.

A. Introduction

The Defense Sciences Office (DSO) at the Defense Advanced Research Projects Agency (DARPA) is soliciting innovative research proposals for developing the tools necessary to create, measure, and test Human Machine Interfaces (HMI) that provide enough situational awareness (SA) of a system’s1 processes and status and of the operational environment so the operator can adapt the system in unexpected situations. The Enhancing Design for Graceful Extensibility (EDGE) program seeks design capabilities that will be fast, quantifiable, repeatable, and manageable enough for HMI concept design, development, and testing to be integrated into the larger system’s design processes. Proposed research should investigate innovative approaches that enable revolutionary advances in science, devices, or capabilities that enable HMI development. Specifically excluded is research that primarily results in evolutionary improvements to the existing state of practice.

B. Background

HMI design has not matured at the same pace as automated and autonomous machines and, as a result, most current interfaces do a poor job supporting the operator’s SA of the machine’s processes, status, and/or operational context. An operator with reduced SA may not adapt to unexpected circumstances, risking catastrophic failure.

Traditionally, designers have assumed that by freeing cognitive resources, operators will be able to adapt to unanticipated and off-nominal situations when necessary. Unfortunately, the drive to limit cognitive workload has too often been at the expense of the operator’s SA of the machine’s processes, status, and operational environment. The result has been that as systems become more automated and autonomous, mission effectiveness increases in expected conditions; yet, in unexpected situations, the failures are proportionally catastrophic to the system’s level of automation, despite the overall reductions in operator workload.2

It was this lack of pilot SA of the aircraft processes and status that contributed to the PT Lion Mentari Airlines (Lion Air) Boeing 737-8 (MAX) accident. “During the accident flight, multiple

1 “System” refers to the combination of machine(s)/platform(s) and the human operator. In the case of EDGE, the machines and platforms include those that are highly automated, autonomous, and/or AI-enabled. An EDGE system comprises a single human operator managing a multi-asset system of the operator’s own ship plus up to four autonomous vehicles.

2 This is known as the Lumberjack effect and has been observed over individual studies and meta analyses. Onnasch, L., Wickens, C. D., Li, H., & Manzey, D. (2014). Human performance consequences of stages and levels of automation: An integrated meta-analysis. Human factors, 56(3), 476-488.

HR001121S0030 EDGE 5

alerts and indications occurred which increased flight crew’s workload. This obscured the problem and the flight crew could not arrive at a solution during the initial or subsequent automatic aircraft nose down stabilizer trim inputs, such as performing the runaway stabilizer procedure or continuing to use electric trim to reduce column forces and maintain level flight.”3 A subsequent National Transportation Safety Board (NTSB) report indicated that multiple simultaneous alerts prevented the crew from efficiently diagnosing the root issue of the Maneuvering Characteristic Augmentation System (MCAS), and current methods for evaluating system function in failure modes are insufficient.4 Managing operator workload is critical when it supports SA, but alone is insufficient for enabling effective operator adaptation to such off-nominal situations.

Likewise, preventing overload by simply limiting information neutralizes years and millions of dollars of operator training by limiting the operator’s span of control. Consider a pilot vehicle interface (PVI) that simplifies enemy entities to red icons on a screen. Without any indication of classification uncertainty or of contributing sensors’ reliability, the simplified HMI diminishes the operator’s role – from a mission commander entrusted to make tactical decisions with strategic-level significance to that of a slow, uninformed actuator.

When an operator has sufficient SA of a system’s process, status, and operational context, they remain our most adaptive asset. It was this kind of adaptive capability that Neil Armstrong demonstrated on Gemini VIII, considered NASA’s first emergency in space. His extensive knowledge of the flight systems and his ability to execute under extreme circumstances resolved a potentially deadly spin, saving the mission and the astronauts’ lives.

Human management of autonomous and AI-enabled systems is, thus, not only a constraint of a moral and ethical mandate of autonomous weapons systems (DoDD 3000.09), but remains the technically and operationally most advantageous path for all AI-enabled systems, as they remain brittle to unexpected situations and contexts. Designs that harden the system against user error fail to take advantage of the fact that humans are the most adaptive asset any system has at its disposal, currently and for the foreseeable future. As we develop autonomous capabilities that allow a single human operator the ability to manage more assets and mission tasks (e.g., planning, control, and maneuver; ISR; targeting; coordinating fires), providing the operator with enough of the right information to establish and support SA becomes a much more complex challenge.

3 Final KNKT.18.10.35.04 Aircraft Accident Investigation Report. PT. Lion Airlines Boeing 737 (MAX); PK-LQP Tanjung Karawang, West Java, Republic of Indonesia 29 October 2018.

4 Sumwalt, R. L., Landsberg, B., & Homendy, J. (2019). Assumptions used in the safety assessment process and the effects of multiple alerts and indications on pilot performance. District of Columbia: National Transportation Safety Board.

HR001121S0030 EDGE 6

While there has been a considerable amount of progress in developing good principles and guidelines for human-centered design,5,6,7,8 the process remains slow, separate from, and lagging behind greater systems design and development processes. As systems become more automated and autonomous and the role of the operator shifts from operating a platform (e.g., flying, driving, navigating) to managing a mission, operator demands shift away from physical tasks and lean more heavily on complex decision making and managing risk across a number of interrelated factors and variables, many of which are abstract. Traditional design tools have not accommodated this shift and struggle to keep pace with automated and autonomous system development due to three major challenges:

1. System-level SA demands are not quantified. It is currently not clear when a given HMI design is “good enough” to meet a system’s requirements. Most modeling capabilities are either too low-level in the cognitive processes they model9 or are task, rather than SA, oriented.10 These methods require lengthy bottom-up construction and offer limited opportunities for reuse.

They also rarely account explicitly for SA requirements. Alternatively, approaches like the Human Autonomous Systems Oversight (HASO) model provide the right categorical elements that must be considered for HMI design but do not generate the quantitative operator performance estimates necessary to integrate HMI design into a larger systems design and development process.

2. There is no way to assemble and deconflict SA-supporting component strategies into system-level compositions. Since human sensory, attentional, and executive bandwidth are limited, HMI designers are faced with the challenge of trying to balance information transmission needs against the need to support operator SA, which includes creating and updating accurate mental models of the system’s processes and status against its service envelope and operational environment. Although there are strategies for transmitting more information per unit data, there is currently no ability to quickly compose these strategies into a unified system-level HMI design. Inserting a single strategy can disrupt or conflict with other design elements; for instance, a spatial auditory display that supports the operator’s awareness of entities in the operational environment may conflict with alerts from the pilot’s platform or be distorted by engine noise. The potential permutations of determining when and how to present information across sensory space and time are more than development budgets and schedules can explore. Moreover, while there have been advances in understanding how to support SA (e.g., studies in decision sciences and demonstrations that show information that is situated and forward better prepares an operator to adapt to changing situations), there is no way currently to inject these strategies into a design process that weighs multiple, sometimes competing, demands.

5 Endsley, M. R. (2016). Designing for situation awareness: An approach to user-centered design. CRC press.

6 Wickens, C. D. (2008). Multiple resources and mental workload. Human factors, 50(3), 449-455.

7 Woods, D. D. (2018). The theory of graceful extensibility: basic rules that govern adaptive systems. Environment Systems and Decisions, 38(4), 433-457.

8 Oury, J. D., & Ritter, F. E. (2021). Building Better Interfaces for Remote Autonomous Systems: An Introduction for Systems Engineers.

9 Such as Soar and ACT-R cognitive architectures, among others.

10 Examples of task-oriented models are Improved Performance Research Integration Tool (IMPRINT) and, Goals, Operators, Methods, Selection Rules or GOMS) models.

HR001121S0030 EDGE 7

3. Meaningful testing requires operational realism. It is almost impossible to discover design flaws without user testing. Current practice is good at examining “surface” level issues with HMI, but less capability exists to explore how well the HMI supports the operator’s handling of challenges from emergent properties of complex systems. Often, the machines must be built before the HMI can be evaluated. Moreover, human perception and cognition are affected by environmental pressures such as motion, noise, and stress. Realistic test simulation environments are too costly and rigid for early concept evaluation, design prototyping, and exploration. As a result, there are currently limited opportunities for early HMI concept testing and exploration against system complexity and environmental stressors.

In response to these limitations, and in order to manage the gap between increasing system complexity and subsequent lack of operator SA, engineers have defaulted to hardening the system against user error. Yet, this strategy exacerbates the problem, making systems, even with their human operators, more brittle and incapable of adaptation.

EDGE will create HMI design tools capable of integrating with a larger systems design and development process. By prioritizing and orienting these tools towards quantifying, supporting, and testing SA, rather than on reducing cognitive load at the expense of SA, EDGE will help designers build HMI systems that allow operators to not just monitor autonomous systems but also adapt their use to meet the needs of unanticipated situations.

C. Program Description/Scope

EDGE’s vision is to develop a new class of HMI design tools that integrate into larger systems design and development processes by:

Developing models that quantify the SA demand of a given system to accurately predict operator performance when using a system before the system has been developed

Creating composable design methods to incorporate and deconflict multiple SA and cognition supporting techniques into a unified HMI for generating more mature designs, more quickly

Building a reconfigurable HMI “breadboard” for rapidly testing design prototypes in ecologically realistic environments

While HMI development includes many elements from new display technologies to actuators and ergonomics, EDGE focuses on the elements that most affect the operator’s ability to adapt to unexpected conditions. Considering John Boyd’s OODA Loop, EDGE is predominantly focused on methods that support Observation, Orientation, and Decision-making, and less on how the HMI supports Action. This means new methods of operator control (e.g., gesture and voice commands) are out of scope.

Domains of interest include land, surface, undersea, air, and/or space. Missions in the cyber domain and information operations are out of scope.

D. Program Structure

EDGE is a four-year research and development (R&D) effort comprising three Phases, as illustrated in Figure 1.

HR001121S0030 EDGE 8

Figure 1 – High-Level Program Schedule

EDGE will use a phased-acquisition approach. Proposers are asked to provide detailed pricing and a Statement of Work (SOW) for the Phase 1 base effort, a detailed SOW and separately priced option for Phase 2, and Rough Order of Magnitude (ROM) cost estimate and draft SOW for Phase 3. Proposals that do not include a separately priced option for Phase 2 and ROM cost estimate and draft SOW for Phase 3 may be deemed non-conforming and removed from consideration. Near the end of Phase 2, the Government may issue Proposal Instructions to Phase 2 performers requesting final SOWs and revised cost proposals for Phase 3. Competition for Phase 3 will be limited to only Phase 2 performers.

Evaluation of Phase 3 proposals will be based on criteria to be specified in the Phase 3 proposal requests. The Phase 3 evaluation criteria will be consistent with the evaluation criteria in this solicitation, but may be tailored to the Phase 3 requests for updated proposals. Phase 3 proposal evaluations will be conducted through a scientific and technical review process in accordance with Section V.A and V.B. The Government reserves the right to issue a new solicitation for Phase 3 with a new award instrument if programmatic circumstances dictate.

Participation in Phase 2 does not guarantee funding in Phase 3; progression to the next phase will be contingent on evaluation of Phase 3 proposals and availability of funds.

Additional details are outlined in the sections below.

Phase 1 (Base) is an 18-month effort to develop tools that turn component-level capabilities into system-level designs.

Phase 2 (Option 1) is an 18-month effort that builds on work in Phase 1 by refining Phase 1 tools to adapt them to operational contexts.

Phase 3 (ROM cost estimate and draft SOW) is a 12-month effort to refine the Phase 2 tools and demonstrate an optimized, integrated HMI design process.

The Government may elect to exercise the option(s) on the award(s) of the selected performer(s) based on progress made meeting the phases’ goals and metrics and the candidate technologies’ potential to meet the subsequent phase’s program goals and metrics. The Government retains the right to award all, some, one, none, or portions of the proposed options to support promising further technology developments. Participation in any given phase does not guarantee funding in a subsequent phase; progression to the next phase will be contingent on performance and

HR001121S0030 EDGE 9

availability of funds. Additional details on the objectives of each phase are included in the Technical Area (TA) (below and in Section I.E.) and Metrics (Section I.F.) descriptions.

EDGE comprises three TAs:

TA1: Quantify the Situational Awareness Demand: Develop models that quantify before a given system has been developed the SA demand of that system to accurately predict operator performance when using the system.

TA2: Composable Design: Create composable design methods to incorporate and deconflict multiple SA and cognition supporting techniques into a unified HMI for rapid prototyping.

TA3: HMI Breadboard: Build a live and virtual reconfigurable HMI “breadboard” for rapidly testing design prototypes in ecologically realistic environments.

Section I.E describes the TAs in more detail.

Since TA1 capabilities will be important for meeting TA2 outcomes, proposers must propose to both TA1 and TA2 with combined technical and cost proposals. Proposals that cover only TA1 or only TA2 may be considered non-conforming. Proposers who submit combined TA1/TA2 proposals may also submit separate technical and cost proposals for TA3. A proposer can be selected for both a combined TA1/TA2 and a separate TA3. TA1/TA2 proposers should separate tasks and costs by TA, and Phase 2 Options should be separated by TA (i.e., Option 1 is Phase 2 TA1, Option 2 is Phase 2 TA2).

The EDGE program seeks to create general-purpose tools that will support HMI development for all operational domains. As such, all TAs will have to demonstrate their capabilities in more than one domain. TA2 will choose two of the following five possible domains in which to develop and demonstrate the generalizability of their capabilities: land, surface, undersea, air, and/or space. For each domain, the Government will define mission requirements and architectures for systems that will include multiple mission tasks the operator must manage. These may include planning and re-planning, ISR, sensor tasking and management, target identification and tracking, target allocation, vehicle control and maneuver, battle damage assessment, and/or others. TA1 performers will have to demonstrate their ability to predict operator performance for each Government defined system, including those in domains in which other TA1/TA2 teams are working (maximum 5 systems per challenge event). TA3 will demonstrate the agility and reconfigurability of their Breadboard by integrating and testing the designs of all TA2 performer teams (1 design per domain for all TA2 selected domains).

DARPA is committed to reproducibility of studies and methods developed under its programs. In support of this ideal, TA1/TA2 teams will be required to pre-register their studies, methods, and hypotheses11 and should clearly delineate within the proposal which proposed studies and methods will be exploratory and which will be confirmatory.

11 See pre-registration sites for instructions on how to pre-register a study. For example: https://help.osf.io/hc/en-us/articles/360019738834-Create-a-Preregistration. For more information about the purpose of pre-registration see https://www.sciencemag.org/news/2018/09/more-and-more-scientists-are-preregistering-their-studies-should-you https://help.osf.io/hc/en-us/articles/360019738834-Create-a-Preregistration https://help.osf.io/hc/en-us/articles/360019738834-Create-a-Preregistration https://www.sciencemag.org/news/2018/09/more-and-more-scientists-are-preregistering-their-studies-should-you

HR001121S0030 EDGE 10

To facilitate these evaluations, DARPA is also soliciting proposals for a Test and Evaluation (T&E) Simulation Engine in this BAA. The role of the T&E Simulation Engine is to create off-nominal virtual test scenarios (“challenge scenarios”) to evaluate performer performance against program metrics. As such, proposers to the T&E Simulation Engine are not permitted to perform on any other TA either as a prime or as a subcontractor. Should a proposer submit proposals for the T&E Simulation Engine and for one or more TAs (as a prime or subcontractor to either team), they may only be selected for, at most, either the T&E Simulation Engine role or the TA role(s).

To evaluate each TA’s progress, the program will pose a series of Government-designed challenge events approximately every six months (see Section I.F). DARPA, along with EDGE’s Government Independent Validation and Verification (IV&V) team will generate challenges specific to each TA2 domain, including a Concept of Operations (CONOPS), set of requirements, and system architecture for TA1/TA2 and TA3 and mission objectives for the T&E Simulation Engine’s challenge scenarios. The challenges are designed to build on each other to enhance overall capabilities over the course of the program.

Proposers should strive to provide a clear understanding of the cost, risk, and organizational expertise to be used within each proposed effort.

E. Technical Area Descriptions

TA1: Quantify the Situational Awareness Demand.

This technical area will develop models that quantify the SA demands imposed on an operator by a given system before it has been built. DARPA will supply three inputs: (1) a CONOPS, (2) a set of system requirements and mission tasks, and (3) a draft system architecture, including system sub-components and details such as expected accuracy, reliability, and service envelopes. TA1 performers will generate a quantified estimate of the system’s SA demands on the operator and a list of design priorities (e.g., which of the SA requirements are most critical for predicting the types and magnitude of performance failures) for HMI designers.

In order to achieve this goal, performers should determine the appropriate level of detail needed to model operational understanding and consider methods that are faster to develop and more predictive than overly-detailed, bottom-up modeling approaches. TA1 performers should pay particular attention to the cognitive processes and demands necessary to create and update accurate mental models of the system and its components.

The output of TA1 is expected to be a model that quantifies the SA demands imposed by system design choices, using only the CONOPS, system requirements, and draft system architecture. TA1 can assume these systems, regardless of domain, will include a forward human operator managing a multi-asset system. The operator’s role will be mission commander managing up to four vehicles in addition to the operator’s own ship. Rather than create a bespoke model for one type of system, TA1 performers should extract, through experimentation, a set of underlying common axes that drive SA demands that would help a designer identify when the demands of an architecture become impossible to manage. Candidate axes may include decision timelines, system or environmental

HR001121S0030 EDGE 11

uncertainty, system or environmental complexity, points of interaction with the world (e.g., sources of stochasticity), and system recursion. By approximating a notional system along these axes, designers should be able to approximate the SA demands of systems that do not yet exist, set performance goals for a system’s functional subcomponents (e.g., sensor accuracies, re-planning speed), and update those estimates as the system is refined.

TA1 technical proposals should include the following:

A description of the kind of model the team will use, including level of description, representation, and the strengths and weaknesses of the proposed modeling approach

A candidate list of common axes that drive SA demand and their theoretical basis A clear description of how the work on the program will derive or refine these axes empirically, including data, methods, and ways the team will assess the model’s accuracy (confirmatory and disconfirmatory methods)

Expected sources of error, how error should be aggregated, and how the proposer will determine what is an acceptable margin of error by domain

A description for how a designer would use the envisioned end-product to approximate the SA demands of a new system in a new domain

DARPA will evaluate TA1 performers on how well their models accurately predict SA demands when tested in the live TA3 Breadboard against the baseline HMI provided by the T&E Simulation Engine. Results of the evaluation will be used to help determine funding for subsequent phases.

TA2: Composable Design.

The possible combinations of information transmission schemas and strategies (e.g., cross-modal cueing, attention management and boost, decision aiding, re-orientation support) far outstrip the available time and resources for HMI development, leaving designers to guess which SA support mechanisms would be most useful. Methods are required that aid the HMI designer in choosing the best combination of interface strategies and components for the system at hand and quickly generating those designs.

TA2 proposers should describe how their approaches will speed HMI composition and incorporate display strategies and components that support operator SA. Both visual and non-visual (e.g., audio, tactile, proprioceptive) display technologies are encouraged.

Approaches should be able to do the following to speed HMI composition:

Negotiate quickly across multiple, oftentimes competing design elements to rapidly identify and begin testing with more mature design concepts

Integrate interface components and SA support strategies into a unified design (or set of initial candidate designs)

Speed the development of HMI components from concepts to reduce the time it takes to generate (code, layout) and revise functional design prototypes for early evaluation

HR001121S0030 EDGE 12

Approaches should address the following to describe how they will incorporate display strategies and components that support operator SA:

System processes. Since system operators need to have a robust understanding of how the system works but are not experts in modern complex systems, DARPA is seeking techniques that make hierarchical complex systems comprehensible despite their complexity. Effective abstractions would help the operator anticipate how the system’s components work as a unified system. These strategies should provide the foundation for future work, including helping the operator manage contextual challenges in Phase 2.

System status. To manage the system, the operators need to maintain awareness of the system status across operational states/phases and the status of the system’s component interactions (e.g., how the status of one system module may affect that of another). To help the operator maintain awareness, DARPA is seeking techniques that provide the operator with more information per unit data transmitted. These techniques should address issues such as determining how to present information across sensory space and time, managing orientation and re-orientation while multitasking, and facilitating memory recall, among others.

System processes and status against environmental and adversarial conditions. The system’s processes and status will change against different environments and when competing against adversary tactics (e.g., jamming, military deception). DARPA is interested in HMI design methods that will help the operator manage the system against these changing dynamics.

TA2 proposers are expected to develop designs for systems in at least two of the five following DoD relevant domains: land, air, ground, surface, and/or undersea. Proposers are encouraged to choose domains that demonstrate the generalizability of their approaches and describe in their proposals how their design tools will account for the particular challenges of each domain.

TA2 performers will develop implementations of unified interface design concepts as software components that interact with TA3 (starting at the end of Phase 1), using an application programming interface (API) developed by TA3 (below). TA2 performers are expected to have a local HMI testing and demonstration environment or capability other than what TA3 provides to facilitate their own testing and demonstrations throughout Phase 1 and between challenge events. These can be separate environments for each domain proposed.

To determine which team(s) will continue to subsequent phases, the TA2 design tools will be evaluated based on the speed by which they develop new designs and on the efficacy of the designs they generate. Efficacy will be evaluated in terms of (1) how well the designs support operator SA (system processes, status, and operational environment) and (2) how well the operator adapts to unanticipated situations.

TA3: HMI Breadboard

HR001121S0030 EDGE 13

The stressors of operational environments affect cognitive processes, yet realistic simulation environments that approximate those stressors are often limited to bespoke, rigid training platforms. Advances over the last decade in immersive technologies have decreased the cost and increased the accessibility and sophistication of immersive experiences, making simulations more realistic without costly, high-maintenance, mechanical hardware systems, as an example. Additionally, trends towards open and modular architectures allow reconfigurability and enable capabilities such as context-sensitive interfaces. DARPA is interested in ways to make realistic HMI testing and exploration environments that reconfigure quickly enough to fit within an Agile software development sprint (2-4 weeks). The goal for TA3 is to create a low-cost, rapidly reconfigurable, immersive, open source HMI kit with an API that connects interface hardware, an immersive environment, and multimodal presentation schemas to test simulations (vehicles, environments, and scenarios). This “HMI Breadboard” should comprise an online, virtual version that enables rapid throughput testing of early design concepts to reach deployed end-users and a live version that boosts realism and operator performance at low cost.

Developments in instrumentation provide the ability to link behavioral and cognitive events such as observation and orientation to system events to identify problem areas and guide design revisions. Diagnostic strategies are important to TA3; such methods should– in the spirit of a robust, quickly reconfigurable platform that enables quick assessments– avoid equipment that imposes lengthy set up, calibration, or heavy post-test data processing and analysis. Diagnostic methods should link scenario and system events and HMI features to very specific behaviors or cognitive events/states. Neuroimaging equipment like electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) of workload are too non-specific, are difficult to set up, and require lengthy post processing with any breadboard reconfiguration. Proposers wishing to use psychophysiological measures should make a convincing argument for how those measures will be employed to detect specific features of cognition and/or performance and why those features would provide diagnostic information above behavioral measures, like button presses and eye-tracking, for instance.

TA3 proposers should describe how they plan to do the following:

Approximate any specified EDGE domain. In the spirit of being a “breadboard” kit rather than a high-fidelity gaming environment, the priority should be to quickly approximate general layouts and constraints of the HMI experience, rather than try to create replicas of the environments. EDGE’s domains of interest include land, surface, undersea, air, and space.

Create a capability for rapidly reconfiguring testing environments. Both live and virtual versions should implement a common open architecture backbone for HMI developers and an API for TA2 integration by the end of Phase 1. The API should include:

Ways for TA2 performers to quickly integrate and iterate designs for testing. This API should include configuration specifications necessary for HMI developers (TA2) to control the spatial and temporal layout of audio, HR001121S0030 EDGE 14 visual, and other forms of data presentation within the breadboard. TA3 will delimit configuration and control options for its reconfigurable environment (e.g., hardware for visual and other sensory displays) and dynamically inform TA2 of T&E Simulation events.

A way for TA2 to inform TA3 of relevant events and operator actions taken, for the purpose of assessing operator performance and cognitive processes.

Mechanisms for connecting and tracking operator-system interactions, such as means for collecting operator actions and system events and linking simulation events to operator behavior.

Methods to send T&E Simulation events and scenario information to TA2 (e.g., sensor information TA2 software can use to manage audio/visual timing and presentation).

Develop a live breadboard to approximate the immersive experience of conducting the mission. The live breadboard should include:

Reconfigurable hardware for visual displays, auditory displays, and other sensory displays (e.g., location, size, timing).

Estimated specifications of the test environment (e.g., physical footprint, CPU requirements).

Mechanisms for the operator to control the simulated systems. While the focus of the program is not on actuators (input mechanisms, etc.), some methods for controlling the systems is required. Proposers should consider existing capabilities that approximate different operational domains, offer a variety of actuator capabilities, and/or include ways to integrate new actuators to support TA2 needs.

Develop a virtual breadboard to increase the speed of early HMI testing.

Increasing the accessibility of test environments for early HMI concept testing will increase the speed of evaluations and, in some cases, enable testing with remote operator/end-user populations. To support this end, proposers should:

Provide a means by which test participants can be consented and personally identifiable information (PII) protected in compliance with Human Subjects Research (HSR) protocols.

Explain how their virtual breadboard can provide HMI approximations in terms of equipment required and processing requirements, with a focus on capabilities commonly available at virtual research subjects’ home stations.

Specify what meaningful behavioral and event-marked metrics can be generated from an online environment.

HR001121S0030 EDGE 15

TA3 will be responsible for providing the test participant populations throughout the program.

For costing purposes, TA3 should assume at least 30 live and 90 virtual participants per challenge and provide detail in the Cost Volume to allow a decrease or increase in participant testing (recruitment, incentives, etc.) based on program needs. Live and virtual populations should be different samples. Populations should be representative of current and future operators and assume a junior officer as the operator.

TA3 proposers should assume testing will be considered HSR and plan for the Independent Review Board (IRB) and secondary Human Research Protection Office (HRPO) reviews necessary for government sponsored HSR in their cost and schedule. Performers will be required to submit IRB approved protocols to HRPO for secondary review no later than 1 month after award. No data collection can begin prior to HRPO approval. To meet this deadline, proposers should submit protocols to their local IRB for initial approval prior to proposal submission.

Test & Evaluation (T&E) Simulation Engine.

The T&E Simulation Engine will provide simulated environments and extensible systems (machines) needed to create challenge scenarios. Proposers should already have robust existing capability in simulating the behavior of military-relevant platforms in outdoor operational environments from which multi-vehicle control systems and challenge scenarios may be constructed. DARPA requests a completely open-source simulation environment and simulated entities that are Robot Operating System (ROS) based to maximize compatibility and integration with other program performers and beyond.

DARPA has a strong preference for solutions that do not impose intellectual property (IP) restrictions.

Competitive proposers will have large existing libraries of environments, vehicles, sensors, and vehicle control, so that new development tasks can focus on constructing scenarios from that existing material. The T&E Simulation Engine team should be able to provide graphical representations of the vehicles in environments but should use methods that do not impose high computational demands on TA3 solutions, whether virtual or live.

Vehicle simulations. Vehicles should include those that operate in land, surface, undersea, air, and space domains and should have the ability to simulate complex autonomous behaviors (Endsley & Kaber’s Levels of Automation 3-612). Proposers should have the ability to construct new hypothetical vehicles and teams of vehicles on a schedule that coincides with the pace of the program.

Environment simulations: Competitive proposals will have existing capability for simulating land, surface, undersea, air, and space domains. The simulated environments should be expansive and complex enough to support 30-45 minute challenge scenarios.

12 Endsley, M.R., & Kaber, D.B., (1999) Level of automation effects on performance, situation awareness, and workload in a dynamic control task. Ergonomics, 42(3), 462-492.

HR001121S0030 EDGE 16

Challenge scenarios: All simulated systems, regardless of domain, will include a common unit of reference: a single, forward human operator managing a multi-asset system. The operator’s role will be mission commander managing up to four vehicles in addition to the operator’s own ship. The operator will not be teleoperating the other vehicles; the operator’s role will be to manage highly automated behaviors across the team at various levels of autonomy. These vehicles may be homogeneous or heterogenous in terms of form, capabilities, automated or autonomous control systems, and payload. The overall system may include various kinds of sensors, data processing, data fusion, and inference capabilities intended to aid the operator, distributed evenly or unevenly across the managed vehicles, as well as control, planning, communications management, and other system-management functions.

To measure how well the HMI solutions developed by TA2 support operator SA, the simulated scenarios must provide a way for the DARPA team to inject system failures and unexpected contextual changes. DARPA has a strong preference for physics-accurate simulators that can explore complexity and emergent behaviors from platforms interacting with each other and the environment as opposed to simulators emphasizing high-fidelity graphics. T&E Simulation Engine proposers should describe their ability to inject system failures and unexpected contexts or environmental features into scenarios.

For example, the challenge scenario may need to simulate a sensor on one platform returning a high number of false positive readings or perception elements detecting the presence of civilian populations where there were none expected. A single challenge event may include various versions of the same scenario; therefore, the environments should include modular elements that can change easily (e.g., buildings and targets can be moved easily), so the scenario can be iterated but present the same basic task.

The T&E Simulation Engine performer will be responsible for providing a baseline HMI (interfaces for operator control of simulated systems) for TA1 predictions and for comparing the improvement of TA2 designs. Ideally, these baselines would be existing HMIs that are minimally modified to accommodate a system developed for each challenge event. Additional visualization of the simulation will be required for managing the scenarios and for evaluators to observe mission effectiveness.

The T&E Simulation Engine performer will work with DARPA and the IV&V team to draw challenge scenarios for each domain from major programs of record or from autonomous system initiatives across the Services and research and development (R&D) laboratories. For example, if a TA2 team proposes to work in the air and surface domains, the T&E Simulation Engine team might pull SA challenge scenarios from Next Generation Air Dominance and Project Overmatch, respectively. In all cases, challenge scenarios will be tied to a single common unit of study: a single forward operator managing a multi-entity control system. Challenge scenarios may include various forms of perception and sensing (e.g., automated target recognition) and control autonomy (e.g., route planners, obstacle avoidance). The purpose of the challenge scenarios is to (1) connect the testing to real needs that the Services and other R&D programs are facing rather than toy problems, (2) facilitate transition, and (3) demonstrate that the resulting tools will be general purpose and not just able to support one type of system or domain.

T&E Simulation Engine will be required to pass the following information types to TA2

HR001121S0030 EDGE 17

and TA3 performers:

Environmental information such as terrain, structures, and weather Vehicle position, speed, trajectory Sensor “readings” at a component level Event metadata such as object identifiers and entity ground truth And other metadata as necessary

For costing purposes, T&E Simulation Engine should assume challenges every six months, across as many as all five simulation domains in Phase 1. It is likely that TA1/TA2 teams may propose to work in the same domains, so it may be fewer than five; cost proposals should allow for adjustments after TA1/TA2 team(s) are selected, depending on the domains proposed by selected TA2 teams. T&E Simulation Engine will follow the same phased program structure as TAs 1-3.

F. Schedule/Milestones

The EDGE performers will be evaluated using a number of milestones and metrics enumerated below. Attaining the milestones and metrics for a given phase does not guarantee transition into the next phase of the program. DARPA will also assess efforts on their expected ability to attain subsequent milestones. The program’s phases will challenge the performers to demonstrate maturity of their methods and tools consistent with engineering standards by producing consistent results from a generalizable, manageable, and repeatable process.

Challenge Events

In order to judge the progress of the technologies developed in TA1, TA2, and TA3, there will be challenge events approximately every six months. For costing purposes, proposers should reference the TA3 metrics for integration timelines for each event and assume three days of testing per challenge. TA3 proposers should estimate costs for three TA2 teams at the end of Phase 1, two TA2 teams for Phase 2, and one TA2 team for Phase 3. These costs should be itemized such that costs may be adjusted if more or fewer TA2 teams are selected. These challenge events consist of a challenge system, against which the TAs will build capability, and a challenge scenario run by the T&E Simulation Engine team against which the TAs will demonstrate efficacy. The challenge systems and scenarios will be developed by DARPA, the Government IV&V team, and the T&E Simulation Engine team. For each event DARPA will specify a challenge system: the CONOPS, the system requirements, and a draft machine architecture design for all performers. The T&E Simulation Engine team will provide the baseline (existing) HMI for each challenge system. For each challenge event:

TA1 will predict operator performance in terms of the types and magnitude of operator errors

TA2 will compose a set of HMI designs to support the system TA3 will reconfigure both the live and virtual breadboards to generate an approximate experience of the operator environment

More specific outcomes for each challenge and TA can be found in Tables 2 and 3 below.

HR001121S0030 EDGE 18

Each challenge event will focus on a particular aspect of SA, with each performer being expected to build upon their advances from the previous challenge. The focus of each of the challenges in Phase 1 and Phase 2 are listed in Tables 2 and 3, respectively. All challenge events will include an event, failure, or condition considered “off-nominal” to test operator responses to unexpected events. The nature of these off-nominal events will differ in accordance with the focus of the challenge event.

Phase 1 development is focused on moving from component level solutions to whole-system solutions; as such, the challenges will focus on elements related to understanding the system’s processes and status as a whole. Since Phase 2 development is focused on understanding how the system changes against context, Phase 2 challenge events will focus on off-nominal conditions related to the environment or opponents. In Phase 3, both types of events will occur.

Phase 3 is focused on maturing tools developed in Phases 1 and 2. As such, Phase 3 challenge events will ask remaining performer(s) to demonstrate that an engineer of the Government’s choosing is able to use the tools developed on the program to predict, develop, and test candidate HMIs. The designs generated by the Government engineer should enable comparable operator performance to those developed by the performer team as demonstrated in the final challenge event and Capstone Demonstration (see Table 1 schedule).

Table 1. Program meetings and challenge event schedule Phase Month Event Location

1 Program Kickoff & Technical Exchange meeting Virtual

6 PI meeting Virtual

10 Challenge Event: System Processes Performer site

14 PI meeting Arlington, VA

Phase 1 From components to systems

16 End of Phase Challenge Event: System status and component interactions TBD

20 Phase 2 Kickoff & PI meeting Arlington

VA

22 Challenge Event: System processes and status against environmental factors TBD

26 PI meeting Arlington

VA

28 Challenge Event: System processes and status against adversary factors TBD

32 PI meeting Arlington

VA

Phase 2 System status and context

34 End of Phase Challenge Event: IV&V-defined challenge TBD

38 Phase 3 Kickoff & PI meeting Arlington

VA

40 Challenge Event: Demonstrate tool manageability TBD

Phase 3 HMI design tool maturity

44 PI meeting Arlington

HR001121S0030 EDGE 19

TA1 will submit performance predictions to the Government IV&V team prior to all test events.

All TA1 performers will predict performance against T&E Simulation Engine provided baseline HMI designs and against experimental designs for all TA2 performers. TA2 performers’ designs will be tested at the end of each Phase in the TA3 Breadboard.

TA2…