HR001124S0006.pdf

Broad Agency Announcement Objective Prediction of Team Effectiveness via Models of

Performance Outcomes (OP TEMPO)

BIOLOGICAL TECHNOLOGIES OFFICE

HR001123S0055

November 2, 2023

HR001123S0055, OP TEMPO

TABLE OF CONTENTS

PART I: OVERVIEW INFORMATION

PART II: FULL TEXT OF ANNOUNCEMENT

1. Funding Opportunity Description

1.1. Program Overview

1.2. Program Structure and Technical Approach

1.3. Program Metrics, Milestones and Deliverables

1.4. General Requirements

2. Award Information

2.1. General Award Information

2.2. Fundamental Research

3. Eligibility Information

3.1. Eligible Applicants

3.2. Organizational Conflicts of Interest

3.3. Cost Sharing/Matching

4. Application and Submission Information

4.1. Address to Request Application Package

4.2. Content and Form of Application Submission

4.3. Funding Restrictions

4.4. Other Submission Information

5. Application Review Information

5.1. Evaluation Criteria

5.2. Review of Proposals

5.2.1. Countering Foreign Influence Program (CFIP)

6. Award Administration Information

6.1. Submission Status Notifications

6.2. Administrative and National Policy Requirements

6.3. Reporting

6.4. Electronic Systems

6.5. DARPA Embedded Entrepreneurship Initiative

7. Agency Contacts

8. Other Information

8.1. Proposers Day

8.2. University Funding

9. APPENDIX 1 – Volume II checklist

PART I: OVERVIEW INFORMATION

Federal Agency Name – Defense Advanced Research Projects Agency (DARPA), Biological Technologies Office (BTO)

Funding Opportunity Title – Objective Prediction of Team Effectiveness via Models of Performance Outcomes (OP TEMPO)

Announcement Type – Initial Announcement Funding Opportunity Number – HR001123S0055 North American Industry Classification System (NAICS) – 541714 Assistance Listing Number (ALN) – 12.910 Research and Technology

Development Dates o Posting Date: November 2, 2023 o Proposal Abstract Due Date and Time: December 1, 2023, 4:00 PM ET o Full Proposal Due Date and Time: January 31, 2024, 4:00 PM ET o BAA Closing Date: January 31, 2024 o Proposers’ Day: November 7, 2023 https://sam.gov/opp/a6291f6feedf4914b3fa107fed6e69ab/view

Concise description of the funding opportunity – Current DoD team training assessment is limited, relying on rudimentary checklists and post-hoc, subjective evaluations of performance that will not be able to scale to future team training requirements. Objective Prediction of Team Effectiveness via Models of Performance Outcomes (OP TEMPO) will explore whether bio-behavioral signals can be used as valid, objective signatures of team performance. OP TEMPO focuses directly on the fundamental research necessary to identify bio-behavioral signatures associated with team cohesion, collaboration, and performance. As such, OP TEMPO will develop the foundation for novel approaches to understand, model, and assess team dynamics during training.

Anticipated individual awards – Multiple awards are anticipated.

Types of instruments that may be awarded – Procurement contract, cooperative agreement, or other transaction for research.

Agency contact o Technical POC: Joeanna Arthur, Program Manager, DARPA/BTO o Contracting Officer: Shane Lomelin, DARPA/CMO o E-Mail: OPTEMPO@darpa.mil o Mailing Address:

DARPA/BTO

ATTN: HR001123S0055

675 North Randolph Street Arlington, VA 22203-2114 https://sam.gov/opp/a6291f6feedf4914b3fa107fed6e69ab/view mailto:OPTEMPO@darpa.mil

PART II: FULL TEXT OF ANNOUNCEMENT

1. Funding Opportunity Description

This publication constitutes a Broad Agency Announcement (BAA) as contemplated in Federal Acquisition Regulation (FAR) 6.102(d)(2) and 35.016 and 2 CFR § 200.203. Any resultant award negotiations will follow all pertinent law and regulation, and any negotiations and/or awards for procurement contracts will use procedures under FAR 15.4, Contract Pricing, as specified in the BAA.

DARPA is soliciting innovative proposals to determine whether bio-behavioral signatures (e.g., combinations of neural signals, physiological signals, and/or communication patterns) of team performance exist and are generalizable across DoD teams. The envisioned approach will involve combining individual biological or behavioral signals into multimodal bio-behavioral signatures of team performance capable of facilitating future strategies to understand, model, and assess team dynamics during training. A successful proposal will clearly address four (4) research tasks: 1) identification and 2) characterization of candidate bio-behavioral signatures of DoD team coordination dynamics; 3) validation of these signatures using predictive models of team training performance; and 4) exploration of signature generalizability within two additional DoD use cases.

Specifically excluded are proposals that involve:

1. Identification and detection of molecular biomarkers

2. Approaches that use invasive or surgically implanted sensors

3. Approaches that include animal research

4. Approaches that cannot be reasonably conducted on-site at military training locations

(e.g., use of functional Magnetic Resonance Imaging systems)

5. Approaches that solely focus on performer-proposed lab-based tasks, rather than

DARPA-specified use cases and data collection environments

6. Approaches that involve human-machine teaming

7. Approaches that focus extensively on individual-level performance and cannot be translated/adapted to team-level performance

8. Approaches that only incrementally improve upon the current state-of-art in teaming optimization and physiological and/or behavior signatures of team performance

Proposals involving any of the approaches described above may be deemed non-conforming and may be rejected without further review.

1.1. PROGRAM OVERVIEW

Current DoD team training assessment is limited, relying on high-level checklists and post-hoc, subjective evaluations of performance. Tracking bio-behavioral signals (e.g., neural signals, physiological signals, and communication patterns) as objective signatures of team performance provides an opportunity to enhance DoD team training assessment; however, fundamental questions of signature validity and generalizability across diverse team training settings need to be answered.

Limitations of the current state-of-the-art in bio-behavioral assessment of team performance include the following:

• The generalizability of bio-behavioral signatures reported to date has not been sufficiently explored.

• Exploration of multi-variate signatures that include multiple biological and/or behavioral signal inputs has been limited.

• No common, ecologically-valid framework exists from which to identify and compare signatures using real world teams and task environments (i.e., much of the research to date has focused on a wide variety of lab-based “toy” tasks, primarily using convenience samples of university students).

• No coherent, theoretically-grounded framework exists for measuring and interpreting empirical results.

The OP TEMPO program will determine whether bio-behavioral signatures of team performance exist and are generalizable across DoD teams. OP TEMPO will focus on four research tasks: 1) Identify candidate bio-behavioral signatures that correlate to DoD team performance; 2) Characterize signatures to assess their reliability and variability; 3) Develop predictive models of team performance and use them to validate signatures; and 4) Explore Generalizability of the validated signatures by testing the models within alternative DoD use cases.

Using existing sensor suites, performers will search for candidate bio-behavioral signals that correlate with team performance metrics, and characterize these signals in real DoD laptop-based team training virtual environments. Candidate signals, stand-alone or in combination, will be validated as signatures of team performance by assessing the predictive potential of models that use the signatures as inputs. After signatures have been validated in a specific DoD training environment (Use Case 1), generalizability will be explored by testing the signatures in two additional DoD training environments (Use Cases 2 and 3). By successfully developing bio-behavioral signatures of team performance, OP TEMPO will enable a completely new way to understand, model, and assess team dynamics during training.

The DoD simulation-based training environments and scenarios for each of the program use cases will be provided by a government-led Test and Evaluation (T&E) team, and all performer teams will be required to collect data within these common testbed environments during designated data collection events at military training installations with DARPA-provided active-duty military personnel as participants.

The T&E team will establish objective team performance metrics for each use case based on a Critical Task Analysis (CTA), and will also establish a ground truth database using the specified metrics to stratify both individual and team performance (e.g., Low, Medium, High) across a series of carefully defined tasks and subtasks for each use case. In order to accomplish this, the T&E team will evaluate performance across novice, intermediate, and expert teams, and will also assess performance over time for teams progressing through training. The databases and detailed descriptions of the metrics (i.e., how they are computed and logged) will be provided to all performers. The ground truth metrics will be automatically captured by the training environments for each use case and scenario, and will be provided to performers along with time synced meta-data during each data collection event. This will provide all performers with common data sets from which to identify, characterize, and validate bio-behavioral signatures, and will provide a mechanism for head-to-head comparisons between performer solutions and experimental results.

Use Case 1 will focus on 4-person U.S. Marine Corps (USMC) Fire Support Team (FiST) training. The T&E team will conduct an in-depth CTA to decompose FiST training and readiness (T&R) mission essential tasks (METs) into discrete, individual and team-focused subtasks, and will design and develop customized training scenarios that leverage a current computer-based USMC FiST training environment. The T&E team will develop a standard framework for team training performance assessment, based on objective training performance metrics (e.g., time required and accuracy to complete each subtask). The T&E team will then use this framework to assess team performance during performer data collection events and share the assessments with performers as ground truth data. This same process will be repeated for Use Cases 2 and 3, which will be selected by DARPA and presented at the Phase 1 Kickoff Meeting based on lessons learned during the T&E pre-kickoff activity (See Section 1.2 for further details).

Notional Use Cases include U.S. Air Force air battle management (ABM) team training and U.S.

Navy submarine navigation (SPAN) team training.

The OP TEMPO program will develop four primary transitionable components: 1) protocols for the identification of bio-behavior signatures (performer provided); 2) databases of raw signals captured during team training (performer provided); 3) predictive models and algorithms for validating prospective bio-behavioral signatures (performer provided); and 4) training assessment frameworks for systematic capture of team performance results for each use case (T&E team provided). See Section 1.3 for further details.

1.2. PROGRAM STRUCTURE AND TECHNICAL APPROACH

OP TEMPO is a 42-month program, composed of a 9-month Phase 0 involving T&E teams only, a 21-month Phase 1 involving both T&E and Performer teams, and a 12-month Phase 2 involving both T&E and Performer teams. A pulse check will occur mid-way through Phase 1 to assess performer progress based on summary characterizations of candidate bio-behavioral signatures identified to-date for Use Case 1. An evaluation will occur at the end of Phase 1, during which the performers’ predictive models will be characterized and assessed against program metrics. DARPA reserves the right to proceed with all, some, or none of the Phase 1 performers into Phase 2. Phase 2 will explore the generalizability of Performer signatures within the context of Use Case 2 and Use Case 3, concluding with an End-of-Program Evaluation.

Figure 1 provides an overview of the OP TEMPO T&E activities, Performer activities, and envisioned program outputs by Phase.

Figure 1: OP TEMPO Program Schedule and Phases

Note: The OP TEMPO program does not include distinct Technical Areas (TAs). All proposals must include plans to address all four research tasks and both phases of the program (See Section

1.4 for further details). All proposals must also include plans to address all program metrics, milestones, and deliverables (See Section 1.3 for further details).

Phase 0 (9 months) Not being solicited under this BAA The 9-month Phase 0 will focus on design and development of the customized training environment and scenarios for Use Case 1 (USMC FiST training; see https://www.29palms.marines.mil/training/magtftcsims/fst/ for more details) in order to enable performer research to start immediately after Phase 1 Kickoff. Pre-kickoff, the T&E team will complete the FiST training CTA, design and develop the customized FiST training scenarios, and build a framework capable of assessing team training performance by embedding objective performance metrics and time-synced meta-data within each scenario. The T&E team will also collect data from FiST training sessions using the evaluation framework to characterize novice, intermediate and expert performance for each of the FiST training scenarios.

During Phase 0, DARPA will also select Use Cases 2 and 3 based on lessons learned by the T&E team. These additional use cases will be presented at the Phase 1 Kickoff Meeting along with all results from the Use Case 1 pre-kickoff activities. For proposal development purposes, offerors can expect Use Cases 2 and 3 to involve 2-6 person teams completing computer/monitor station-based training scenarios involving collaborative subtasks and individual role subtasks, similar to the FiST training scenarios.

Phase 1 (21 months) Phase 1 T&E efforts will begin at the OP TEMPO Phase 1 Kickoff Meeting by providing the selected Performer teams with a detailed overview of all Phase 0 activities. This will include https://www.29palms.marines.mil/training/magtftcsims/fst/ deep dives into the CTA process and results, the design and development of the customized FiST simulation-based training environment and scenarios, the Use Case 1 established performance metrics and metadata, and initial results from the team performance evaluation framework. At the Phase 1 Kickoff Meeting, the T&E team will also provide performers with details regarding the planned Phase 1 data collection events and a detailed overview of planned methods for assessing performer results against the established program metrics at the Pulse Check and the End-of-Phase Evaluation (See Section 1.3 for further details).

Following the kickoff, Phase 1 T&E efforts will focus initially on finalizing the FiST training assessment framework by Month 4. T&E efforts for the remainder of Phase 1 will be focused primarily on supporting Performer data collection efforts and using the training assessment framework to generate ground truth data for each training session. Throughout each Phase, the T&E team will support DARPA in evaluating performer solutions, technical results, and progress toward achieving all program Metrics, and particularly at the Pulse Check and the End-of-Phase Evaluation. The T&E team will also conduct Independent Validation and Verification (IV&V) efforts by performing additional analyses on performer predictive models and collected datasets.

Additionally, during Phase 1, the T&E team will initiate CTAs for Use Case 2 and Use Case 3, and will begin designing and developing the customized training environments and scenarios for both Use Cases and the associated training assessment frameworks. T&E efforts to build out these additional Use Case scenarios will ensure timely scenario availability at the start of Phase 2.

Phase 1 Performer efforts will focus on the first three OP TEMPO research tasks within the context of Use Case 1 (USMC FiST training): 1) Identify candidate bio-behavioral signatures that correlate to DoD team performance; 2) Characterize signatures to assess their reliability, stability, and variability; 3) Develop predictive models of team performance to validate prospective bio-behavioral signatures.

All performer teams will be required to participate in formal data collection events at USMC FiST training installations in accordance with DoD human subjects research (HSR) policies (see Section 4.2.3 for additional details). DARPA estimates 3 data collection events will occur during Phase 1, providing performers with the ability to collect data from approximately 30+ team training sessions. For planning purposes related to data availability, scale, and complexity, performers should assume each training session is approximately 1 hour and will be comprised of multiple individual and team collaboration subtasks. During these events, performer teams will be required to collect data using the T&E-provided customized FiST training environment and scenarios with DARPA-provided active-duty military personnel as participants. All performer teams will be provided with the automatically generated objective individual- and team- performance metrics and time-synced metadata for each of the experimental scenarios run throughout these events, as well as ground truth data on training performance.

Proposers are not discouraged from conducting their own data collection events in other test beds in addition to the DARPA-provided Use Cases and training scenarios; however, proposals must include a clear rationale on applicability and relevance to the goals of OP TEMPO. All program metric evaluations will be conducted in the DARPA-provided Use Cases.

In the first research task (Identify), performers will identify candidate bio-behavioral signatures that correlate with team performance, using team performance ground truth data provided by the T&E team. It is anticipated that performers will search for multiple candidate bio-behavioral signatures such as neural activity (electrical and/or imaging), eye tracking, heart rate variability, breathing rate, and voice communications. Performers will need to develop approaches and leverage technologies that enable efficient, high-quality capture of the candidate signatures without disrupting the training. Proposals should address challenges associated with finding signatures having sufficient correlation to performance and working in noisy/high-background environments, and should provide mitigation strategies for addressing these challenges. Strong proposals will also consider approaches that make efficient use of sparse datasets to identify candidate signatures.

In the second research task (Characterize), performers will seek to understand the dynamics of the candidate bio-behavioral signatures. These approaches will likely require careful selection and use of denoising algorithms to enable real-world data capture and associated removal of motion and other background artifacts. Performers will need to characterize the signals with respect to stability, as well as inter- and intra-individual variability. Proposals should address challenges associated with 1) signature latency and persistence across dynamic tasks, 2) inter-and intra-individual variation, 3) balancing rigorous signature measurement approaches with ease-of-use, and 4) development of approaches capable of rapidly capturing signatures.

Proposals should also provide mitigation strategies for addressing these challenges. Strong proposals will include characterization methods and approaches capable of adapting to real-world data capture dynamics and contingencies.

In the third research task (Validate), performers will validate candidate bio-behavioral signatures in predictive models of team performance that use the signatures as inputs. Modeling approaches will likely involve combinatorial use of signatures to bolster predictive power, and performers will demonstrate which signature combinations are most effective. A validation dataset of ground truth data (captured during the Use Case data collection events) will be reserved by the T&E team to evaluate model performance at the end of the Phase. Proposals should address challenges associated with maximizing utility from sparse datasets and imbalanced ground truth datasets, fusion of diverse data types, maximizing parsimony of model types, and should provide mitigation strategies for addressing these challenges.

As shown in Figure 1, anticipated outputs from performer efforts in Phase 1 will include 1) Validated bio-behavioral signatures of team performance, 2) Database of raw signals for future analysis, 3) Protocols for the identification of bio-behavioral signatures, 4) Predictive models and algorithms to validate signatures, and 5) Draft plan to research the extensibility of and generalizability of performer’s approach (See Section 1.3 for further details).

Phase 2 (12 months) Phase 2 T&E efforts will begin at the OP TEMPO Phase 2 Kickoff Meeting by providing the performer team(s) with deep dives into the Use Case 2 and Use Case 3 CTA processes and results, the design and development of the customized Phase 2 simulation-based training environments and scenarios, the established performance metrics and metadata, and the performance assessment frameworks for generating ground truth data for Use Cases 2 and 3. At the Phase 2 Kickoff Meeting, the T&E team will also provide performers with details regarding each of the planned Phase 2 data collection events and a detailed overview of planned methods for assessing performer results against the established program metrics at the End-of-Program Evaluation (See Section 1.3 for further details).

Additional T&E efforts for the remainder of Phase 2 will be focused on finalizing frameworks capable of assessing team training performance, supporting performer data collection efforts, and conducting ground truth assessments of team training performance. The T&E team will also support DARPA by evaluating performer solutions, assessing technical results (e.g., generalizability of predictive models), and reviewing progress toward achieving all program Metrics throughout the Phase, and particularly at the End-of-Phase Evaluation for Use Cases 2 and 3.

Phase 2 Performer efforts will focus on the fourth research task (Explore Generalizability).

Performers will assess the predictive potential of their validated Phase 1 FiST signatures within the context of Use Case 2 and 3 teams. Similar to Phase 1, the team(s) will assess the predictive potential of models that use the bio-behavioral signatures as inputs.

All Phase 2 performer teams will be required to participate in formal data collection events at DoD training installations. During these events, performer teams will be required to collect data using the T&E-provided customized Use Case 2 and Use Case 3 training environments and scenarios with DARPA-provided active-duty military personnel as participants. All performer teams will be provided with the automatically generated objective individual- and team-performance metrics and time-synced metadata for each of the experimental scenarios run throughout these events, as well as ground truth data on training performance. Proposals should address challenges associated with signature generalizability to novel use cases, tasks, and types of teams, including potentially scaling to larger teams, and should provide mitigation strategies for addressing these challenges.

Proposals should include Phase 2 as a priced option.

As shown in Figure 1, anticipated outputs from performer efforts in Phase 2 will include 1) Generalizable bio-behavioral signatures of team performance in Use Case 2 and Use Case 3, 2) Databases of raw signals for future analysis, and 3) Updated predictive models & algorithms (See Section 1.3 for further details).

1.3. PROGRAM METRICS, MILESTONES AND DELIVERABLES

OP TEMPO will measure performer progress by assessing the quality of the models developed to use bio-behavioral signature input data to predict team performance. This includes measuring and reporting the accuracy and the F1 score (harmonic mean between recall and precision measures) of the classification models for subtask performance assessments and also the overall performance assessment for each training session. The T&E team will withhold a portion of the ground truth performance assessments obtained during the Use Case data collection events for end-of-phase evaluation activities and calculation of program metrics. Overall team training performance will be defined by T&E as novice, intermediate or expert performance. The Pulse Check in Phase 1 is a self-assessment activity conducted by each performer. Table 1 provides a summary of the OP TEMPO program metrics. Metric thresholds will be the same across all three use cases. Performers are expected to achieve these metrics for Use Case 1 by the conclusion of Phase 1, and are expected to achieve these metrics for Use Cases 2 and 3 by the conclusion of Phase 2.

Table 1. OP TEMPO Program Metrics Program Component Criteria Metric(s)

Model accuracy for overall team performance >0.8

Model F1 Score for overall team performance >0.85

Model accuracy for subtask team performance >0.8

Validate Signatures with Predictive Models

Model F1 Score for subtask team performance >0.85

Performers must propose specific deliverables (reports, data, demonstrations, etc.) that demonstrate progress towards, or completion of, program metrics for each Phase of the program.

The format and content of each deliverable will vary from task to task, but must be designed such that the Government can evaluate performer progress towards the metrics and end goals of the program. Refer to Section 1.4 and Section 6.3 for additional deliverables-based reporting requirements.

Progress toward the stated goals will be assessed throughout the program, with critical assessments and milestones occurring at 12 months (Phase 1 Pulse Check), 18 months (End-of- Phase Evaluation), and 32 months (End-of-Program Evaluation) after Phase 1 Kickoff. Table 2 provides the anticipated schedule for OP TEMPO milestones for Phase 1 and Phase 2. Locations for Meetings and Reviews are for planning purposes only and subject to change.

Table 2. OP TEMPO Program Milestones

Phase 1 Program Milestones Anticipated Date: Months After Contract Award (MAC)

Milestone 1: Phase 1 Kickoff Meeting (West Coast) Month 1 Milestone 2: Performers obtain local IRB approval for Phase 1 Human Subjects Research (HSR) Month 2

Milestone 3: Performers obtain HRPO approval for Phase 1 HSR Month 3

Milestone 4: Use Case 1 Data Collection Event 1 Month 4 Milestone 5: Technical Review Meeting (East Coast) Month 6 Milestone 6: Use Case 1 Data Collection Event 2 Month 9 Milestone 7: Pulse Check Technical Review (West Coast) Month 12 Milestone 8: Use Case 1 Data Collection Event 3 Month 15

Milestone 9: End-of-Phase Evaluation Month 18 Milestone 10: Performers obtain local IRB approval for Phase 2 HSR Month 20

Milestone 11: PI Meeting and Stakeholder Capability Demonstration (East Coast) Month 21

Phase 2 Program Milestones Anticipated Date: Months After Contract Award (MAC)

Milestone 12: Phase 2 Kickoff Meeting (West Coast) Month 22 Milestone 13: Performers obtain HRPO approval for Phase 2 HSR Month 22

Milestone 14: Use Case 2 Data Collection Event Month 24 Milestone 15: Technical Review Meeting (East Coast) Month 27 Milestone 16: Use Case 3 Data Collection Month 29 Milestone 17: Technical Review Meeting (West Coast) Month 32 Milestone 18: End-of-Program Evaluation and Stakeholder Capability Demonstration (East Coast) Month 33

Stakeholder Capability Demonstrations will be held at the end of each Phase in support of program technology transition efforts. These demonstrations will include summary results from recent data analyses and potentially opportunities for end users to interact with the OP TEMPO systems and will be separate from the data collection and assessment events associated with metrics evaluations.

The OP TEMPO program will develop four primary transitionable components: 1) protocols for the identification of bio-behavior signatures (performer provided); 2) databases of raw signals captured during team training (performer provided); 3) predictive models and algorithms for validating prospective bio-behavioral signatures (performer provided); and 4) training assessment frameworks for systematic capture of team performance results for each use case (T&E team provided).

It is envisioned that the customized training environments and scenarios developed by the T&E for use throughout the program may be transitionable to end users in the targeted DoD training communities for each Use Case; however, it is also possible that performer-developed tools may be appropriate for transition with those testbeds. The databases of raw signals captured by the performers during formal data collection events also have transition potential as future analysis by DoD T&E organizations may yield additional signatures with predictive value. Finally, the predictive models and algorithms themselves have potential to be transitioned to DoD training centers for further analysis and validation of newly discovered signatures.

Proposals should include detailed plans to support transition of these components, as well as any asserted data and intellectual property rights associated with delivered hardware, software, and data sets.

Successful transition of these components will be facilitated by early, regular engagement with critical stakeholders throughout program progress, including direct involvement by transition partners on the T&E team and via the End-of-Phase Stakeholder Capability Demonstrations.

Performers may be encouraged to pursue additional, transition-focused funding from DARPA via the Commercial Strategies Team (CST). See Section 6.5 for more details.

1.4. GENERAL REQUIREMENTS

Proposing Teams The OP TEMPO program does not include distinct Technical Areas (TAs). All proposals must include plans to address all four research tasks and both phases of the program. Furthermore, it is expected that proposals will involve diverse, multidisciplinary expertise to support the scientific aims relevant to each critical research task. Proposer teams must submit a single, integrated proposal led by a Principal Investigator (PI), under a single prime contractor that addresses all program phases.

DARPA will hold a Proposers’ Day (see Section 8.1) to facilitate the formation of proposer teams and enable sharing of information among interested proposers through the DARPA Opportunities Page. Attendance is voluntary and is not required to propose to solicitations associated with this program. DARPA will not provide reimbursement for any costs incurred to participate in Proposers’ Day.

Government Test and Evaluation (T&E) The Government is not soliciting T&E proposals under HR001123S0055. To avoid potential conflicts of interest, performers for HR001123S0055 will not be allowed to compete for the T&E contract. Throughout the program, the performers will work with a government-led T&E team.

This partnership will be facilitated by the Government. The T&E team will consist of subject matter experts from the Government, Federally Funded Research and Development Centers (FFRDCs), and/or other relevant domains capable of meeting the desired T&E goals of the program as established by DARPA. The T&E team will objectively evaluate the performance of each performer’s capabilities and/or components against OP TEMPO program requirements and specifications after pulse checks, end-of-phase evaluations, stakeholder demonstrations, and other milestones.

Data Management and Sharing All raw data, metadata, and informatics analyses, and tools specific to each experiment must be curated and made available. All data (raw data, highly-detailed metadata, and key analysis files) from data collection events will be uploaded to an appropriate server and be made available to DARPA, T&E team and Government Stakeholders. Software design and analyses must be systematically documented with coding tools (e.g., Jupyter Notebook, R Notebook, Spark) for evaluation and reproducibility.

Ethical, Legal, and Societal Implication (ELSI) DARPA assures that all of its sponsored research activities related to neurotechnology and human subjects will be guided by the ethical principles. Research and Development in human performance optimization, warfighter readiness, human machine teaming, and brain-inspired artificial intelligence all have ethical, legal, and social implications. Proposals are required to integrate ethical considerations and implications of their research including: research integrity, diversity of the research team, and societal impacts. This could include consideration of issues such as: transparency, inclusivity, social responsibility, public perceptions, impartiality, reliability, security and privacy. Related issues are openness and fairness of societal access to new technologies, post-trial access, responsible development of safe, secure, and robust operations in expected or foreseeable social and/or engineering contexts. Investigators are encouraged to consider the ethical implications of both intended and possible unintended consequences of new technologies (i.e., countermetrics) and how best to mitigate risk to society.

It is strongly recommended that the research team includes appropriate expertise from the social, behavioral, ethics and educational sciences to address these issues.

Deliverables All products – material and otherwise – to be provided to the Government as outcomes from conducted research should be defined in the proposal. Performers need to allot time and budget to fulfill obligations for travel to review meetings and the preparation and presentation of status reports.

Monthly Technical Status Reports: Each month during program execution, performers are required to present to DARPA a status report briefing, in the form of a Microsoft (MS) PowerPointTM file, of all research activities, technical progress, cost/schedule issues and performance, metrics satisfied, risk mitigations, and intended actions for the next reporting period.

Monthly Financial Status Reports: Each month during program execution, performers are required to provide financial status updates. The prime Performer shall include information for itself and all subawardees/subcontractors. These reports shall be in the form of an editable MS ExcelTM file and shall provide financial data including, but not limited to:

Program spend plan by Phase and task Incurred program expenditures to date by Phase and task Invoiced program expenditures to date by Phase and task

Quarterly Status Reports: Each quarter during program execution, performers are required to submit a written report (MS WordTM file) summarizing the technical progress and financial status of the effort, including cost/schedule issues and performance, metrics satisfied, risk mitigations.

Technical Review Meetings and Reports: Leadership from each performer team (with additional key personnel at the discretion of the PI) will be required to present research progress in person or virtually at program review meetings. The purpose of these reviews is to ensure adequate engagement with the DARPA team to discuss details that might otherwise fall outside the scope of a routine technical brief; progress towards milestones and scientific goals; and any ongoing technical or programmatic challenges that must be overcome to achieve the overarching program goals. Performers must submit to DARPA their briefing materials (MS PowerPoint).

Phase I Final Report: At the end of Phase I, performer teams must provide a summary report on all research activities and outcomes during the Phase. The report should also list publications, research presentations, and patent applications that result from the research pursued, and any additional deliverables specified in the award document.

Final Program Report: Performer teams must provide a final report summarizing all research activities and outcomes during the program; publications, research presentations, patent applications that result from the research pursued; and any additional deliverables specified in the award document.

2. Award Information

2.1. GENERAL AWARD INFORMATION

Multiple awards are possible. The amount of resources made available under this BAA will depend on the quality of the proposals received and the availability of funds.

The Government reserves the right to select for negotiation all, some, one, or none of the proposals received in response to this solicitation and to make awards without discussions with proposers. The Government also reserves the right to conduct discussions if it is later determined to be necessary. If warranted, portions of resulting awards may be segregated into pre-priced options. Additionally, DARPA reserves the right to accept proposals in their entirety or to select only portions of proposals for award. In the event that DARPA desires to award only portions of a proposal, negotiations may be opened with that proposer. The Government reserves the right to fund proposals in phases with options for continued work, as applicable.

The Government reserves the right to request any additional, necessary documentation once it makes the award instrument determination. Such additional information may include but is not limited to Representations and Certifications (see Section VI.B.2., “Representations and Certifications”). The Government reserves the right to remove proposers from award consideration should the parties fail to reach agreement on award terms, conditions, and/or cost/price within a reasonable time, and the proposer fails to timely provide requested additional information. Proposals identified for negotiation may result in a procurement contract, cooperative agreement, or other transaction, depending upon the nature of the work proposed, the required degree of interaction between parties, whether or not the research is classified as Fundamental Research, and other factors.

Proposers looking for innovative, commercial-like contractual arrangements are encouraged to consider requesting Other Transactions. To understand the flexibility and options associated with Other Transactions, consult http://www.darpa.mil/work-with-us/contract-management#OtherTransactions.

In accordance with 10 U.S.C. § 4022(f), the Government may award a follow-on production contract or Other Transaction (OT) for any OT awarded under this solicitation if: (1) that participant in the OT, or a recognized successor in interest to the OT, successfully completed the entire prototype project provided for in the OT, as modified; and (2) the OT provides for the award of a follow-on production contract or OT to the participant, or a recognized successor in interest to the OT.

http://www.darpa.mil/work-with-us/contract-management#OtherTransactions http://www.darpa.mil/work-with-us/contract-management#OtherTransactions

In all cases, the Government contracting officer shall have sole discretion to select award instrument type, regardless of instrument type proposed, and to negotiate all instrument terms and conditions with selectees. DARPA will apply publication or other restrictions, as necessary, if it determines that the research resulting from the proposed effort will present a high likelihood of disclosing performance characteristics of military systems or manufacturing technologies that are unique and critical to defense. Any award resulting from such a determination will include a requirement for DARPA permission before publishing any information or results on the program. For more information on publication restrictions, see the section below on Fundamental Research

2.2. FUNDAMENTAL RESEARCH

It is DoD policy that the publication of products of fundamental research will remain unrestricted to the maximum extent possible. National Security Decision Directive (NSDD) 189 defines fundamental research as follows:

‘Fundamental research’ means basic and applied research in science and engineering, the results of which ordinarily are published and shared broadly within the scientific community, as distinguished from proprietary research and from industrial development, design, production, and product utilization, the results of which ordinarily are restricted for proprietary or national security reasons.

As of the date of publication of this solicitation, the Government expects that program goals as described herein may be met by proposers intending to perform fundamental research and does not anticipate applying publication restrictions of any kind to individual awards for fundamental research that may result from this solicitation. Notwithstanding this statement of expectation, the Government is not prohibited from considering and selecting research proposals that, while perhaps not qualifying as fundamental research under the foregoing definition, still meet the solicitation criteria for submissions. If proposals are selected for award that offer other than a fundamental research solution, the Government will either work with the proposer to modify the proposed statement of work to bring the research back into line with fundamental research or else the proposer will agree to restrictions in order to receive an award.

University or non-profit research institution performance under this solicitation will include effort categorized as fundamental research. In addition to Government support for free and open scientific exchanges and dissemination of research results in a broad and unrestricted manner, the academic or non-profit research performer or recipient, regardless of tier, acknowledges that such research may have implications that are important to U.S. national interests and must be protected against foreign influence and exploitation. As such, the academic or non-profit research performer or recipient agrees to comply with the following requirements:

(a) The University or non-profit research institution performer or recipient must establish and maintain an internal process or procedure to address foreign talent programs, conflicts of commitment, conflicts of interest, and research integrity. The academic or non-profit research performer or recipient must also utilize due diligence to identify Foreign Components or participation by Senior/Key Personnel in Foreign Government

Talent Recruitment Programs and agree to share such information with the Government upon request.

i. The above described information will be provided to the Government as part of the proposal response to the solicitation and will be reviewed and assessed prior to award. Generally, this information will be included in the Research and Related Senior/Key Personnel Profile (Expanded) form (SF-424) required as part the proposer’s submission through Grants.gov.

1. Instructions regarding how to fill out the SF-424 and its biographical sketch can be found through Grants.gov.

ii. In accordance with USD(R&E) direction to mitigate undue foreign influence in DoD-funded science and technology, DARPA will assess all Senior/Key Personnel proposed to support DARPA grants and cooperative agreements for potential undue foreign influence risk factors relating to professional and financial activities. This will be done by evaluating information provided via the SF-424, and any accompanying or referenced documents, in order to identify and assess any associations or affiliations the Senior/Key Personnel may have with foreign strategic competitors or countries that have a history of intellectual property theft, research misconduct, or history of targeting U.S. technology for unauthorized transfer. DARPA’s evaluation takes into consideration the entirety of the Senior/Key Personnel’s SF-424, current and pending support, and biographical sketch, placing the most weight on the Senior/Key Person’s professional and financial activities over the last 4 years. The majority of foreign entities lists used to make these determinations are publicly available. The DARPA Countering Foreign Influence Program (CFIP) “Senior/Key Personnel Foreign Influence Risk Rubric” details the various risk ratings and factors. The rubric can be seen at the following link:

https://www.darpa.mil/attachments/092021DARPACFIPRubric.pdf

iii. Examples of lists that DARPA leverages to assess potential undue foreign influence factors include, but are not limited to:

1. Executive Order 13959 “Addressing the Threat From Securities Investments That Finance Communist Chinese Military Companies”:

https://www.govinfo.gov/content/pkg/FR-2020-11-17/pdf/2020-25459.pdf

2. The U.S. Department of Education’s College Foreign Gift and Contract Report: College Foreign Gift Reporting (ed.gov)

3. The U.S. Department of Commerce, Bureau of Industry and Security, List of Parties of Concern: https://www.bis.doc.gov/index.php/policy-guidance/lists-of-parties-of-concern

4. Georgetown University’s Center for Security and Emerging Technology (CSET) Chinese Talent Program Tracker:

https://chinatalenttracker.cset.tech

5. Director of National Intelligence (DNI) “World Wide Threat Assessment of the US Intelligence Community”: 2021 Annual Threat Assessment of the U.S. Intelligence Community (dni.gov) https://www.darpa.mil/attachments/092021DARPACFIPRubric.pdf https://www.govinfo.gov/content/pkg/FR-2020-11-17/pdf/2020-25459.pdf https://sites.ed.gov/foreigngifts/ https://www.bis.doc.gov/index.php/policy-guidance/lists-of-parties-of-concern https://www.bis.doc.gov/index.php/policy-guidance/lists-of-parties-of-concern https://chinatalenttracker.cset.tech/ https://www.dni.gov/index.php/newsroom/reports-publications/reports-publications-2021/item/2204-2021-annual-threat-assessment-of-the-u-s-intelligence-community https://www.dni.gov/index.php/newsroom/reports-publications/reports-publications-2021/item/2204-2021-annual-threat-assessment-of-the-u-s-intelligence-community

6. Various Defense Counterintelligence and Security Agency (DCSA) products regarding targeting of US technologies, adversary targeting of academia, and the exploitation of academic experts: https://www.dcsa.mil/

(b) DARPA’s analysis and assessment of affiliations and associations of Senior/Key Personnel is compliant with Title VI of the Civil Rights Act of 1964. Information regarding race, color, or national origin is not collected and does not have bearing in DARPA’s assessment.

(c) University or non-profit research institutions with proposals selected for negotiation that have been assessed as having high or very high undue foreign influence risk, will be given an opportunity during the negotiation process to mitigate the risk. DARPA reserves the right to request any follow-up information needed to assess risk or mitigation strategies.

i. Upon conclusion of the negotiations, if DARPA determines, despite any proposed mitigation terms (e.g. mitigation plan, alternative research personnel), the participation of any Senior/Key Research Personnel still represents high risk to the program, or proposed mitigation affects the Government’s confidence in proposer’s capability to successfully complete the research (e.g., less qualified Senior/Key Research Personnel) the Government may determine not to award the proposed effort. Any decision not to award will be predicated upon reasonable disclosure of the pertinent facts and reasonable discussion of any possible alternatives while balancing program award timeline requirements.

(d) Failure of the academic or non-profit research performer or recipient to reasonably exercise due diligence to discover or ensure that neither it nor any of its Senior/Key Research Personnel involved in the subject award are participating in a Foreign Government Talent Program or have a Foreign Component with an a strategic competitor or country with a history of targeting U.S. technology for unauthorized transfer may result in the Government exercising remedies in accordance with federal law and regulation.

i. If, at any time, during performance of this research award, the academic or non-profit research performer or recipient should learn that it, its Senior/Key Research Personnel, or applicable team members or subtier performers on this award are or are believed to be participants in a Foreign Government Talent Program or have Foreign Components with a strategic competitor or country with a history of targeting U.S. technology for unauthorized transfer , the performer or recipient will notify the Government Contracting Officer or Agreements Officer within 5 business days.

1. This disclosure must include specific information as to the personnel involved and the nature of the situation and relationship. The Government will have 30 business days to review this information and conduct any necessary fact-finding or discussion with the performer or recipient.

2. The Government’s timely determination and response to this disclosure may range anywhere from acceptance, to mitigation, to termination of this award at the Government’s discretion.

https://www.dcsa.mil/

3. If the University receives no response from the Government to its disclosure within 30 business days, it may presume that the Government has determined the disclosure does not represent a threat.

ii. The performer or recipient must flow down this provision to any subtier contracts or agreements involving direct participation in the performance of the research.

(e) Definitions

i. Senior/Key Research Personnel

1. This definition would include the Principal Investigator or Program/Project Director and other individuals who contribute to the scientific development or execution of a project in a…