HR001122S0032.pdf

HR001122S0032 NEAT 1

Broad Agency Announcement Neural Evidence Aggregation Tool (NEAT)

Defense Sciences Office

HR001122S0032

March 11, 2022

HR001122S0032 NEAT 2

Table of Contents I. Funding Opportunity Description

A. Introduction B. Background C. Program Description/Scope D. Program Structure E. Technical Area Descriptions F. Schedule/Milestones G. TA-specific Deliverables H. Government-furnished Property/Equipment/Information I. Other Program Objectives and Considerations

II. Award Information A. General Award Information B. Fundamental Research

III. Eligibility Information A. Eligible Applicants B. Organizational Conflicts of Interest C. Cost Sharing/Matching D. Ability to Receive Awards in Multiple Technical Areas - Conflicts of Interest

IV. Application and Submission Information A. Address to Request Application Package B. Content and Form of Application Submission C. Submission Dates and Times D. Funding Restrictions E. Other Submission Requirements

V. Application Review Information A. Evaluation Criteria B. Review and Selection Process C. Countering Foreign Influence Program (CFIP) D. Federal Awardee Performance and Integrity Information (FAPIIS)

VI. Award Administration Information A. Selection Notices B. Administrative and National Policy Requirements C. Reporting

VII. Agency Contacts VIII. Other Information

A. Proposers Day B. Frequently Asked Questions (FAQs) C. Collaborative Efforts/Teaming

BAA Attachments:

Attachment A: ABSTRACT SUMMARY SLIDE TEMPLATE Attachment B: ABSTRACT TEMPLATE Attachment C: PROPOSAL SUMMARY SLIDE TEMPLATE Attachment D: PROPOSAL TEMPLATE VOLUME 1: TECHNICAL & MANAGEMENT Attachment E: PROPOSAL TEMPLATE VOLUME 2: COST

HR001122S0032 NEAT 3

Attachment F: MS ExcelTM DARPA COST PROPOSAL SPREADSHEET Attachment G: PROPOSAL TEMPLATE VOLUME 3: ADMINISTRATIVE & NATIONAL POLICY

REQUIREMENTS

HR001122S0032 NEAT 4

PART I: OVERVIEW INFORMATION

Federal Agency Name: Defense Advanced Research Projects Agency (DARPA), Defense Sciences Office (DSO)

Funding Opportunity Title: Neural Aggregation Evidence Tool (NEAT)

Announcement Type: Initial Announcement

Funding Opportunity Number: HR001122S0032

Catalog of Federal Domestic Assistance (CFDA) Number(s): 12.910 Research and Technology Development

Dates (All times listed herein are Eastern Time.)

o Posting Date: March 11, 2022 o Proposers Day: March 15, 2022. See Section VIII.A.

o Abstract Due Date: March 29, 2022, 4:00 p.m.

o FAQ Submission Deadline: April 9, 2022, 4:00 p.m. See Section VIII.B.

o Full Proposal Due Date: May 23, 2022, 4:00 p.m.

Anticipated Individual Awards: DARPA anticipates multiple awards for Technical Area 1 and a single award for Technical Area 2.

Types of Instruments that May be Awarded: Procurement contracts, cooperative agreements, or Other Transactions. Award instruments will be limited to procurement contracts and Other Transactions for Proposers whose proposed solution includes Controlled Unclassified Information (CUI).

Agency contacts o Technical POC: Gregory Witkop, M.D., Program Manager, DARPA/DSO o BAA Email: NEAT@darpa.mil o BAA Mailing Address:

DARPA/DSO

ATTN: HR001122S0032

675 North Randolph Street Arlington, VA 22203-2114 o DARPA/DSO Opportunities Website: http://www.darpa.mil/work-with-us/opportunities

Teaming Information: See Section VIII.C for information on teaming opportunities.

Frequently Asked Questions (FAQ): FAQs for this solicitation may be viewed on the DARPA/DSO Opportunities Website. See Section VIII.B for further information.

Security: NEAT is an UNCLASSIFIED program. If proposers would like to work with Controlled Unclassified Information (CUI) please specify so in the abstract and proposal and refer to section IV.B.4.

mailto:NEAT@darpa.mil https://www.darpa.mil/work-with-us/opportunities?oFilter=DSO https://www.darpa.mil/work-with-us/opportunities?oFilter=DSO

HR001122S0032 NEAT 5

PART II: FULL TEXT OF ANNOUNCEMENT

I. Funding Opportunity Description This Broad Agency Announcement (BAA) constitutes a public notice of a competitive funding opportunity as described in Federal Acquisition Regulation (FAR) 6.102(d)(2) and 35.016 as well as 2 C.F.R. § 200.203. Any resultant negotiations and/or awards will follow all laws and regulations applicable to the specific award instrument(s) available under this BAA, e.g., FAR

15.4 for procurement contracts.

A. Introduction

The Defense Sciences Office (DSO) at the Defense Advanced Research Projects Agency (DARPA) is soliciting innovative research proposals in the area of novel assessment tools to elicit and aggregate preconscious signals for determining what people believe to be true.

Proposed research should investigate innovative approaches that enable revolutionary advances in science, devices, or systems. Specifically excluded is research that primarily results in evolutionary or incremental improvements to the existing state of practice.

B. Background

Trends in mental health and mental fitness were alarming before the COVID-19 pandemic.

These trends have worsened during the pandemic, with rates of depression and anxiety rising precipitously and exacting a significant toll on national health and productivity.i These findings affect all Americans but have been particularly harmful for DoD personnel who face the additional strains of combat, long deployments, and over two decades of war. These trends have resulted in veterans between the ages of 18-34 being almost three times more likely to commit suicide than their non-veteran peers.ii Current methods to detect early signs of behavioral health risk factors (such as anxiety, depression, or substance abuse) leading to suicide rely on self-reporting and screening questionnaires. Unfortunately, a recent metanalysis of longitudinal cohort suicide risk assessments concluded there are no reliable means to predict suicidality.iii

Moreover, the combination of warfighter’s commitment to “stay in the fight” and the persistent stigma associated with seeking behavioral health assistance make current screening methods particularly challenging to use in military personnel. In order to save lives through early detection, the goal of NEAT is to use preconscious signals to identify what someone believes to be true about their own behavioral health risk factors – especially when what they believe to be true can be difficult to outwardly acknowledge, as would be required for current screening assessment. The use of preconscious signals will eliminate the possibility of rationalization or minimization because the signals will be collected before someone has the ability to consciously formulate their responses. NEAT will revolutionize behavioral health screening to assist clinicians, minimize long term vulnerabilities, and maximize warfighter readiness.

Of note, the purpose of NEAT is to help people with issues that can be difficult to discuss, and the method relies on using preconscious signals obtained before they have time to consciously formulate responses. Therefore, any proposal involving credibility assessment or detection of deception techniques reflects a proposer’s fundamental misunderstanding of the program and will be deemed out of scope.

HR001122S0032 NEAT 6

C. Program Description/Scope

NEAT aims to develop a novel cognitive science tool that could be used to augment behavioral health screenings by accurately detecting what someone believes to be true. By bringing together recent advances in cognitive science, neuroscience, physiological sensors, data science, and machine learning, the NEAT program will develop processes that can measure what a person believes to be true by (1) presenting carefully crafted stimuli that are designed to evoke specific preconscious mental processes, (2) detecting the resulting preconscious processes using current physiological sensors combined with state-of-the-art signal processing and neural analytics, and

(3) using advances in machine learning and data science to aggregate the preconscious responses collected across a set of stimuli into a final measurement that quantifies what a person believes to be true for a specific topic. While the NEAT program will include advancing the state of the art in these areas as necessary, it will primarily focus on the multidisciplinary integration of state-of-the-art capabilities and/or approaches to achieve its goals. For example, recent work in psycholinguistics and decision theory has shown stimuli can elicit neural responses that offer insights into an individual’s moral conclusions or familiarity with specific topics. iv,v,vi Similarly, NEAT efforts could build on work pertaining to the effective detection of neurophysiological responses despite low signal to noise without relying on averaging across repeated stimuli and using commercially available sensors. NEAT efforts will adapt advances in machine learning, data science, and AI algorithms to develop models that can aggregate the collected preconscious responses into quantified measures of knowledge. By combining advances such as these, NEAT efforts will develop processes that will be able to determine with both high sensitivity and specificity what a person believes to be true for a specific set of well-defined topics.

Determining what someone believes to be true about clinically relevant categories in behavioral health populations creates responsibilities for care and presents challenges establishing requisite ground truth to evaluate the sensitivity and specificity of novel preconscious knowledge detection tools. Therefore, the NEAT efforts should focus on testable knowledge-detection scenarios that can be well-bounded in order to develop the necessary cognitive insights and analytic approaches. Similarly, proposals to NEAT can seek to develop the necessary tools and analytics for knowledge assessments in populations and domains that are outside of behavioral health, particularly in the early phases of the NEAT program. Proposals should clearly describe how the findings from the proposed domains and study populations provide proof of concept demonstrations that could be applied to behavioral health and other scenarios in the future.

Proposals should also describe how the complexity of the proposed study domains are advanced between the initial and final phases of the NEAT program and potential commercial transition opportunities.

DARPA will leverage an independent Ethical, Legal, and Societal Issues (ELSI) group to advise program leadership and performers on ELSI concerns, see Section E.3.

NEAT Key Words and Definitions The following clarifies key terminology as it is used for the purposes of this solicitation and the NEAT program:

Topic of Interest (TOI): four types of information of interest to the NEAT program:

HR001122S0032 NEAT 7

o Biographic information. Specific information related directly to knowledge of one’s own person and/or relationships to other individuals o Past/Present actions. Specific, well-defined information regarding an action that a person carried out in the past or is currently carrying out.

o Past/Present intentions. Specific, well-defined information regarding actions that one either intended or intends to do but which have not yet occurred o Declarative statements. Specific, well-defined statements that reflect a person’s knowledge or viewpoint regarding a specific topic.

Knowledge: what a person believes to be true about a specific TOI at a given moment in time.

Preconscious: Uncontrolled neural and/or physiological response elicited by stimuli that precedes consciously controlled processing. It can be difficult to precisely define neural processes related to areas of knowledge and memory and the specific boundaries between cognitive hierarchies and functions related to preconscious and consciously-controlled processing. For the purposes of the NEAT program, preconscious activity is assumed to typically occur within 500-750 milliseconds (or less) of the eliciting stimulus event.

While such responses are not restricted to purely autonomic reflexes, for the purposes of NEAT, they must occur in such a way as to precede the activation of higher-order cognitive processes to avoid any direct control.

Stimulus: Event that triggers a specific cascade of neural and/or physiological processes.

Examples include (but are not restricted to) external sensory stimuli such as auditory or visual inputs or the initiation of a motor output, with the key feature being the ability of defining the uncontrolled component of the chain of neural processes that result from the event.

Event-Related Neural Responses (ERNRs): Either central or peripheral responses to stimuli that reflect a specific neural process that may be proposed to achieve the goals of the NEAT program.

NEAT Process: Refers to the overall NEAT system, i.e., the combination of the sensor hardware, the paradigm of stimuli that are utilized, the resulting ERNRs, and the analytic pipeline that produces the final output regarding what a person believes to be true.

The goal of the NEAT program is to develop a new tool for quantitatively measuring a person’s knowledge. This NEAT Process will measure and aggregate an individual’s preconscious neural and/or physiological responses into actionable evidence to provide an interviewer with information about what an interviewee believes to be true, false, or indeterminate for specific, well-defined TOIs. To accomplish this, NEAT research efforts will:

Convert TOI into stimuli that clearly evoke different types of neural and/or physiological responses that precede consciously controlled processing, Determine which preconscious signals significantly relate to a person’s knowledge of a

TOI,

Create models for aggregating the significant preconscious signals into a composite metric of a person’s knowledge regarding a specific TOI, HR001122S0032 NEAT 8

Optimize techniques for the rapid and accurate detection and signal processing of neural processes evoked by TOI-related stimuli, and

Develop a tool for preconscious knowledge detection that can be individualized.

NEAT approaches must use neural and/or physiological measurements that reflect preconscious processing and thus avoid the confounds associated with evaluating verbal communication. The specific boundaries between preconscious and conscious processing are difficult to define.

However, for the purposes of the NEAT program, preconscious neural processes should typically occur within 500 to 750 milliseconds of an evoking stimuli (or less). Approaches that propose to use processes that occur with longer lags may be in scope if a clear justification is provided regarding how they reflect preconscious processing.

The NEAT Process must focus on the quantitative detection of knowledge, categorized into what someone believes to be true, false or indeterminant. As such, approaches specifically out of scope include:

Approaches solely aimed at providing estimates of general cognitive states or cognitive processes such as fatigue, attention, cognitive load, or the presence or absence of deception;

Evaluations based upon the content of a person’s conscious responses to specific stimuli or queries, such as conventional tests for knowledge using written or verbal exams; and

Evaluations that are aimed at credibility assessments or the detection of deception, whether verbal, written, or any other means.

Approaches that leverage consciously controlled responses to TOI stimuli, e.g., selecting options from a forced choice task, may be in scope so long as the proposed NEAT Process is aimed at utilizing only the preconscious aspect(s) of the response. For example, this could include detection of an error neural process that occurs within a few hundred milliseconds of the reaction to forced choice stimuli.

NEAT models for aggregating and using responses to stimuli should rely on neural processes that have clear theoretical validity (e.g., error, recognition, incongruence) that could be used to support construct validity of the final knowledge score. However, NEAT approaches do not need to only use ERNRs that involve direct measurements of brain activity (e.g., electroencephalogram (EEG) measurements); approaches that take advantage of peripheral physiological processes (e.g., measurement of pupillary responses to stimuli) to augment the use of neural signals (e.g. augment the sensitivity/specificity with which knowledge can be detected) are potentially within scope as long neural sources of information are not completely neglected.

Proposals must include scientific justification for all ERNRs planned to be collected and aggregated.

The use of machine learning, AI, and similar statistical learning approaches for analyzing and utilizing neural responses are in scope. However, models that overly rely on ‘black box’ analytic approaches that are not accompanied by a strategy for how to demonstrate a justification of the final knowledge score are of less interest. Proposed approaches using machine learning should

HR001122S0032 NEAT 9

clearly identify existing data for training the system or articulate a plan to collect the data necessary for training their systems.

NEAT proposers should leverage existing, commercial-off-the-shelf (COTS) sensor technologies (e.g., EEG, Functional Near-Infrared Spectroscopy ((fNIRS)), pupillometry) to support development of NEAT Processes. Proposals should clearly justify any research efforts that are devoted to developing new neural monitoring hardware or novel sensor technologies. Similarly, while the NEAT program does not require that sensor technologies be used in operational field environments during the program, efforts must support demonstrating NEAT technologies in standard office environments as part of the Phase 2 demonstrations. Phase 1 proposals should provide evaluations that assess the feasibility of the utilized hardware to be eventually transportable to the Phase 2 demonstration sites and provide information regarding the feasibility of eventually adapting the proposed sensor technologies to future possible operational field or clinical environments following the NEAT program.

D. Program Structure

NEAT is a 42-month, two phase effort divided into two Technical Areas (TAs) that run through both phases:

TA1 – Research and Development (R&D);

TA2 – Independent Validation and Verification (IV&V).

Phase 1 (Demonstrate Efficacy) will be 24 months, and Phase 2 (Develop System) will be 18 months. Proposers should address both phases and provide details for the Phase 1 (Base) and Phase 2 (Option) technical efforts as outlined in Section I.E. Technical Area Descriptions.

Phase 1 (Demonstrate Efficacy) will demonstrate essential proof of principle and show basic feasibility of the NEAT Process for detecting knowledge about NEAT TOIs. Phase 2 (Develop System) will build on work in Phase 1 by refining models and stimuli, improving overall performance, assessing the possible sensitivity of the NEAT Process to confounding variables, and testing the NEAT Process outside of laboratory settings. Additional details regarding the technical objectives of each phase are included in Section I.E. Technical Area Descriptions.

Proposals should address both phases and provide full details for Phase 1 (Base) and Phase 2 (Option). Phase 2 selection decisions are at the sole discretion of the Government and will be based on performance against the Phase 1 goals and metrics of each performer’s individual programmatic objectives (as well as the common dataset provided by TA2), overall progress towards the NEAT program objectives, and availability of funds. The Government retains the right to award all, some, one, none, or portions of the proposed Phase 2 options to support promising further technology developments. Participation in any given phase does not guarantee funding in a subsequent phase.

To evaluate progress, TA1 performers will demonstrate their NEAT Process at Month 20 in Phase 1 and Month 42 in Phase 2 (see Table 2 - NEAT TA1 Milestones). These demonstrations will showcase the efficacy of the NEAT Process for all NEAT TOIs. The Phase 1 demonstrations will take place over several days at sites chosen by each TA1 performer. TA1 proposals should describe how Phase 1 and Phase 2 demonstrations will best showcase their approach for

HR001122S0032 NEAT 10

conducting NEAT evaluations. Prior to the Phase 1 and Phase 2 demonstrations, the TA2 IV&V team will work with the TA1 performers to identify a set of stimuli and/or TOI assessments that are consistent with each TA1 approach that the TA1 performers’ must incorporate into their evaluations to create a common subset of assessments across all of the demonstrations. The Phase 2 demonstrations will take place at a DARPA-chosen, unclassified location similar to a typical office environment. The IV&V team will work with the TA1 performers to organize the Phase 2 demonstrations. For planning purposes, proposer(s) may assume the Phase 2 demonstration will take place in Washington, D.C. over approximately four days.

DARPA is committed to reproducibility of studies and methods developed under its programs. In support of this ideal, TA1 teams will be required to pre-register their studies, methods, and hypotheses1 and should clearly delineate within the proposal which proposed studies and methods will be exploratory and which will be confirmatory.

E. Technical Area Descriptions

The NEAT program is soliciting proposals for TA1 and TA2, outlined below. Each proposal should only address a single TA. Please note that to avoid conflicts of interest among TAs, no person or organization may be a performer on more than one TA, either as a prime or subcontractor. A single person or organization may be included in multiple proposals and those proposals can be submitted to different TAs, however, a single person or organization can be part of multiple awarded proposals only if those awards are all in the same TA. The program will also include a Government Ethical, Legal, and Societal Implications (ELSI) assessment effort.

1. TA1 – Research and Development

Proposed TA1 efforts should comprise a tightly-integrated, multidisciplinary team that can address all of the key R&D challenges. For example, proposers may consider collaboration with psycholinguistics expertise partners if they intend to leverage language-based ERNR approaches.

The breadth and depth of relevant expertise in the technical team will be considered in the evaluation of proposals. These teams should be well positioned to address all three of the fundamental TA1 goals as stated below:

Goal 1: Develop methods for converting TOI into stimuli that are designed to clearly evoke preconscious neural processing that can be clearly related to a person’s knowledge of a TOI, such as incongruence, recognition, or error detection.

Goal 2: Optimize techniques for the rapid and accurate detection and signal processing of neural processes evoked by TOI-related stimuli.

Goal 3: Create models that produce a metric of a person’s TOI knowledge by aggregating the information obtained from the various ERNRs. The models should be interpretable to explain how the NEAT Process output was generated.

1 See pre-registration sites for instructions on how to pre-register a study. For example: https://help.osf.io/hc/en-us/articles/360019738834-Create-a-Preregistration. For more information about the purpose of pre-registration see https://www.sciencemag.org/news/2018/09/more-and-more-scientists-are-preregistering-their-studies-shoul19d-you https://help.osf.io/hc/en-us/articles/360019738834-Create-a-Preregistration https://help.osf.io/hc/en-us/articles/360019738834-Create-a-Preregistration https://www.sciencemag.org/news/2018/09/more-and-more-scientists-are-preregistering-their-studies-shoul19d-you

HR001122S0032 NEAT 11

Goal 1: Develop methods for converting TOI into stimuli that are designed to clearly evoke preconscious neural processing that can be clearly related to a person’s knowledge of a TOI, such as incongruence, recognition, or error detection.

ERNRs indicate the body’s reaction to specific types of stimuli. Often, these responses are detected using EEG recordings to illustrate particular Event-Related Potentials (ERPs) that can be detected in response to certain classes of stimuli. For example, such ERPs include the N400 response that occurs when the brain is presented with an incongruent statement, e.g., “I will take my coffee with cream and socks,” or the P3b response that can be detected following recognition of a task relevant and/or meaningful stimuli, e.g., a response following one word that has meaning to a person among a series of more generic words. Proposed approaches can include both direct detection of neural responses to stimuli (such as ERPs) or the detection of peripheral signal responses to stimuli (e.g., leveraging pupil responses). For the case of peripheral responses, they should be clearly shown to reflect well-defined neural processes relevant to NEAT goals.

ERNRs (especially EEG ERPs) are typically characterized by a low signal-to-noise ratio (SNR) and are usually most detectable under carefully constructed stimulus conditions. Such conditions typically include the timing and duration of the stimulus/stimuli, the stimulus modality (e.g., text words, pictures, auditory), prior contextual briefing, etc. Nonetheless, ERNRs that reflect a wide range of neural processes are well established. For example, in addition to N400 or P3b responses, other ERNRs are detectable that reflect error processing when making decisions, selecting options in human computer interfaces, or observing others carrying out tasks.

This first goal of TA1 is to develop a process for converting NEAT TOIs into sets of stimuli that are optimized for evoking detectable ERNRs with high SNR. This process should be generalizable such that a given set of stimuli can be rapidly produced for a specific knowledge test within one or more TOI categories. The set of stimuli that are produced should be sufficiently well structured such that the resulting ERNRs can be detected using optimized methods and provide a sufficient library of useful ERNRs stimuli that can support the necessary battery of data needed by the aggregation models.

Please note that example uses of specific ERNRs/ERPs (such as the N400, error, and P3b) in this document are for illustrative purposes only, and proposers are encouraged to propose a variety of ERNR approaches in order to evoke the detectable neural processes they propose could form the basis of an effective quantitative measure of TOI knowledge. Similarly, the exact form of the stimuli modality that is proposed (e.g., visual text, auditory cues, imagery) are not predefined.

Successful proposals to TA1 must include the following with regard to Goal 1:

A description of the types of approaches the proposer plans to use for converting TOIs into a battery of ERNR stimuli that will evoke a desired set of neural processes including the proposed modalities (e.g., images, text).

A candidate list of common ERNRs the proposers intend to pursue, their theoretical basis (including what neural processes they are intended to detect), and a rationale of how using those ERNR processes would support TA1 Goal 3.

HR001122S0032 NEAT 12

Expected parameters that are believed to be key to optimizing ERNR stimuli such that they will be capable of producing responses of sufficient SNR to facilitate TA1 Goal 2 detection objectives and how the proposer will determine what are acceptable ranges for those parameters (e.g., acceptable ranges of stimuli timing and duration to produce sufficient SNR, while also balancing flexibility of information content in individual stimuli).

A specific description of how the proposer envisions establishing ground truth data for supporting accurate evaluations of the sensitivity and specificity of each of the TOIs, including specific examples of the types of TOI tests that are proposed for testing during Phases 1 and 2. It is expected that the ground truth data can be clearly shown to fall into specific knowledge categories (true/false/indeterminate) for individuals at the given time of testing (e.g., it is well defined whether a participant did or did not have knowledge about a TOI the day of the test and prior to the test).

A description for how the proposer envisions identifying any necessary performance needs of this goal as design specifications relative to the other two goals to support development efforts across all three goals and ultimately integrate all components of the final NEAT Process.

Goal 2: Optimize techniques for the rapid and accurate detection and signal processing of neural processes evoked by TOI-related stimuli.

ERNRs are typically hard to detect, with EEG approaches often using methods that require multiple (usually ‘wet’) electrode sensors for measuring data and then averaging over a large number of repetitions (‘trials’) of a particular stimulus. For NEAT, proposals must develop analytic approaches for effectively analyzing data such that ERNRs can be detected with only a few trials and should consider developing methods that would support single trial ERNR detection. This will both support operational utility of NEAT approaches by reducing the time necessary to administer the NEAT Process as well as avoid situations in which ERNRs could show decreasing amplitudes over time due to habituation, familiarization with a given stimuli, or other situations in which stimuli lose their relevance due to repetition. NEAT proposals should also propose studies that provide information regarding the number and complexity (e.g., active/passive, wet/semiwet/dry) of sensors and the information obtained for the NEAT Process.

Additionally, many ERNR approaches (especially when involving EEG) require significant amounts of calibration in order to determine how to detect ERNRs effectively for a given individual. NEAT goals involve quickly and effectively detecting ERNRs from distinct individuals with minimal calibration. Proposed approaches should be effective across a variety of individuals without requiring calibration processes that are overly burdensome.

Successful proposals to TA1 must include the following with regard to Goal 2:

A description of the types of approaches the proposer plans to use for detecting ERNRs with sufficient accuracy to reduce the number of stimuli.

Expected sources of sensor error or sensor noise and strategies for identifying and overcoming them to achieve NEAT goals.

HR001122S0032 NEAT 13

Approaches for assessing the sensitivity and specificity (and other relevant measures such as accuracy) with which the ERNRs can be detected.

A clear description of how the proposer plans to ensure ERNR detection is calibrated or otherwise developed to ensure it can perform accurately for an individual within a single session. This includes a strategy for ensuring that the approach is compatible with operating within the end of phase session duration metrics. Additionally, this should include assessments of the effectiveness of the NEAT approaches across a range of adult ages and demographics and other such characteristics to show that NEAT methods are widely applicable across diverse adult populations.

A strategy for assessing the number of sensors that are necessary for accurately detecting the desired ERNRs as well as the necessary complexity of the sensors (e.g., in the case of EEG whether wet, semi-wet, or dry sensors are sufficient).

A description of how the proposer envisions identifying any necessary performance needs of this goal (e.g., accuracy, acceptable number of repetitions) as design specifications relative to the other two goals (e.g., what would be the acceptable margin of detection error and acceptable number of stimuli trials) to support development efforts across all three goals and ultimately integrate all components of the final NEAT Process.

Goal 3: Create models that produce a metric of a person’s TOI knowledge by aggregating the information obtained from the various ERNRs. The models should be interpretable to explain how the NEAT Process output was generated.

TA1 will develop an approach that uses a battery of ERNR stimuli to probe an individual’s knowledge regarding a given TOI and develop a model(s) capable of integrating that data into a composite score that quantitatively determines what someone believes to be true, false, or indeterminant. The ultimate output of the NEAT Process is to give an interviewer evidence that the interviewee believes a specific TOI is true, false, or indeterminate. For both Phase 1 and Phase 2, the TA2 IV&V team will provide a subset of stimuli and/or TOI investigations that TA1 performers will be expected to integrate into their tests of the TA1 NEAT Process to support a set of standardized assessments of program metrics (e.g., sensitivity/specificity measures).

Successful proposals to TA1 must include the following with regard to Goal 3:

A strategy for the NEAT Process that can provide both high sensitivity and high specificity knowledge detection (see Section I.F on NEAT TA1 metrics).

A clear description of how the sensitivity and specificity of the NEAT Process will be assessed for the different TOIs (e.g., how ground truth data will be used).

Expected challenges and potential solutions for how to integrate information from

ERNRs that come from both central (e.g., EEG data) and peripheral (e.g., pupillometry) sources.

A plan for developing a NEAT Process that leverages ERNRs that reflect a variety of neural processes (e.g., semantic incongruence, error) to the degree necessary to refine the NEAT knowledge score for a given TOI across multiple ERNR stimuli of one or more types and produce a final output score.

HR001122S0032 NEAT 14

A list of potential challenges and possible solutions that may be associated with ensuring the NEAT Process is effective for both single individuals (e.g., any necessary calibration) as well as across a variety of individuals (e.g., use if applied across a significant demographic range).

A research plan for assessing the vulnerability of the NEAT Process to potential confounding variables and mitigating such vulnerabilities. Confounding variables are factors that could interfere with the efficacy of NEAT Process assessments (e.g., clutter interfering with sensors, lack of attention, uniformity of environmental conditions).

Knowledge of such confounding variables will be important for implementing systems capable of collecting quality data and to avoid situations that would reduce performance.

In Phase 1, the plan must identify candidate mechanisms that reduce performance (e.g., collating observations of factors that may spontaneously occur during data collections and experiments that reduce efficacy) and then develop strategies for mitigating such factors (e.g., updates to experimental best practices). In Phase 2, this plan must be expanded to include identifying factors that could potentially be deliberately employed by people who might wish to actively spoof or obfuscate a NEAT assessment (e.g., factors that could be leveraged by a person who desired to conceal suicidal ideation) as well as developing approaches for improving robustness against such obfuscation.

A description for how the proposer envisions identifying any necessary performance needs (e.g., range of ERNR needs and variety of necessary stimuli types, allowable degree of repetition) of this goal as design specifications relative to the other two goals to support development efforts across all three goals and ultimately integrate all components of the final NEAT Process.

In addition, TA1 proposals should include:

A strategy for assessing the Ethical, Legal, and Societal Implications (ELSI) of the

NEAT tool and its potential impacts that could interface with the program ELSI effort (see Section 3.E).

Ultimately, TA1 efforts must achieve the NEAT metrics described in Table 1.

Table 1 - NEAT TA1 Metrics

Phase 1 Phase 2 For 4 of 4 TOIs, classify knowledge:

With ≥ 70% sensitivity and ≥ 70% specificity (not necessarily simultaneously) into categories of what a person believes to be: true, false, or indeterminate.

For 4 of 4 TOIs, demonstrate the capability of classifying knowledge with ≥ 90% sensitivity and ≥ 90% specificity (not necessarily simultaneously) into Phase 1 categories

Perform automated detection of poor sensor signal or misplacement ≥ 95% confidence

For 4 of 4 TOIs, complete NEAT Process in ≤ 120 minutes

HR001122S0032 NEAT 15

2. TA2 – Independent Validation and Verification

TA2 will provide independent assessments of TA1 NEAT Processes as well as subject matter expertise (SME) support to DARPA regarding overall NEAT program status. This will include working with all TA1 performers to craft and implement a subset of NEAT evaluations that can be consistently implemented by the TA1 performers and that support assessment of NEAT Processes. Proposers should describe their specific experience, staff qualifications, and facilities that make the team uniquely qualified to perform TA2 tasks.

Phase 1 IV&V efforts will focus on helping both DARPA and the TA1 performers understand the fundamental limits of NEAT technologies, illuminate appropriate technical approaches, and conduct independent evaluation of TA1 technologies. TA2 will also work with DARPA and the TA1 performers to ensure there are consistent aspects of the evaluations that are carried out by the TA1 performers both for the TA1 Phase 1 Evaluation Report and the TA1 on-site demonstrations (see Table 2 - NEAT TA1 Milestones and Section II.1.D: Program Structure).

Successful proposals to TA2 must include the following:

A description of subject matter expertise to be provided to DARPA, including traveling to and attending meetings A strategy for assessing TA1 proposed techniques for generating NEAT Processes, evaluating TA1 technical approaches, and providing recommendations to DARPA An in-phase and end-of-phase evaluation plan for TA1 performers’ algorithms, technologies, and NEAT Processes A plan to create subsets of stimuli and TOI investigations that TA1 performers can integrate into their own assessment efforts. This will provide the basis for a consistent set of NEAT Process assessments both for the Phase 1 Evaluation Report and the Phase 1 on-site demonstrations (see Table 2 - NEAT TA1 Milestones)

An approach for reviewing TA1 efforts to identify confounding variables as well as a preliminary assessment of the TA1 proposed approaches for mitigating confounding variables

In addition, TA2 proposals should include:

An assessment of the potential portability of the TA1 employed technologies A plan using any additional metrics (i.e., outside of the program metrics in Table 1) that would be useful for better understanding NEAT system performance, as appropriate

Phase 2 In Phase 2, IV&V efforts will continue to evaluate hardware, software, and techniques being developed by the TA1 performer teams as well as organize end of phase demonstrations at a DARPA-chosen location (see Section II.1.D: Program Structure). TA2 proposals must include the following:

A description of subject matter expertise to be provided to DARPA, including traveling to and attending meetings

HR001122S0032 NEAT 16

A strategy for assessing TA1 proposed techniques for improving NEAT Processes, evaluating TA1 technical approaches, and providing recommendations to DARPA

An in-phase and end-of-phase evaluation plan for TA1 performers’ algorithms, technologies, and NEAT processes

A plan to create subsets of stimuli and TOI investigations that TA1 performers can integrate into their efforts to provide consistent evaluations of NEAT Process performances both for the Phase 2 evaluations and the Phase 2 demonstrations

A strategy for working with TA1 performers and DARPA to design, develop, and organize the Phase 2 final demonstrations and report on the results

An approach for reviewing TA1 efforts to identify potential methods of utilizing confounding variables to deliberately obfuscate assessments as well as an assessment of the efficacy of TA1 proposals to mitigate attempts of obfuscating NEAT Processes

In addition, TA2 proposals should include:

A strategy to identify potential operational stakeholders and potential transition end users and report on operational considerations relative to NEAT Processes A description of how the proposer will support assessment modeling of the sensitivity of

NEAT Processes and stimuli to various performance characteristics, e.g., stimulus modality, timing, number and types of sensors, repetition

A plan using any additional metrics (i.e., outside of the program metrics in Table 1) that would be useful for better understanding NEAT system performance, as appropriate

3. Ethical, Legal and Societal Implications (ELSI)

The NEAT program will also include an Ethical, Legal, and Societal Implications (ELSI) assessment effort. This ELSI effort is not being solicited under this BAA; however, organizations or institutions interested in assisting in this effort can inform the NEAT program of their interest via the NEAT@darpa.mil email. The ELSI effort will assist the program and the research teams by:

Providing feedback to the Government and research teams that highlights areas for consideration where NEAT might generate ELSI concerns (e.g., privacy concerns)

Working with research teams’ ELSI efforts to help consider ELSI concerns that may arise regarding how to construct NEAT Processes and how to conduct NEAT research in an ethical manner

Considering the potential societal and legal impacts of the outcomes of NEAT research efforts

Creating opportunities for public awareness and transparency of NEAT Processes as well as avenues for engaging with different scientific communities to garner feedback regarding the scientific foundations of NEAT that would be necessary for future ethical and legal use of NEAT Processes

F. Schedule/Milestones mailto:NEAT@darpa.mil

HR001122S0032 NEAT 17

Performers will be evaluated using a number of metrics and milestones enumerated below.

Attainment of the milestones (indicated by month after award) and metrics for a given phase does not guarantee transition into the next phase of the program. DARPA will also assess efforts on their expected ability to attain subsequent milestones and their expected ability to have a transformative impact on DoD and DARPA priorities.

Technical and Management Milestones

TA1 performers are required to quantify individuals’ knowledge regarding specific TOI using stimuli that evoke ERNRs reflecting neural processes that occur prior to conscious processing.

The usability of these knowledge assessments is ultimately measured by the effectiveness with which the tools developed can establish an individual’s knowledge regarding a specific TOI.

This includes determining (a) whether a person has no specific knowledge or belief regarding a particular TOI (e.g., accurately establishing situations in which an individual’s ground truth reality regarding a TOI is effectively “I don’t know/indeterminate”) and (b) whether there is an absence or presence of knowledge in situations in which a person does have a firm view or knowledge regarding a TOI (e.g., accurately determining if a person’s circumstance is either “I do have knowledge about that TOI/I believe that TOI to be true” or “I don’t have knowledge about that TOI/I believe that TOI to be false”). Significant program milestones geared to show progress to those ends are listed in Table 2 and should be integrated into proposed efforts. If these milestones are not applicable to a particular approach, appropriate alternative milestones at similar intervals must be proposed.

Table 2 - NEAT TA1 Milestones

Month-after-award

TA1 Milestone

3 Regulatory submission for Human Research Protection Office

(HRPO)

9 Preliminary process for converting questions into ERNR evoking stimuli

12 Identify variables that confound data collection for knowledge detection

15 1st TOI (non-biographic) assessment process 18 Mitigation strategies for variables that confound data collection 20 Phase 1 Evaluation Report summarizing NEAT Process performance relative to all TOIs 20 Phase 1 performer site demonstration of all TOIs

Phase 1

24 Phase 1 summary report 30 Identification of potential techniques a person might employ to deliberately obfuscate a NEAT assessment 33 Performance impacts on sensitivity and specificity stemming from types of sensors used and other portable hardware 36 Proposed mechanisms for mitigating attempts to obfuscate or spoof NEAT assessments 42 Phase 2 demonstration of all TOIs

Phase 2

42 Phase 2 summary report

HR001122S0032 NEAT 18

TA2 performer(s) must develop an IV&V strategy to support and evaluate TA1 performer efforts in both phases of the program. This must include the following program milestones in Table 3, but proposals should include additional IV&V objectives and milestones, where appropriate.

Table 3 - NEAT TA2 Milestones

Month-after-award

TA2 Milestone

8 Initial assessment of TA1 proposed techniques for generating NEAT Processes

15 Assessment plan to enable consistent Phase 1 test and evaluation of TA1 NEAT Processes

18 Proposed strategy for testing processes and assessment strategies for the Phase 1 demonstrations

21 Support Phase 1 demonstration for all TOIs 22 Report on TA1 Phase 1 evaluations regarding performance on all

TOIs

Phase 1

24 End of Phase report 27 Report on characteristics of potential future operational use of

Phase 1 NEAT Processes (e.g., practicality of sensors utilized, environmental characteristics of likely interview settings)

30 Initial assessment of TA1 proposed techniques for improving NEAT Processes

33 Assessment plan to enable consistent Phase 2 test and evaluation of TA1 NEAT Processes

36 Preliminary assessment of sensitivity of NEAT Processes to obfuscation

36 Finalize transition plans 39 Plan for Phase 2 demonstrations by TA1 performers at DARPA selected site 42 Implement Phase 2 demonstration for all TOIs

Phase 2

42 End of Phase report

NEAT TA1 Metrics

In order to meet the goals of the NEAT Program, TA1 performers will need to meet the program metrics outlined in Table 1. Proposals must explicitly cite the quantitative and qualitative success criteria that the proposed effort is aiming to achieve at the time of each phase’s program metric evaluation. Additional descriptions for TA1 metrics are provided below.

Topics of Interest NEAT is focused on assessing knowledge regarding four TOIs, defined in Section I.C under Key Words and Definitions. All TOIs must be addressed in both Phase 1 and 2.

Sensitivity, Specificity

HR001122S0032 NEAT 19

While accuracy is important, sensitivity and specificity provide more information for assessing how well NEAT Processes can detect knowledge. NEAT Processes must ultimately be capable of providing high specificity and high sensitivity for knowledge detection without sacrificing one for the other. However, processes do not need to provide high sensitivity and high specificity simultaneously. Approaches can be implemented in a manner that uses a combination of high sensitivity and high specificity modes, which could occur in serial. For example, detecting breast cancer often utilizes a combination of a highly sensitive test (e.g., mammograms) as a screen for the potential of cancer often followed by a highly specific test (e.g., biopsy). Similarly, proposer approaches could sequentially employ a high sensitivity screening to attempt to identify all true positives followed by a highly specific mode that attempts to identify all true negatives.

Proposals should describe the approach for determining sensitivity and specificity and must include clear strategies for how ground truth data will be established and used when determining the sensitivity and specificity of NEAT Processes. Additionally, proposals should describe how confidence could be calculated for the ultimate output class (true, false, or indeterminate) of a given test to clearly indicate to the administrator when the output of the NEAT Process is unclear.

Detection of poor sensor signal or misplacement, ≥95% confidence To facilitate usability, Phase 2 systems must alert the operator if any of the critical sensors are not collecting data of sufficient quality to carry out the NEAT assessment, whether due to improper placement, malfunction, or other sources of interference or noise.

Process time: ≤ 120 minutes The NEAT assessment for each of the TOIs for a given individual must be completely executable within 120 minutes (i.e., a single TOI assessment must be done within 120 min, and this must be achievable for each of the four TOIs). This includes all time necessary to set up and carry out a TOI assessment, including, but not restricted to, instrumenting the person, any instrumentation calibration time necessary for that individual, any baseline engagement or information gathering from the person, any calibration of the ERNRs (e.g., baseline or calibration of EEG ERPs for the individual), converting the TOI into stimuli, administering all stimuli and collection or ERNR data, and analyzing the data to produce the final NEAT output.

Activities that can be realistically anticipated to occur prior to a given individual’s assessment do not count towards the process time metric (e.g., defining generic TOI stimuli that are yet to be calibrated for the individual or building EEG analytics that are based on other individuals and may or may not need adjusting for the administration of a NEAT assessment for an individual).

Additionally, for the final Phase 2 systems, participants must not be expected to remove normal indoor clothing for sensor placement, even if this could be done within the 120-minute metric limit.

Other proposal properties

Both TA1 and TA2 proposals must assume human subject testing will be considered Human Subjects Research (HSR) and plan for the Institutional Review Board (IRB) and secondary Human Research Protection Office (HRPO) reviews that are necessary for Government sponsored HSR in the proposed cost and schedule.