IARPA-BAA-20-04_BRIAR_AMENDMENT 2 - FINAL 01-12-2021C.pdf

IARPA

BROAD AGENCY ANNOUNCEMENT

IARPA-BAA-20-04

Biometric Recognition and Identification at Altitude and Range (BRIAR) Program

IARPA-BAA-20-04

Release Date:

AMENDMENT #2 - JANUARY 12, 2021 –

SEE SECTION 1.D.1.2, p. 14, para. 4

AMENDMENT #1 – JANUARY 5, 2021-

SEE GENERAL INFORMATION

#6, pg. 1

December 10, 2020 i

BROAD AGENCY ANNOUNCEMENT: IARPA-BAA-20-04

BRIAR Program

TABLE OF CONTENTS

GENERAL INFORMATION

SECTION 1: FUNDING OPPORTUNITY DESCRIPTION

1.A. Program Overview

1.A.1. Technical Challenges and Objectives

1.A.2. Program Phases

1.A.2.1. Phase 1

1.A.2.2. Phase 2

1.A.2.3. Phase 3

1.B. Team Expertise

1.C. Program Scope and Limitations

1.D. Program Data

1.D.1. Development Data

1.D.1.1. Government Research Sets

1.D.1.2. Researcher Collections

1.D.1.3. External Data Sources

1.D.1.4. Government Augmented Data

1.D.2. Evaluation Data

1.D.3. Privacy Plan

1.E. Test and Evaluation (T&E)

1.F. Program Metrics

1.G. Program Waypoints, Milestones, and Deliverables

1.G.1. Program Waypoints

1.G.2. Program Milestone, Waypoint, and Deliverables Timeline

1.G.3. Software Deliverable Formatting

1.G.3.1. Program API

1.H. Meeting and Travel Requirements

1.H.1. Workshops

1.H.2. Site Visits

1.I. Period of Performance

1.J. Place of Performance

SECTION 2: AWARD INFORMATION

ii

SECTION 3: ELIGIBILITY INFORMATION

3.A. Eligible Applicants

3.A.1 Organizational Conflicts of Interest (OCI)

3. A.2 Multiple Submissions to the BAA

3. B. U.S. Academic Institutions

3. C. Other Eligibility Criteria

4. A. Proposal Information

4. B. Proposal Format and Content

4. B.1 Volume 1: Technical and Management Proposal

4. B.1.a Section 1: Cover Sheet & Transmittal Letter

4. B.1.b Section 2: Summary of Proposal (see below for page limit)

4. B.1.c. Section 3: Detailed Proposal Information

4.B.1.d. Section 4: Attachments

4.B.2. Volume 2: Cost Proposal (No Page Limit)

4. B.2.a Section 1: Cover Sheet

4. B.2.b. Section 2: Estimated Cost Breakdown

4. B.2.c. Section 3: Supporting Information

4. C. Submission Details

4. C.1. Due Dates

4. C. 2 Proposal Delivery

4. D. Funding Restrictions

SECTION 5: PROPOSAL REVIEW INFORMATION

5.A. Technical and Funding Availability Evaluation Factors

5.A.1. Technical Evaluation Factor (technical criteria listed below)

5. A.1.a. Overall Scientific and Technical Merit

5. A.1.b. Effectiveness of Proposed Work Plan

5. A.1.c. Contribution and Relevance to the IARPA Mission and Program Goal

5. A.1.d Relevant Experience and Expertise

5. A.1.e Resource Realism

5.A.2. Funding Availability Factor

5. A.2.a. Budget Constraints

5. A.2.b. Program Balance

5.B. Method of Evaluation and Selection Process

5.C. Negotiation and Contract Award iii

5.D. Proposal Retention

SECTION 6: AWARD ADMINISTRATION INFORMATION

6.A. Award Notices

6.B. Administrative and National Policy Requirements

6.B.1. Proprietary Data

6.B.2. Intellectual Property

6.B.3. Human Use

6.B.4. Animal Use

6.B.5. Publication Approval

6.B.6. Export Control

6.B.7. Subcontracting

6.B.8. Reporting

6.B.9. System for Award Management (SAM)

6.B.10. Representations and Certifications

6.B.11. Lawful Use and Privacy Protection Measures

6.B.12. Public Access to Results

Appendix A: Templates for Volume 1: Technical Proposal

A.1 Cover Sheet for Volume 1: Technical Proposal

Appendix A.2 Academic Institution Acknowledgment Letter

Appendix A.3 Intellectual Property Rights

Appendix A.4 Organizational Conflicts of Interest Certification Letter

Appendix A.5 Three Chart Summary of the Proposal

Appendix A.6 Research Data Management Plan (RDMP) BAA 20-04

Appendix B: Templates for Volume 2: Cost Proposal

Appendix B.1 Cover Sheet for Volume 2: Cost Proposal

Appendix B.2 Prime Contractor/Subcontractor Cost Element Sheet for Volume 2: Cost Proposal

Appendix B.3 - Software and IP Costs

Appendix B.4 – Travel Costs Trip breakdown

Appendix B.5 – Contract Deliverables Table

GENERAL INFORMATION

This notice constitutes a Broad Agency Announcement (BAA) and sets forth research of interest in the area of whole-body biometric identification. The solicitation process will follow Federal Acquisition Regulation (FAR) Part 35, Research and Development Contracting, as supplemented with additional information included in this notice. Awards based on responses to this BAA will be considered the result of full and open competition.

1. Federal Agency Name – Office of the Director of National Intelligence (ODNI)/Intelligence

Advanced Research Projects Activity (IARPA)

2. Funding Opportunity Title – Biometric Recognition and Identification at Altitude and Range

(BRIAR)

3. Announcement Type – Initial

4. Funding Opportunity Number – IARPA-BAA-20-04

5. Catalog of Federal Domestic Assistance Numbers (CFDA) – Not applicable

6. Questions

Submit questions on administrative, technical, or contractual issues by email to dni-IARPA-BAA-20- 04@iarpa.gov. All requests must include the full name and affiliation of a point of contact. Do not send questions with proprietary content. A consolidated Question and Answer response will be posted on the beta.SAM.gov for Contract Opportunities website (https://beta.sam.gov/) and linked from the IARPA website (https://www.iarpa.gov/index.php/research-programs/BRIAR). No answer will go directly to the submitter. IARPA will accept questions until January 15, 2021 @ 5:00 PM EST. January 6, 2021 @ 5:00 PM EST.

7. Dates

7.1 Posting Date: December 10, 2020

7.2 Proposal Due Date for Initial Round of Selections: February 10, 2021, 4:00 PM EST

7.3 BAA Closing Date: March 10, 2021 4:00 PM EDT

8. Anticipated individual awards – Multiple awards anticipated

9. Types of instruments that may be awarded – Procurement Contracts and Other Transactions 1

10. Agency Points of Contact

ATTN: IARPA-BAA-20-04

Office of the Director of National Intelligence Intelligence Advanced Research Projects Activity Washington, DC 20511 Electronic mail: dni-IARPA-BAA-20-04@iarpa.gov

11. Program Manager (PM) – Lars Ericson, PhD

12. Program Website – http://www.iarpa.gov/index.php/research-programs/BRIAR

13. BAA Summary – The BRIAR Program aims to develop software algorithm-based systems capable of performing whole-body (WB) biometric identification at long-range and from elevated platforms.

Many Intelligence Community (IC) and Department of Defense (DoD) agencies require the ability to identify or recognize individuals under challenging scenarios, such as at long-range (e.g., 300+

1 Procurement Contract: This is a standard government contract that follows the processes, format and terms and conditions as outlined in the Federal Acquisition Regulations (FAR) and supplementing Agency specific regulations.

Other Transaction Authority: Agreements generally are not subject to the federal laws and regulations governing procurement contracts and thus are not required to comply with the Federal Acquisition Regulation (FAR), its supplements, or laws that are limited in applicability to procurement contracts. They may be used with non-traditional contractors under certain circumstances.

meters), through atmospheric turbulence, or from elevated and/or aerial sensor platforms (e.g., ≥20° sensor view angle from watch towers or Unmanned Aerial Vehicles (UAVs)). Expanding the range of conditions in which accurate and reliable biometric-based identification could be performed would greatly improve the number of addressable missions, types of platforms and sensors from which biometrics can be reliably used, and quality of outcomes and decisions.

SECTION 1: FUNDING OPPORTUNITY DESCRIPTION

IARPA often selects its research efforts through the BAA process. The use of a BAA solicitation allows a wide range of innovative ideas and concepts. The BAA will appear on https://beta.sam.gov/, Contract Opportunities, as well as the IARPA website at http://www.iarpa.gov/. The following information is for those wishing to respond to this Program BAA.

This BAA (IARPA-BAA-20-04) is for the Biometric Recognition and Identification at Altitude and Range (BRIAR) program. IARPA is seeking innovative solutions for the BRIAR Program in this BAA. BRIAR is envisioned to be a 4-year effort, beginning approximately August 1, 2021 through July 31, 2025.

1.A. Program Overview The BRIAR Program aims to develop software algorithm-based systems capable of performing whole-body (WB) biometric identification at long-range and from elevated platforms. Many IC DoD agencies require the ability to identify or recognize individuals under challenging scenarios, such as at long-range (e.g., 300+ meters), through atmospheric turbulence, or from elevated and/or aerial sensor platforms (e.g., ≥20° sensor view angle from watch towers or UAVs. Expanding the range of conditions in which accurate and reliable biometric-based identification could be performed would greatly improve the number of addressable missions, types of platforms and sensors from which biometrics can be reliably used, and quality of outcomes and decisions.

Figure 1: BRIAR Program Concept

Face recognition (FR) has increasingly become the biometric modality best suited for IC and DoD missions in which conditions are unconstrained; that is, conditions in which there is no control over factors that would improve the likelihood of being able to make a biometric identification (e.g., orientation of subject to sensor, lighting, facial expression). In the FR literature, these factors are often abbreviated as “PIE” for pose, illumination, and expression. Over the past six years, there have been notable advances in computer vision to facilitate unconstrained FR. The IARPA Janus Program (https://www.iarpa.gov/index.php/research-programs/janus), for example, drove many recent advances.

However, there remain challenges, in particular for diverse face detection, verification, and identification when dealing with low-resolution or noisy imagery (e.g., motion blur, atmospheric turbulence). In addition, limited research has been performed on FR using images captured at high camera pitch angles, such as those collected from security cameras on building tops or from airborne platforms, such as UAVs. This is primarily due to a lack of appropriately collected and sharable research data that represent this type of imagery.

Figure 2: Imagery from volunteer subjects collected under an Institutional Review Board-approved protocol, courtesy of US Army C5ISR Center Night Vision and Electronic Sensors Directorate, demonstrating that images captured under various atmospheric turbulence conditions are challenging for conventional FR.

Imagery captured at long-range or high altitude may also require that additional biometric signatures be used with FR to provide the necessary accuracy or confidence for person identification. Example biometrics that may be relevant include (but are not limited to) WB identification, gait recognition, and/or anthropometric classification (e.g., height, gender). These biometric signatures can enable the identification of a person in a degraded, noisy, or low-resolution video without relying on medium to high resolution views of a subject’s face. WB approaches utilize the shape, movement, measurements, or other aspects of a human form to determine biometric signals that are universal, unique, and permanent human characteristics for the purposes of verification, recognition, or identification.

The BRIAR Program will address these technical needs and challenges by pursuing cutting-edge research into multi-modal biometric recognition algorithms to develop innovations in WB biometrics that fuse FR with other biometric signatures. The research will be supported throughout the program with robust, demographically diverse, domain-relevant biometric data that directly capture the full range of challenging conditions represented in long-range and elevated platform imagery. The BRIAR program will focus on visible wavelength imagery only; research involving biometric matching of non-visible band imagery will be out of scope. Research outcomes from the BRIAR Program are intended to support missions such as counterterrorism, protection of critical infrastructure and transportation facilities, military force protection, and border security.

Performers will conduct research to develop algorithms, to be represented in robust software systems, that can perform biometric verification, recognition, and identification of persons in visible-band video imagery captured under challenging range, atmospheric, and view conditions. In addition, the Performer’s systems will be required to perform person and face detection, association across video frames, and multi-image template2 generation. The developed systems will be required to perform biometric matching on unconstrained imagery in multiple modes – face only, WB, and a multi-modal fusion of FR and WB.

Performers will not be developing custom optical sensors, sensor systems, or sensor platforms (e.g., UAVs), but will be focused on advancing the state-of-the-art for end-to-end multi-modal WB biometric recognition software. Research approaches should pursue algorithm solutions that are agnostic to the collection hardware, including platforms and sensors. The delivered interim and final systems will be evaluated by independent Test and Evaluation (T&E) according to a set of program metric targets. Additional details on program metrics are available in 1.F. Program Metrics.

A robust data collection initiative will be part of the BRIAR Program to assemble diverse and relevant imagery for use in R&D for algorithm training and in T&E for comparing the algorithm accuracy against ground truth. The T&E Team3 will be conducting several field and laboratory data collections throughout the life of the program using Institutional Review Board (IRB)-approved Human Subject Research (HSR) protocols. This data will be made available to Performers for Research and Development (R&D). In addition, Performers will be required to conduct their own supplemental IRB-approved HSR data collections and make that data available to the Program. Additional details on program data can be found in 1.D. Program Data.

The BRIAR Program will pursue rigorous and comprehensive T&E to ensure that research outcomes are well characterized, deliverables are aligned with program objectives, and that algorithm performance is measured across the full range of demographic, sensor, and environmental conditions. Such T&E activities will not only inform Government stakeholders on BRIAR research progress but will also serve as valuable feedback to the Performers to improve their research approaches, algorithm training practices, and system development. The BRIAR Program will work closely with Government leaders in biometrics to continually refine and improve T&E methodologies.

1.A.1. Technical Challenges and Objectives Offerors shall address the following technical challenges (TCs) and objectives in order to meet the goal of BRIAR to advance whole-body and face recognition at long range and from elevated platforms:

• Biometric system capable of matching at long ranges out to 1,000 meters (e.g., body height ≈ 200 pixels, face width ≈ 20 pixels).

• Biometric system capable of matching at severe sensor pitch angles (e.g., 50°).

• Image processing or algorithm development techniques that mitigate the effect of atmospheric turbulence and image degradations on biometric recognition performance.

• Exploitation of the contextual and temporal content in visible-band full motion video imagery to produce robust biometric encoded representations.

2 Biometric Template = A digital representation of the unique features that have been extracted from a biometric sample; used in matching to determine the similarity between two biometric samples.

3 T&E support for the BRIAR program is being provided by three organizations: the Department of Energy Oak Ridge National Laboratory, US Army Combat Capabilities Development Command C5ISR Center Night Vision and Electronic Sensors Directorate, and the National Institute of Standards and Technology.

• Multi-image templates that leverage video imagery to produce a single biometric encoding for each individual, regardless of the number of imagery files or frames.

• Detection and tracking of body and face modalities in video clips, including videos involving moving subjects and moving/changing camera views.

• FR invariant to PIE and robust against degraded visible-band imagery.

• WB biometric recognition invariant to pose, illumination, and clothing changes and degraded visible-band imagery.

• WB biometric recognition capable of cross-view visible-band matching in indoor and outdoor settings.

• Algorithmic approaches robust against incomplete or partially occluded views of biometric signatures, to include both WB and face.4

• Fusion of face with biometric signatures derived from whole body motion, shape, or other characteristics for increased matching performance.

• WB and face recognition that perform accurately across diverse demographic and human body shape groups.

• Ability to adapt algorithms to edge processing environments with restricted computing resources.

• Ability to process streaming imagery and generate results at real-time speeds.

• Algorithms robust against changes in sensor platform and optics.

• Research methods to adapt or transfer algorithm solutions to platform-specific environments for further performance recovery, optimization, or improvements.

1.A.2. Program Phases The BRIAR program will proceed in three (3) phases. Offerors are required to submit a proposal that addresses all three phases. The phases are designed to drive Performers to develop solutions that are suited to a range of biometric capture conditions, robust against imagery degradations, and computationally efficient. The following paragraphs describe the phases and Table 1 summarizes and distinguishes the differences among the program phases.

Table 1: BRIAR Program Phases

Area Phase 1 (18 months) Phase 2 (18 months) Phase 3 (12 months) System R&D Subcomponent Integrated Optimized Accuracy5 Feasible Relevant Confident Probes Unconstrained data with moderate pitch or range Unconstrained data with severe range or pitch

Unconstrained data with severe pitch and range

Gallery Controlled Partially controlled Unconstrained Processing Offline Real-time Edge

4 Possible occlusions could include but are not limited to: objects in the scene, objects being carried, poses or motions by subjects that occlude face or body views, clothing such as long coats or sunglasses. In addition, occlusions may exist in some or all of a given video sequence.

5 Qualitative terms are descriptive of the philosophical focus of each phase; more details on target metrics are available in 1.F Program Metrics.

The ultimate goal of BRIAR is to deliver an end-to-end WB biometric system capable of accurate and reliable verification, recognition, and identification of persons from elevated platforms and at distances out to 1,000 meters across a range of challenging capture conditions.

Figure 3: Example Concept Cartoon for a BRIAR End-to-End System.6 Possible stages of the pipeline could include: 1) Video input with image degradations (e.g., blur); 2) Image enhancement fuses information from frames; 3) Person and face localization; 4) Image processing into biometric-relevant chips/tracklets; 5) Encoding into feature vectors; 6) Fusion of face and WB features; and

7) Similarity matching against reference template(s).

1.A.2.1. Phase 1 Phase 1 will have a duration of 18 months. In Phase 1, Performers will research WB biometrics, FR, and supporting subcomponents. Supporting subcomponents include anything needed to research and demonstrate end-to-end automated WB biometric matching including, but not limited to, person detector(s), person tracker(s), face detector(s), image enhancement pre-processing module(s), atmospheric turbulence mitigation module(s), and/or biometric fusion module(s). Performer research will focus on foundational explorations and investigations into different approaches, theories, and concepts to establish the building blocks of their BRIAR system(s). To mitigate risk and investigate competing approaches, multiple research endeavors in the same subtopic may take place to compare and contrast performance, characteristics, and scientific insights within a given team. During Phase 1, each Performer will carry out one (1) HSR biometric data field collection to capture diverse program-relevant data for R&D. These datasets will be delivered with Unlimited Rights, in accordance with FAR 52.227-14, to allow use by the USG. More details on these researcher-executed collections are available in 1.D.1.2. Researcher Collections.

The Performers will work towards meeting accuracy targets across a range of biometric use cases that demonstrate the feasibility of their research approaches. While the full range of capture conditions are to be considered during Phase 1, the focus will be on biometric matching involving probe videos containing individual unconstrained subjects with either moderate pitch or range conditions. For the purposes of this BAA, moderate pitch shall mean a 20 – 30° view angle and moderate range shall mean 100 – 500 meters, but the pitch and range may be further refined during the course of the program by the BRIAR PM and T&E Team. In Phase 1, Performer R&D and Government T&E of algorithms will involve biometric database galleries containing controlled enrollment videos of diverse subjects presenting WB, face, and gait information. BRIAR systems will only need to be operated in offline processing and demonstrate batch

6 The notional system is not prescriptive. There exist opportunities for innovation or alternate system designs in any or all stages of the BRIAR end-to-end system.

image processing with post-test fusion of modalities. A set of final Deliverables will be due in Phase 1 at a scheduled Milestone (see 1.G.2. Program Milestone, Waypoint, and Deliverables Timeline).

In Phase 1, IARPA will provide datasets and a BRIAR program Application Programming Interface (API) developed by the BRIAR T&E Team. This Government Furnished Information (GFI) will facilitate productive and useful research in a consistent and standardized manner. Data provided by the T&E Team will be partitioned into development and evaluation datasets. Datasets will be provided at approximately six-month intervals to Performers; development datasets will be for use in algorithm R&D and training, and evaluation datasets for performance characterization. Evaluation of Performer progress towards program metrics will primarily utilize evaluation datasets that are not sequestered but shared with Performers so as to facilitate communication and internal Performer error analyses. Three research datasets and three evaluation datasets will be provided in Phase 1 in addition to Performer self-collected data. The baseline API provided in Phase 1 is to standardize system and subcomponent design and streamline comparative evaluations. More details on the datasets are available in 1.D. Program Data and more details on the API are available in 1.G.3.1. Program API.

1.A.2.2. Phase 2 Phase 2 will have a duration of 18 months. In Phase 2, the Performers will focus on R&D to explore how best to combine their modular research approaches with the objective of integrating and optimizing subcomponent modules into an end-to-end WB biometric matching system. By the midpoint of the phase, the Performers will be required to have demonstrated a prototype system that integrates person detection, face detection, image degradation mitigation technique(s), and the fusion of FR and at least one WB modality in a single, automated pipeline. During Phase 2, each Performer will carry out two (2) HSR biometric data field collections to capture diverse program-relevant data for R&D. These datasets will be delivered with Unlimited Rights, in accordance with FAR 52.227-14, to allow use by the USG. More details on these researcher-executed collections are available in 1.D.1.2. Researcher Collections.

The Performers will work towards meeting accuracy targets across a range of biometric use cases that demonstrate performance relevant to United States Government (USG) needs. While the full combination of capture conditions is to be considered during Phase 2, the focus will be on biometric matching involving probe videos containing unconstrained subjects with either moderate to severe pitch or range conditions.

Severe pitch is defined as 30 – 50° view angle and severe range as 500 – 1,000 meters. Moderate pitch or range will continue to be relevant in Phase 2. Probe videos containing small groups of up to three (3) subjects, in addition to single individuals, will be introduced in Phase 2 for detection, tracking, and encoding of biometric signatures. In addition, Performers will begin addressing biometric matching with partially occluded views of subjects for WB, gait, or face views. Possible occlusions could include but are not limited to: objects in the scene, objects being carried, poses or motions by subjects that occlude face or body views, clothing such as long coats or sunglasses. In addition, occlusions may exist in some or all of a given video sequence. In Phase 2, Performer R&D and Government T&E of algorithms will involve biometric database galleries containing both controlled enrollment videos and partially controlled enrollment imagery of diverse subjects presenting WB, face, and gait information. For purposes of this BAA, partially controlled is defined as short range enrollment with limited instruction to subjects or non-ideal lighting. Examples include an outdoor enrollment or a non-cooperative person in an indoor hallway.

Performers will also focus their efforts on identifying, characterizing, and addressing imaging data that are proving particularly challenging to their research approaches, subcomponent modules, or integrated systems, to include specific BRIAR environmental conditions, platform/sensor types or behaviors, or demographic or body-type populations. In Phase 2, BRIAR systems will be required to operate in a real-time configuration, processing both batch video files and streaming videos on commodity computing resources. For the purposes of this BAA, “real time” is defined as a single 1080p video stream (i.e., 1920 x 1080 resolution) at 30 fps video. Two sets of interim Deliverables and a set of final Deliverables will be due in Phase 2 at scheduled Milestones (see 1.G.2. Program Milestone, Waypoint, and Deliverables Timeline).

In Phase 2, the IARPA will be providing additional datasets and an expanded version of the BRIAR API.

Three research datasets and three evaluation datasets will be provided in Phase 2 in addition to Performer self-collected data. More details on the datasets are available in 1.D. Program Data and more details on the API are available in 1.G.3.1. Program API.

1.A.2.3. Phase 3 Phase 3 will have a duration of 12 months. In Phase 3, the Performers will focus on advancing the coupled integration of all subcomponents and optimizing biometric signal fusion in the end-to-end system. R&D will expand image processing and fusion to include secondary components and improving the adaptive nature of the pipeline to the full range of capture and environmental conditions. Research will also pursue methods to retrain or adapt the matching algorithms to transfer to new sensor and/or platform conditions as well as methods to optimize performance if deployed on a specific dedicated sensor or platform system.

New sensors and platforms will be identified early in Phase 3 by the BRIAR PM and T&E Team, in coordination with USG partners. Performers will continue efforts to address imaging data that are proving particularly challenging to their BRIAR systems across diverse environmental, sensor, platform, and human subject types. During Phase 3, each Performer will carry out one (1) HSR biometric data field collection to capture diverse program relevant data for R&D. These datasets will be delivered with Unlimited Rights, in accordance with FAR 52.227-14, to allow use by the USG. More details on these researcher-executed collections are available in 1.D.1.2. Researcher Collections.

The Performers will work towards meeting performance targets across a range of biometric use cases that demonstrate reliable accuracy and low false match rates. The full combination of capture conditions will be considered during Phase 3 R&D. Biometric matching will involve probe videos containing unconstrained diverse subjects with any combination of moderate to severe pitch and/or range conditions.

Research will continue advancing methods to mitigate or utilize partially occluded views of biometric signatures while minimizing the impact to matching accuracy. In Phase 3, Performer R&D and Government T&E of algorithms will include biometric database galleries containing fully unconstrained videos of non-cooperative diverse subjects captured at small to medium range and/or pitch views presenting WB, face, and gait information. Small range is defined as 1 – 100 meters; small pitch is defined as 0 – 20°. In Phase 3, Performers will develop a version of their main BRIAR system that will be capable of processing real-time streaming video at 15 fps on an edge computing platform with matching accuracies to Phase 2 metric targets. Note that this requirement does not apply to the main BRIAR system, which is to process real-time streaming or batch videos in commodity desktop computing environments. One set of interim Deliverables and a set of final Deliverables will be due in Phase 3 at scheduled Milestones (see 1.G.2. Program Milestone, Waypoint, and Deliverables Timeline).

In Phase 3, the IARPA will be providing additional datasets and a completed version of the BRIAR API.

Two research datasets and two evaluation datasets will be provided in Phase 3 in addition to Performer self-collected data. More details on the datasets are available in 1.D. Program Data and more details on the API are available in 1.G.3.1. Program API.

1.B. Team Expertise Collaborative efforts and teaming among Offerors are highly encouraged. It is anticipated that teams will be multidisciplinary and may include expertise in one or more of the disciplines listed below. This list is included only to provide guidance for the Offerors; satisfying all the areas of technical expertise below is not a requirement for selection and unconventional or innovative team expertise may be needed based on the proposed research. Proposals should include a description and the mix of skills and staffing that the Offeror determines will be necessary to carry out the proposed research and achieve program metrics.

• Biometrics theory, applications, and systems with specific emphasis on face, WB, and gait modalities

• Computer vision to include object detection, tracking, camera modeling

• Image processing to address noise, degradations, turbulence, and motion blur

• Optics and sensors

• UAV sensor design and operation, as well as UAV flight concepts of operations

• HSR and data collections

• Anatomy and kinesiology

• Modeling and simulation

• Machine learning, deep learning, or hierarchical modeling

• Artificial intelligence

• Systems integration

• Systems engineering

• Software engineering

• Data reduction and analysis

• Distributed processing, stream processing

1.C. Program Scope and Limitations Proposals shall explicitly address all the following:

• Underlying theory: Proposed strategies to meet program-specified metrics must have firm theoretical bases that are described with enough detail that reviewers will be able to assess the viability of the approaches. Proposals shall properly describe and reference previous work upon which their approach is founded.

• Research & Development approach: Proposals shall describe the technical approach to meeting program metrics.

• Technical risks: Proposals shall identify technical risks and proposed mitigation strategies for each.

• Software development: Proposals shall describe the approach to software architecture and integration.

The following areas of research are out of scope for the BRIAR program:

• Research that does not have strong theoretical and experimental foundations.

• Development of optical sensor hardware.

• Development of sensor platforms, such as UAVs, vehicles, aerostats, towers, or camera systems.

• Development of other hardware.

• Approaches that rely on secondary external data signals, such as cell tower tracking or non-optical sensors, to carry out biometric matching tasks.

• Research that utilizes proprietary data that are not made available to other Performers.

• Methods that require a human-in-the-loop as part of the integrated end-to-end system in Phases 2-

3.

• Approaches that consist merely of integrating currently existing software.

• Research involving biometric matching of non-visible band imagery (e.g., Near Infrared, Short-

Wave Infrared, Mid-Wave Infrared, or Long-Wave Infrared)

The use of additional sensors, imaging or sensing modalities, filters, or calibration targets to assemble training data sets or to support modeling and simulation research is in scope if it is relevant to the proposed research approach.7 However, testing will be restricted to data collected in the visible spectrum without any additional metadata, data feeds, or sensor inputs.

Delivered software will be evaluated by an independent team on sequestered and shared evaluation datasets.

During each performance period, Performers will build prototype algorithms, and subcomponent modules, and/or systems for end-to-end WB and FR under challenging imaging conditions that will be run and evaluated by the T&E Team. Testing protocols do not allow for expert operators, human-in-the-loop operation, or any operations not deemed “turnkey”. However, systems or algorithms that have been trained using human-in-the-loop methods may be submitted, provided they run autonomously.

1.D. Program Data BRIAR is a unique research program in the field of computer vision and biometrics in that data that represent the domain of interest are not commonly available in the academic community due to lack of commercial interest, complexity of collection logistics, and high costs to collect. However, for BRIAR to facilitate innovative R&D and achieve program metrics, diverse program data in sufficient quantities are needed for development and statistically reliable evaluations. As a result, the program will include robust and explicit data collection by the BRIAR T&E Team and R&D Performers.

The program will include two types of data – Development Data and Evaluation Data. The data will consist of WB, gait, and face imagery collected with a range of conditions, sensors, and platforms involving a diverse group of demographic and body types. Development data will be utilized by Performers, as they see fit, to conduct research, development, error analysis, and algorithm training. Evaluation data will be explicitly excluded from any algorithm training approaches and will be used by both Performers and the T&E Team for performance evaluations of subcomponents, modules, and systems measured against the BRIAR program metrics. A summary of HSR collected datasets across the entire life of the program is included in Table 2.

Servers with Globus Endpoints8 will be used to facilitate the transfer of datasets between the Government, the T&E Team, and individual Performers. The free version of Globus may be adequate for the data transfer needs of the program. Offerors shall include the cost and resources to acquire, implement, and maintain the necessary computing infrastructure. Organizations may utilize existing Globus Endpoint capabilities if appropriate. High speed internet connections are needed and hundreds of terabytes (TB) of data are expected to be made available to the Performers over the course of the project. The memory size of each dataset release is estimated at ~50 TB.

Table 2: BRIAR Program Datasets

Phase Type Origin Dataset9 Month # Subjects (Probes)10,11

# Subjects (Gallery)11

1 Development T&E BRS1 1 200 200 1 Evaluation T&E BTS1 1 100 25012 1 Development T&E BRS2 7 200 200 1 Evaluation T&E BTS2 7 200 50012

7 Includes the use of non-visible imagery to support supplemental research or algorithm training activities.

8 https://www.globus.org/ 9 BRS = BRIAR Research Set, BTS = BRIAR Test Set, BRC = BRIAR Researcher Collection 10 Minimum number of unique individuals in the probe set with mated entries in the dataset gallery.

11 Each BTS evaluation test set will be a superset of previously released BTS.

12 Gallery includes additional distractor subjects for improved statistical analysis.

Phase Type Origin Dataset9 Month # Subjects (Probes)10,11

# Subjects (Gallery)11

1 Development R&D BRC113 13 8009 8009 1 Development T&E BRS3 13 200 200 1 Evaluation T&E BTS3 13 300 75012 2 Development T&E BRS4 19 200 200 2 Evaluation T&E BTS4 19 400 1,00012 2 Development R&D BRC213 25 6009 6009 2 Development T&E BRS5 25 200 200 2 Evaluation T&E BTS5 25 500 1,25012 2 Development R&D BRC313 31 6009 6009 2 Development T&E BRS6 31 200 200 2 Evaluation T&E BTS6 31 600 1,50012 3 Development T&E BRS7 37 200 200 3 Evaluation T&E BTS7 37 700 1,75012 3 Development R&D BRC413 43 4009 4009 3 Development T&E BRS8 43 200 200 3 Evaluation T&E BTS8 43 800 2,00012

TOTAL Development T&E + R&D

-- -- 4,00014 4,00014

TOTAL Evaluation T&E -- -- 800 2,000

1.D.1. Development Data Development data will be utilized by Performers, as they see fit, to conduct internal research, development, error analysis, and algorithm training.

It is anticipated that Performers will need development data specific to each research approach and each subcomponent module of their BRIAR system(s) to achieve program goal, objectives, and metrics.

However, because Performer approaches will differ, it is anticipated that Performers may need different development data depending on their individual approaches.

Proposals must specify the development data needed to carry out the proposed research and what data characteristics are necessary for their approach(es) to be successful at meeting program objectives.

Offerors’ unique data needs in a particular technical area should not be interpreted as requirements input into the program’s T&E data collections, but part of a comprehensive research approach that communicates a possible path towards meeting BRIAR program objectives.

There will be four types of development data used in the BRIAR program:

• Government Research Sets – Datasets collected, annotated, and curated by the T&E Team and provided to Performers for R&D.

• Researcher Collections – Datasets collected, annotated, and curated by R&D Performers. These datasets will be delivered with Unlimited Rights, in accordance with FAR 52.227-14, to allow use by the USG.

13 Each Performer will conduct a field data collection and make the data available to T&E and other R&D Performers.

14 Calculation assumes four Performers in Phase 1, three Performers in Phase 2, and two Performers in Phase 3.

Numbers may increase or decrease based on the actual number of Performers in each phase.

• External Data Sources – Data obtained by Performers that are available from third parties or that have been collected by a Performer outside of BRIAR with rights to allow all Performers to use them for BRIAR.

• Government Augmented Data – Data resulting from the T&E Team applying modeling and simulation tools to transform Performer-identified or provided datasets.

1.D.1.1. Government Research Sets The Government will be providing BRIAR Research Sets (BRS) and BRIAR Test Sets (BTS) to Performers throughout the program. See Table 2 for a full summary of all BRS datasets. These datasets will include visible-band WB biometric video imagery with ground truth computer vision annotations, identity labels, and metadata information. The datasets will include field collected imagery, enrollment imagery, and may include imagery augmented using Government Owned Technology Solution (GOTS) modeling and simulation tools. The datasets may also include platform or sensor data (e.g., roll, pitch, yaw, or camera angle), but Performer systems should neither expect nor require these metadata inputs during operation or at test time.

The T&E Team will be carrying out HSR biometric field data collections approximately every six months throughout the life of the program. Collections will occur in a variety of outdoor terrain and temperature environments to capture wide variations in atmospheric turbulence. Severe weather (e.g., snow, heavy rain, fog, or heavy winds) will be out of scope for these field collections. This data will be collected using many different optical sensors and platforms at ranges out to 1,000 meters and sensor view angles as steep as 50°.

The specifics of these field collections will be determined by the T&E Team during the program, but for purposes of this BAA, Offerors can assume the following characteristics within each of the BRS datasets:

• Full motion visible-band video imagery of varying durations (e.g., seconds up to minutes)

• A minimum of 200 consenting volunteer subjects with a diverse range of demographic and body type characteristics

• A minimum of two different clothing configurations for each subject

• Video containing scenes of single individuals up to small groups of three people

• Ground-truth enrollment for gait, WB, anthropometry, and face biometrics o Enrollments are to follow NIST/ANSI standards or best practices for biometric enrollments o Enrollment conditions will depend on the Phase of the program (see 1.A.2.

Program Phases) o Enrollment imagery for each subject to include a combination of video and still photos

• Imagery captured from the following types of platforms:

o Minimum of one (1) type of Class 2 fixed-wing UAV o Minimum of one (1) type of Class 2 quadcopter UAV o Elevated surveillance cameras mounted on poles or buildings o Man-portable telephoto camera systems o Other platforms determined by the Government (e.g., Class 1 quadcopter UAV)

• Imagery captured from multiple optical sensors with varying resolution, optics, stabilization o Visible wavelength imagery only o Threshold # of different sensors = 10 o Objective # of different sensors = 20

BRS datasets will be provided to Performers with ground-truth annotations and associated metadata information to include the following:

• WB bounding boxes,

• Face bounding boxes,

• Tracklet associations within full motion video,

• Identity and sighting instance labels,

• Capture range,

• Capture pitch,

• Subject pose,

• Human presence (in scene),

• Subject demographic information,

• Environmental conditions,

• Capture sensor or platform characteristics, and

• Atmospheric turbulence conditions.

Additional platform or sensor characteristics may be included for development or covariant analysis.

Examples may include UAV telemetry data, extrinsic or intrinsic camera information. Specific details will not be made available until program kickoff. Proposers with research approaches that rely on or benefit from these data types during algorithm development are encouraged to account for such data needs in planning their BRIAR Researcher Collections (BRCs), see 1.D.1.2. Researcher Collections.

In addition to real imagery captured during field collections, the T&E Team may utilize GOTS modeling and simulation tools to produce augmented data. It is anticipated that these tools will allow well characterized captured data to be transformed into simulated data with expanded or different characteristics (e.g., range, atmospheric turbulence, sensor resolution). This technique will be used by the T&E Team to address gaps in captured data that are relevant to BRIAR program objectives. Priority will be placed on capturing real data across the full range of conditions with simulated data serving as a mitigation measure.

BRS and BTS datasets will include both lossless and lossy low-resolution imagery to reflect a wide range of use cases and types of imagery. Research approaches shall address both types of imagery.

Details regarding the specific UAV and sensor platforms to be used by the T&E Team during field data collections are not available at this time. Offerors shall use their best judgement based on the information provided here on suitable or proxy platforms when planning and proposing their research approaches.

Offerors are encouraged to take into account that an objective of BRIAR is the development of algorithms that are robust to changes in sensor platform and optics.

1.D.1.2. Researcher Collections Each Performer will be required to plan, execute, and annotate several BRIAR Researcher Collections (BRCs) to supplement their own development data and to share among the other BRIAR Performers and the T&E Team. For each Performer, Phase 1 will have one BRC, Phase 2 will have two BRCs, and Phase 3 will have one BRC. Each will be similar in concept, but the specifics of the capture conditions, sensors, and platforms will be determined during the BRC planning stage by the Government with input from the Performers. As part of their proposal, an Offeror may propose additional BRCs beyond the required number. The proposal shall describe how each BRC will be used in the R&D and justify the need for any data collections beyond the required BRCs. BRCs serve two roles in the program – 1) Expand the volume of domain-relevant data available to the entire program, and 2) Provide for customized data collections to directly support a specific Performer’s research approach needs. As such, BRCs should be viewed as a mechanism to address any research data needs that may fall outside of the scope of the BRIAR Government Research Sets. Performers cannot control the quality and depth of data from other teams’ collections, but they can ensure their own collections meet or exceed their own research needs. Annotated BRC datasets will be provided to the Government as Deliverables on a set schedule, as described in 1.G.2.

Program Milestone, Waypoint, and Deliverables Timeline.

Each Performer’s BRC will be an IRB-approved HSR biometric field data collection involving the capture of WB and face video imagery across different visible-band sensors and platforms. The specifics of each BRC are intended to support that Performer’s research approach(es) and will have flexibility to accommodate their needs in pursuit of BRIAR program objectives. In general, each BRC will be similar to Government BRS datasets with differences indicated in bold, and are required to include all of the elements described below:15

• Full motion visible-band video imagery of varying durations (e.g., seconds up to minutes)

• A minimum of 200 unique16 consenting volunteer subjects with a diverse range of demographic and body type characteristics

• Video containing scenes of single individuals up to small groups of three people

• A minimum of two different clothing configurations for each subject

• Ground-truth enrollment for gait, WB, anthropometry, and face biometrics o Enrollments are to follow NIST/ANSI standards or best practices for biometric enrollments o Enrollment conditions will depend on the Phase of the program (see 1.A.2.

Program Phases) o Enrollment imagery for each subject to include a combination of video and still photos

• Imagery captured from the following types of platforms:

o R&D fixed-wing UAV o R&D quadcopter UAV o Elevated surveillance camera mounted on poles or buildings o Man-portable telephoto camera system

• Imagery captured from multiple optical sensors with varying resolution, optics, stabilization o Visible wavelength imagery only o Threshold # of different sensors = 417 o Objective # of different sensors = 8

There is no requirement regarding the time period over which BRCs must be collected. However, annotated BRC datasets are due as deliverables at specified times throughout the program, as described in 1.G. Program Waypoints, Milestones, and Deliverables.

For planning purposes, Offerors are to use their best judgement based on the information provided here for their selection of UAV and sensor equipment when planning and proposing their BRCs. IARPA will not provide a list of approved or acceptable UAVs as part of the BRIAR BAA. At program kickoff, Performers will be provided a list of UAV platforms that are known to meet the requirements of the BRIAR program.

The list may include either complete UAV platforms and/or a list of subcomponents with instructions on assembling and operating the UAV. The naming of any UAV or component on this list is not a product endorsement. Offerors may propose other platforms but, prior to use in any BRC, shall obtain verification from IARPA that the proposed platform will meet BRIAR program requirements. The UAVs to be used in the BRIAR program by Performers will be suitable for operation at both a DoD training facility (pending approvals) and or a civilian facility (e.g., university campus) and allowable for purchase and operation by third party organizations with foreign national staff (i.e., non-US citizens). Included with each UAV on the IARPA-provided list will be a selection of 1 – 3 sensor payloads for Performers to utilize in field BRC

15 BRC requirements should be considered minimum threshold requirements. Researchers may propose additional collection details that go beyond the minimum (e.g., larger subject counts, additional changes of clothing), but should include a justification related to their proposed research approach.

16 Each BRC is to include data from a minimum of 200 new subjects that have not participated in previous BRCs.

Subjects may participate in multiple future BRCs but will not count towards the unique subject requirement.

17 Each of the 200 subjects should be captured in imagery from a minimum of four (4) different sensors, such that a given individual is represented in imagery from 4+ sensors.

events for capturing imagery aligned with BRIAR program objectives. The BRIAR PM may opt to revise the recommended BRIAR UAV platforms and sensors at the start of each program phase or as needed during the program.

Performers will be responsible for annotating all BRC data for their R&D needs.