BRIAR_BAA_TECH_SECTION_DRAFT-1.pdf

SECTION 1: FUNDING OPPORTUNITY DESCRIPTION

The Intelligence Advanced Research Projects Activity (IARPA) often selects its research efforts through the Broad Agency Announcement (BAA) process. The use of a BAA solicitation allows a wide range of innovative ideas and concepts. The BAA will appear on https://beta.sam.gov/, as well as the IARPA website at http://www.iarpa.gov/. The following information is for those wishing to respond to this Program BAA.

This BAA (IARPA-BAA-20-04) is for the Biometric Recognition and Identification at Altitude and Range (BRIAR) Program. IARPA is seeking innovative solutions for the BRIAR Program in this BAA. The BRIAR Program is envisioned to begin in July 2021.

1.A. Program Overview The BRIAR Program aims to develop software algorithm-based systems capable of performing whole-body (WB) biometric identification at long-range and from elevated platforms. Many Intelligence Community (IC) and Department of Defense (DoD) agencies require the ability to identify or recognize individuals under challenging scenarios, such as at long-range (e.g., 300+ meters), through atmospheric turbulence, or from elevated and/or aerial sensor platforms (e.g., ≥20° sensor view angle from watch towers or Unmanned Aerial Vehicles (UAVs)). Expanding the range of conditions in which accurate and reliable biometric-based identification could be performed would greatly improve the number of addressable missions, types of platforms and sensors from which biometrics can be reliably used, and quality of outcomes and decisions.

Figure 1: BRIAR Program Concept

Face recognition (FR) has increasingly become the biometric modality best suited for IC and DoD missions in which conditions are unconstrained; that is, conditions in which there is no control over factors that would improve the likelihood of being able to make a biometric identification (e.g., orientation of subject to sensor, lighting, facial expression). In the FR literature these factors are often abbreviated as “PIE” for pose, illumination, and expression. Over the past six years, there have been notable advances in computer vision to facilitate unconstrained FR. The IARPA Janus Program (https://www.iarpa.gov/index.php/research-programs/janus), for example, drove many recent advances. However, there remain challenges, in particular for diverse face detection, verification, and identification when dealing with low-resolution or noisy imagery (e.g., motion blur, atmospheric turbulence). In addition, limited research has been performed on FR using images captured at high camera pitch angles, such as those collected from security cameras on building tops or from airborne platforms, such as UAVs. This is primarily due to a lack of appropriately collected and sharable research data that represent this type of imagery.

Figure 2: Imagery from volunteer subjects collected under an Institutional Review Board-approved protocol, courtesy of US Army C5ISR Center Night Vision and Electronic Sensors Directorate, demonstrating that images captured under various atmospheric turbulence conditions are challenging for conventional FR.

Finally, imagery captured at long-range or high altitude may also require that additional biometric signatures be used with FR to provide the necessary accuracy or confidence for person identification. Example biometrics that may be relevant include (but are not limited to) WB identification, gait recognition, and/or anthropometric classification (e.g., height, gender). These biometric signatures can enable the identification of a person in a degraded, noisy, or low-resolution video without relying on medium to high resolution views of a subject’s face. WB approaches utilize the shape, movement, measurements, or other aspects of a human form to determine biometric signals that are universal, unique, and permanent human characteristics for the purposes of verification, recognition, or identification.

The BRIAR Program will address these technical needs and challenges by pursuing cutting-edge research into multi-modal biometric recognition algorithms to develop innovations in WB biometrics that fuse FR with other biometric signatures. The research will be supported throughout the program with robust, demographically diverse, domain-relevant biometric data that directly capture the full range of challenging conditions represented in long-range and elevated platform imagery. The BRIAR program will focus on visible wavelength imagery only; research involving non-visible imagery will be out of scope. Research outcomes from the BRIAR Program are intended to support missions such as protection of critical infrastructure and transportation facilities, military force protection, and border security.

Performers will conduct research to develop algorithms, to be represented in robust software systems, that can perform biometric verification, recognition, and identification of persons in visible-band video imagery captured under challenging range, atmospheric, and view conditions.

In addition, the Performer’s systems will be required to perform person and face detection, association across video frames, and multi-image template1 generation. The developed systems will be required to perform biometric matching on unconstrained imagery in multiple modes – face only, WB, and a multi-modal fusion of FR and WB. Performers will not be developing custom optical sensors, sensor systems, or sensor platforms (e.g., UAVs), but will be focused on advancing the state-of-the-art for end-to-end multi-modal WB biometric recognition software. The delivered interim and final systems will be evaluated by independent Test and Evaluation (T&E) according to a set of program metric targets. Additional details on program metrics are available in SECTION 1.F. Program Metrics.

A robust data collection initiative will be part of the BRIAR Program to assemble diverse and relevant imagery for use in R&D for algorithm training and in T&E for comparing the algorithm accuracy against ground truth. The T&E Team will be conducting several field and laboratory data collections throughout the life of the program using Institutional Review Board (IRB)-approved Human Subject Research (HSR) protocols. These data will be made available to Performers for Research and Development (R&D). In addition, Performers will be required to conduct their own supplemental IRB-approved HSR data collections and make that data available to the Program. Additional details on program data can be found in SECTION 1.D. Program Data.

The BRIAR Program will pursue rigorous and comprehensive T&E to ensure that research outcomes are well characterized, deliverables are aligned with program objectives, and that algorithm performance is measured across the full range of demographic, sensor, and environmental conditions. Such T&E activities will not only inform IARPA and Government stakeholders on BRIAR research progress but will also serve as valuable feedback to the Performers to improve their research approaches, algorithm training practices, and system development. The BRIAR Program will work closely with Government leaders in biometrics (e.g., National Institute of Standards and Technology) to continually refine and improve T&E methodologies.

1.A.1. Technical Challenges and Objectives Offerors shall address the following technical challenges (TCs) and objectives in order to meet the goal of BRIAR to advance whole-body and face recognition at long range and from elevated platforms:

1 Biometric Template = A digital representation of the unique features that have been extracted from a biometric sample; used in matching to determine the similarity between two biometric samples.

• Biometric system capable of matching at long ranges out to 1,000 meters (e.g., body height ≈ 200 pixels, face width ≈ 20 pixels).

• Biometric system capable of matching at severe sensor pitch angles (e.g., 50°).

• Image processing or algorithm development techniques that mitigate the effect of atmospheric turbulence and image degradations on biometric recognition performance.

• Exploitation of the contextual and temporal content in visible-band full motion video imagery to produce robust biometric encoded representations.

• Multi-image templates that leverage video imagery to produce a single biometric encoding for each individual, regardless of the number of imagery files or frames.

• Detection and tracking of body and face modalities in video clips, including videos involving moving subjects and moving/changing camera views.

• FR invariant to PIE and robust against degraded visible-band imagery.

• WB biometric recognition invariant to pose, illumination, and clothing changes and degraded visible-band imagery.

• WB biometric recognition capable of cross-view visible-band matching in indoor and outdoor settings.

• Algorithmic approaches robust against incomplete or partially occluded views of biometric signatures, to include both WB and face.

• Fusion of face with biometric signatures derived from whole body motion, shape, or other characteristics for increased matching performance.

• WB and face recognition that perform accurately across diverse demographic and human body shape groups.

• Ability to adapt algorithms to edge processing environments with restricted computing resources.

• Ability to process streaming imagery and generate results at real-time speeds.

• Algorithms robust against changes in sensor platform and optics.

• Research methods to adapt or transfer algorithm solutions to platform-specific environments for further performance recovery, optimization, or improvements.

1.A.2. Program Phases The BRIAR program will proceed in three (3) phases. The phases are designed to drive Performers to develop solutions that are suited to a range of biometric capture conditions, robust against imagery degradations, and computationally efficient. The following paragraphs describe the phases and Table 1 summarizes and distinguishes the differences among the program phases.

Table 1: BRIAR Program Phases

Area Phase 1 (18 months) Phase 2 (18 months) Phase 3 (12 months) System R&D Subcomponent Integrated Optimized Accuracy2 Feasible Relevant Confident Probes Unconstrained data with moderate pitch or range Unconstrained data with severe range or pitch

Unconstrained data with severe pitch and range

Gallery Controlled Partially controlled Unconstrained Processing Offline Real-time Edge

The ultimate goal of BRIAR is to deliver an end-to-end WB biometric system capable of accurate and reliable verification, recognition, and identification of persons from elevated platforms and at distances out to 1000 meters across a range of challenging capture conditions.

Figure 3: Example Concept Cartoon for a BRIAR End-to-End System. Possible stages of the pipeline could include: 1) Video input with image degradations (e.g., blur); 2) Image enhancement fuses information from frames; 3) Person and face localization; 4) Image processing into biometric-relevant chips/tracklets; 5) Encoding into feature vectors; 6) Fusion of face and WB features; and

7) Similarity matching against reference template(s).

1.A.2.1. Phase 1 Phase 1 will have a duration of 18 months. In Phase 1, Performers will research WB biometrics, FR, and supporting subcomponents. Supporting subcomponents include anything needed to research and demonstrate end-to-end automated WB biometric matching including, but not limited to, person detector(s), person tracker(s), face detector(s), image enhancement pre-processing module(s), atmospheric turbulence mitigation module(s), and/or biometric fusion module(s).

Performer research will focus on foundational explorations and investigations into different approaches, theories, and concepts to establish the building blocks of their BRIAR system(s). To mitigate risk and investigate competing approaches, multiple research endeavors in the same subtopic may take place to compare and contrast performance, characteristics, and scientific

2 Qualitative terms are descriptive of the philosophical focus of each phase; more details on target metrics are available in SECTION 1.F.

insights within a given team. During Phase 1, each Performer will carry out one (1) HSR biometric data field collection to capture diverse program-relevant data for R&D. These datasets shall be delivered with a minimum of Government Purpose Rights to allow use by USG and its contractors (including all other BRIAR R&D Performers). More details on these researcher-executed collections are available in SECTION 1.D.1.2. Researcher Collections.

The Performers will work towards meeting accuracy targets across a range of biometric use cases that demonstrate the feasibility of their research approaches. While the full range of capture conditions are to be considered during Phase 1, the focus will be on biometric matching involving probe videos containing individual unconstrained subjects with either moderate pitch or range conditions. For the purposes of this BAA, moderate pitch shall mean a 20 – 30° view angle and moderate range shall mean 100 – 500 meters, but may be further refined during the course of the program by the BRIAR PM and T&E Team. In Phase 1, Performer R&D and Government T&E of algorithms will involve biometric database galleries containing controlled enrollment videos of diverse subjects presenting WB, face, and gait information. BRIAR systems will only need to be operated in offline processing and demonstrate batch image processing with post-test fusion of modalities. One set of interim Deliverables and a set of final Deliverables will be due in Phase 1 at scheduled Milestones (see SECTION 1.G.2. Program Milestone, Waypoint, and Deliverables Timeline).

In Phase 1, the BRIAR T&E Team will provide datasets and a BRIAR program Application Programming Interface (API). This Government Furnished Information (GFI) will facilitate productive and useful research in a consistent and standardized manner. Data provided by the T&E Team will be partitioned into development and evaluation datasets. Datasets will be provided at approximately six-month intervals to Performers; development datasets will be for use in algorithm R&D and training, and evaluation datasets for performance characterization. Evaluation of Performer progress towards program metrics will primarily utilize evaluation datasets that are not sequestered but shared with Performers so as to facilitate communication and internal Performer error analyses. Three research datasets and three evaluation datasets will be provided in Phase 1 in addition to Performer self-collected data. The baseline API provided in Phase 1 is to standardize system and subcomponent design and streamline comparative evaluations. More details on the datasets are available in SECTION 1.D. Program Data and more details on the API are available in SECTION 1.G.3.1. Program API.

1.A.2.2. Phase 2 Phase 2 will have a duration of 18 months. In Phase 2, the Performers will focus on R&D to explore how best to combine their modular research approaches with the objective of integrating and optimizing subcomponent modules into an end-to-end WB biometric matching system. By the midpoint of the phase, the Performers will be required to have demonstrated a prototype system that integrates person detection, face detection, image degradation mitigation technique(s), and the fusion of FR and at least one WB modality in a single, automated pipeline. During Phase 2, each Performer will carry out two (2) HSR biometric data field collections to capture diverse program-relevant data for R&D. These datasets shall be delivered with a minimum of Government Purpose Rights to allow use by USG and its contractors (including all other BRIAR R&D Performers).

More details on these researcher-executed collections are available in SECTION 1.D.1.2.

Researcher Collections.

The Performers will work towards meeting accuracy targets across a range of biometric use cases that demonstrate performance relevant to United States Government (USG) needs. While the full combination of capture conditions is to be considered during Phase 2, the focus will be on biometric matching involving probe videos containing unconstrained subjects with either moderate to severe pitch or range conditions. Severe pitch is defined as 30 – 50° view angle and severe range as 500 – 1000 meters. Moderate pitch or range will continue to be relevant in Phase 2. Probe videos containing small groups of up to three (3) subjects, in addition to single individuals, will be introduced in Phase 2 for detection, tracking, and encoding of biometric signatures. In addition, Performers will begin addressing biometric matching with partially occluded views of subjects for WB, gait, or face views. In Phase 2, Performer R&D and Government T&E of algorithms will involve biometric database galleries containing both controlled enrollment videos and partially controlled enrollment imagery of diverse subjects presenting WB, face, and gait information. For purposes of this BAA, partially controlled is defined as short range enrollment with limited instruction to subjects or non-ideal lighting. Examples include an outdoor enrollment or a non-cooperative person in an indoor hallway. Performers will also focus their efforts on identifying, characterizing, and addressing imaging data that are proving particularly challenging to their research approaches, subcomponent modules, or integrated systems, to include specific BRIAR environmental conditions, platform/sensor types or behaviors, or demographic or body-type populations. In Phase 2, BRIAR systems will be required to operate in a real-time configuration, processing both batch video files and streaming videos on commodity computing resources. For purposes of this BAA, this requirement means a single video stream at a minimum of 30 frames per second (fps). Two sets of interim Deliverables and a set of final Deliverables will be due in Phase 2 at scheduled Milestones (see SECTION 1.G.2. Program Milestone, Waypoint, and Deliverables Timeline).

In Phase 2, the BRIAR T&E Team will be providing additional datasets and an expanded version of the BRIAR API. Three research datasets and three evaluation datasets will be provided in Phase 2 in addition to Performer self-collected data. More details on the datasets are available in SECTION 1.D. Program Data and more details on the API are available in SECTION 1.G.3.1.

Program API.

1.A.2.1. Phase 3 Phase 3 will have a duration of 12 months. In Phase 3, the Performers will focus on advancing the coupled integration of all subcomponents and optimizing biometric signal fusion in the end-to-end system. R&D will expand image processing and fusion to include secondary components and improving the adaptive nature of the pipeline to the full range of capture and environmental conditions. Research will also pursue methods to retrain or adapt the matching algorithms to transfer to new sensor and/or platform conditions as well as methods to optimize performance if deployed on a specific dedicated sensor or platform system. New sensors and platforms will be identified early in Phase 3 by the BRIAR PM and T&E Team, in coordination with USG partners.

Performers will continue efforts to address imaging data that are proving particularly challenging to their BRIAR systems across diverse environmental, sensor, platform, and human subject types.

During Phase 3, each Performer will carry out one (1) HSR biometric data field collection to capture diverse program relevant data for R&D. These datasets shall be delivered with a minimum of Government Purpose Rights to allow use by USG and its contractors (including all other BRIAR

R&D Performers). More details on these researcher-executed collections are available in SECTION 1.D.1.2. Researcher Collections.

The Performers will work towards meeting performance targets across a range of biometric use cases that demonstrate reliable accuracy and low false match rates. The full combination of capture conditions will be considered during Phase 3 R&D. Biometric matching will involve probe videos containing unconstrained diverse subjects with any combination of moderate to severe pitch and/or range conditions. Research will continue advancing methods to mitigate or utilize partially occluded views of biometric signatures while minimizing the impact to matching accuracy. In Phase 3, Performer R&D and Government T&E of algorithms will include biometric database galleries containing fully unconstrained videos of non-cooperative diverse subjects captured at small to medium range and/or pitch views presenting WB, face, and gait information. Small range is defined as 1 – 100 meters; small pitch is defined as 0 – 20°. In Phase 3, Performers will develop a version of their full BRIAR system that will be capable of processing real-time streaming video at 15 fps on an edge computing platform with matching accuracies to Phase 2 metric targets. Note that this requirement does not apply to the main BRIAR system, which is to process real-time streaming or batch videos in commodity desktop computing environments. One set of interim Deliverables and a set of final Deliverables will be due in Phase 3 at scheduled Milestones (see SECTION 1.G.2. Program Milestone, Waypoint, and Deliverables Timeline).

In Phase 3, the BRIAR T&E Team will be providing additional datasets and a completed version of the BRIAR API. Two research datasets and two evaluation datasets will be provided in Phase 3 in addition to Performer self-collected data. More details on the datasets are available in SECTION 1.D. Program Data and more details on the API are available in SECTION 1.G.3.1.

Program API.

1.B. Team Expertise Collaborative efforts and teaming among Offerors are highly encouraged. It is anticipated that teams will be multidisciplinary and may include expertise in one or more of the disciplines listed below. This list is included only to provide guidance for the Offerors; satisfying all the areas of technical expertise below is not a requirement for selection and unconventional or innovative team expertise may be needed based on the proposed research.

• Biometrics theory, applications, and systems with specific emphasis on face, WB, and gait modalities

• Computer vision to include object detection, tracking, camera modeling

• Image processing to address noise, degradations, turbulence, and motion blur

• Optics and sensors

• UAV sensor design and operation, as well as UAV flight concepts of operations

• HSR and data collections

• Anatomy and kinesiology

• Modeling and simulation

• Machine learning, deep learning or hierarchical modeling

• Artificial intelligence

• Systems integration

• Systems engineering

• Software engineering

• Data reduction and analysis

• Distributed processing, stream processing

1.C. Program Scope and Limitations Proposals shall explicitly address all the following:

• Underlying theory: Proposed strategies to meet program-specified metrics must have firm theoretical bases that are described with enough detail that reviewers will be able to assess the viability of the approaches. Proposals shall properly reference previous work upon which their approach is founded.

• Research & Development approach: Proposals shall describe the technical approach to meeting program metrics.

• Technical risks: Proposals shall identify technical risks and proposed mitigation strategies for each.

• Software development: Proposals shall describe the approach to software architecture and integration.

The following areas of research are out of scope for the BRIAR program:

• Research that does not have strong theoretical and experimental foundations.

• Development of optical sensor hardware.

• Development of sensor platforms, such as UAVs, vehicles, aerostats, towers, or camera systems.

• Development of other hardware.

• Approaches that rely on secondary external data signals, such as cell tower tracking or non-optical sensors.

• Research that utilizes proprietary data that are not made available to other Performers.

• Methods that require a human-in-the-loop as part of the integrated end-to-end system in

Phases 2-3.

• Approaches that consist merely of integrating currently existing software.

• Research involving non-visible band imagery (e.g., Near Infrared, Short-Wave Infrared, Mid-Wave Infrared, or Long-Wave Infrared)

Delivered software will be evaluated by an independent team on sequestered and shared evaluation datasets. During each performance period, Performers will build prototype algorithms, and subcomponent modules, and/or systems for end-to-end WB and FR under challenging imaging conditions that will be run and evaluated by the T&E Team. Testing protocols do not allow for expert operators, human-in-the-loop operation, or any operations not deemed “turnkey”. However, systems or algorithms that have been trained using human-in-the-loop methods may be submitted, provided they run autonomously.

1.D. Program Data BRIAR is a unique research program in the field of computer vision and biometrics in that data that represent the domain of interest are not commonly available in the academic community due to lack of commercial interest, complexity of collection logistics, and high costs to collect.

However, for BRIAR to facilitate innovative R&D and achieve program metrics, diverse program data in sufficient quantities are needed for development and statistically reliable evaluations. As a result, the program will include robust and explicit data collection by the BRIAR T&E Team and R&D Performers.

The program will include two types of data – Development Data and Evaluation Data. The data will consist of WB, gait, and face imagery collected with a range of conditions, sensors, and platforms involving a diverse group of demographic and body types. Development data will be utilized by Performers, as they see fit, to conduct research, development, error analysis, and algorithm training. Evaluation data will be explicitly excluded from any algorithm training approaches and will be used by both Performers and the T&E Team for performance evaluations of subcomponents, modules, and systems measured against the BRIAR program metrics. A summary of HSR collected datasets across the entire life of the program is included in Table 2.

Table 2: BRIAR Program Datasets

Phase Type Origin Dataset3 Month # Subjects (Probes)4,5

# Subjects (Gallery)5

1 Development T&E BRS1 1 200 200 1 Evaluation T&E BTS1 1 100 2506 1 Development T&E BRS2 7 200 200 1 Evaluation T&E BTS2 7 200 5006 1 Development R&D BRC17 13 8009 8009 1 Development T&E BRS3 13 200 200 1 Evaluation T&E BTS3 13 300 7506 2 Development T&E BRS4 19 200 200 2 Evaluation T&E BTS4 19 400 1,0006 2 Development R&D BRC27 25 6009 6009 2 Development T&E BRS5 25 200 200 2 Evaluation T&E BTS5 25 500 1,2506 2 Development R&D BRC37 31 6009 6009 2 Development T&E BRS6 31 200 200 2 Evaluation T&E BTS6 31 600 1,5006

3 BRS = BRIAR Research Set, BTS = BRIAR Test Set, BRC = BRIAR Researcher Collection 4 Minimum number of unique individuals in the probe set with mated entries in the dataset gallery.

5 Each BTS evaluation test set will be a superset of previously released BTS.

6 Gallery includes additional distractor subjects for improved statistical analysis.

7 Each Performer will conduct a field data collection and make the data available to T&E and other R&D Performers.

Phase Type Origin Dataset3 Month # Subjects (Probes)4,5

# Subjects (Gallery)5

3 Development T&E BRS7 37 200 200 3 Evaluation T&E BTS7 37 700 1,7506 3 Development R&D BRC47 43 4009 4009 3 Development T&E BRS8 43 200 200 3 Evaluation T&E BTS8 43 800 2,0006

TOTAL Development T&E + R&D

-- -- 4,0008 4,0008

TOTAL Evaluation T&E -- -- 800 2,000

1.D.1. Development Data Development data will be utilized by Performers, as they see fit, to conduct internal research, development, error analysis, and algorithm training.

It is anticipated that Performers will need development data specific to each research approach and each subcomponent module of their BRIAR system(s) to achieve program goal, objectives, and metrics. However, because Performer approaches will differ, it is anticipated that Performers may need different development data depending on their individual approaches.

Proposals must specify the development data needed to carry out the proposed research and what data characteristics are necessary for their approach(es) to be successful at meeting program objectives. Offerors’ unique data needs in a particular technical area should not be interpreted as requirements input into the program’s T&E data collections, but part of a comprehensive research approach that communicates a possible path towards meeting BRIAR program objectives.

There will be four types of development data used in the BRIAR program:

• Government Research Sets – Datasets collected, annotated, and curated by the T&E Team and provided to Performers for R&D.

• Researcher Collections – Datasets collected, annotated, and curated by R&D Performers and delivered with Government Purpose Rights to share with other BRIAR Performers.

• External Data Sources – Data obtained by Performers that are available from third parties or that have been collected by a Performer outside of BRIAR with rights to allow all Performers to use them for BRIAR.

• Government Augmented Data – Data resulting from the T&E Team applying modeling and simulation tools to transform Performer-identified or provided datasets.

1.D.1.1. Government Research Sets The Government will be providing BRIAR Research Sets (BRS) to Performers throughout the program. See Table 2 for a full summary of all BRS datasets. These datasets will include visible-band WB biometric video imagery with ground truth computer vision annotations, identity labels, and metadata information. The datasets will include field collected imagery, enrollment imagery, 8 Calculation assumes four Performers in Phase 1, three Performers in Phase 2, and two Performers in Phase 3.

Numbers may increase if Performer approaches are successful to the extent de-scoping does not occur.

and may include imagery augmented using Government Owned Technology Solution (GOTS) modeling and simulation tools.

The T&E Team will be carrying out HSR biometric field data collections approximately every six months throughout the life of the program. Collections will occur in a variety of outdoor terrain and temperature environments to capture wide variations in atmospheric turbulence. Severe weather (e.g., snow, heavy rain, fog, or heavy winds) will be out of scope for these field collections.

These data will be collected using many different optical sensors and platforms at ranges out to 1000 meters and sensor view angles as steep as 50°. The specifics of these field collections will be determined by the T&E Team during the program, but for purposes of this BAA, Offerors can assume the following characteristics within each of the BRS datasets:

• Full motion visible-band video imagery of varying durations (i.e., seconds up to minutes)

• A minimum of 200 consenting volunteer subjects with a diverse range of demographic and body type characteristics

• A minimum of two different clothing configurations for each subject

• Video containing scenes of single individuals up to small groups of three people

• Ground-truth enrollment for gait, WB, anthropometry, and face biometrics o Enrollments are to follow NIST/ANSI standards or best practices for biometric enrollments o Enrollment conditions will depend on the Phase of the program (see SECTION 1.A.2. Program Phases)

• Imagery captured from the following types of platforms:

o Minimum of one (1) type of Class 2 fixed-wing UAV o Minimum of one (1) type of Class 2 quadcopter UAV o Elevated surveillance cameras mounted on poles or buildings o Man-portable telephoto camera systems o Other platforms determined by the Government

• Imagery captured from multiple optical sensors with varying resolution, optics, stabilization o Visible wavelength imagery only o Threshold # of different sensors = 10 o Objective # of different sensors = 20

BRS datasets will be provided to Performers with ground-truth annotations and associated metadata information to include the following:

• WB bounding boxes,

• Face bounding boxes,

• Tracklet associations within full motion video,

• Identity and sighting instance labels,

• Capture range,

• Capture pitch,

• Subject pose,

• Human presence (in scene),

• Subject demographic information,

• Environmental conditions,

• Capture sensor or platform characteristics, and

• Atmospheric turbulence conditions.

In addition to real imagery captured during field collections, the T&E Team may utilize GOTS modeling and simulation tools to produce augmented data. It is anticipated that these tools will allow well characterized captured data to be transformed into simulated data with expanded or different characteristics (e.g., range, atmospheric turbulence, sensor resolution). This technique will be used by the T&E Team to address gaps in captured data that are relevant to BRIAR program objectives. Priority will be placed on capturing real data across the full range of conditions with simulated data serving as a mitigation measure.

1.D.1.2. Researcher Collections Each Performer will be required to plan, execute, and annotate several BRIAR Researcher Collections (BRCs) to supplement their own development data and to share among the other BRIAR Performers and the T&E Team. Phase 1 will have one BRC; Phase 2 will have two BRCs;

and Phase 3 will have one BRC per Performer. Each will be similar in concept, but the specifics of the capture conditions, sensors, and platforms will be determined during the BRC planning stage by the Government with input from the Performers. An Offeror may propose additional BRCs beyond the required number in their proposal. The proposal shall describe how each BRC will be used in the R&D and justify the need for any data collections beyond the required BRCs.

Annotated BRC datasets will be provided to the Government as Deliverables on a set schedule, as described in SECTION 1.G.2. Program Milestone, Waypoint, and Deliverables Timeline.

Each Performer’s BRC will be an IRB-approved HSR biometric field data collection involving the capture of WB and face video imagery across different visible-band sensors and platforms. The specifics of each BRC are intended to support that Performer’s research approach(es) and will have flexibility to accommodate their needs in pursuit of BRIAR program objectives. In general, each BRC will be similar to Government BRS datasets with differences indicated in bold, but will be required to include all of the elements described below:

• Full motion visible-band video imagery of varying durations (i.e., seconds up to minutes)

• A minimum of 200 consenting volunteer subjects with a diverse range of demographic and body type characteristics

• Video containing scenes of single individuals up to small groups of three people

• A minimum of two different clothing configurations for each subject

• Ground-truth enrollment for gait, WB, anthropometry, and face biometrics o Enrollments are to follow NIST/ANSI standards or best practices for biometric enrollments o Enrollment conditions will depend on the Phase of the program (see SECTION 1.A.2. Program Phases)

• Imagery captured from the following types of platforms:

o R&D fixed-wing UAV o R&D quadcopter UAV o Elevated surveillance camera mounted on poles or buildings o Man-portable telephoto camera system

• Imagery captured from multiple optical sensors with varying resolution, optics, stabilization o Visible wavelength imagery only o Threshold # of different sensors = 4 o Objective # of different sensors = 8

For the purposes of BRCs, Performers will be provided a list of UAV platforms that are known to meet the requirements of the BRIAR program. The list may include either complete UAV platforms and/or a list of subcomponents with instructions on assembling and operating the UAV.

The naming of any UAV or component on this list is not a product endorsement. Offerors may propose other platforms but, prior to use in any BRC, must obtain verification from IARPA that the proposed platform will meet BRIAR program requirements. The UAVs to be used in the BRIAR program will be suitable for operation at both a DoD training facility (pending approvals) or a civilian facility (e.g., university campus) and allowable for purchase and operation by third party organizations with foreign national staff (i.e., non-US citizens). Included with each UAV on the IARPA-provided list will be a selection of 1 – 3 sensor payloads for Performers to utilize in field BRC events for capturing imagery aligned with BRIAR program objectives. The BRIAR PM may opt to revise the recommended BRIAR UAV platforms and sensors at the start of each program phase or as needed during the program.

Performers will be responsible for annotating all BRC data for their R&D needs. All annotation data are to be delivered to the Government regardless of whether they are developed after the collection or after initial delivery of the BRC dataset. At a minimum, the following pieces of information are to be annotated:

• WB bounding boxes,

• Face bounding boxes,

• Identity and sighting instance labels,

• Capture range,

• Capture pitch,

• Human presence (in scene),

• Subject demographic information,

• Capture sensor or platform characteristics.

All data collected by Performers under the BRIAR program for R&D or any other purpose will be provided to the Government as Deliverables with usage rights that allow the data to be used by the USG and its contractors (including all branches, departments, agencies, and offices and research and applications communities, private industry, academia, and non-profit organizations) (i.e., Government Purpose Rights).

1.D.1.3. External Data Sources Performers may utilize external datasets in addition to BRS and BRC datasets. External data are data obtained by Performers that are available from third parties or that have been collected by the Performer outside of the BRIAR program. An example of a third-party dataset would be a biometric dataset, collected by a university under an IRB-approved HSR protocol, that has been approved by the dataset owner for release to the research community. Data collected by a

Performer under a different program are considered external data, even if the other program’s data collection was Government-sponsored.

All external datasets must be approved for use in the BRIAR program by IARPA, in accordance with HSR and applicable privacy policies, statutes, and regulations. IRB approval or a ruling of IRB exemption may be required before IARPA approval of the use of data in BRIAR.

Performers may not use proprietary datasets unless these datasets are made available to all R&D Performers and Government T&E in the BRIAR program without restriction. Public release of proprietary datasets is not a requirement; however, limited release restricted for use within the BRIAR program is required. Moreover, for any dataset not collected under the scope of the BRIAR program, Performers must provide the Government with an accounting of all resources used and sources from which data are drawn and describe how the data will be used for development, testing, and training of algorithms.

All external datasets that are part of an Offeror’s proposed research approach must be summarized in the proposal with the following minimum information:

• Dataset Name

• Short Description

• Data Owner

• License or use rights

• Method or URL link to obtain data

1.D.1.4. Government Augmented Data In addition to using GOTS modeling and simulation tools to augment BRS datasets, the T&E Team will be making available their services to Performers to apply the simulation tools to other datasets relevant to Performer algorithm training. Using known or estimated capture conditions and sensor characteristics, simulation tools can transform a given image to one seemingly collected under different (simulated) conditions. For example, an image captured at 100 meters can be modified to simulate the same scene captured at 300 meters or can be modified to simulate moderate turbulence effects in a stable image. Using these supplemental research support activities, more traditional computer vision or biometric training datasets can better approximate the desired imaging conditions within the BRIAR program.

Determination of the details of the data augmentation task and the targeted range of simulated conditions will be by collaboration among IARPA, BRIAR T&E, and the Performer. At fixed waypoints during each Phase, Performers will be invited to submit a Data Augmentation Request (DAR) to BRIAR T&E that details the source dataset(s), use rights within the BRIAR program, and the desired target dataset output characteristics. Validation that the original data are approved for use within BRIAR will be a prerequisite before any augmentation services are conducted by BRIAR T&E. Each Performer will be allowed one DAR per phase, not to exceed 10 million frames or images per pre-augmented dataset. Additional dataset augmentations may be possible but will be dependent on BRIAR T&E resources and staff availability. Any such additional dataset augmentations will be conducted only with the BRIAR PM’s approval. All augmented data sets will be made available to all Performers regardless of which Performer initiated the request.

Proposals shall indicate whether the Offeror intends to leverage this service as part of the proposed research, which datasets they have initially identified for augmentation, what their DAR would cover, and how the augmented data will directly address BRIAR objectives. Modeling and simulation by the T&E Team is not intended to replace research activities carried out by the Performers, but instead to supplement them. Offerors are welcome to propose their own modeling and/or simulation research as part of their overall BRIAR approach.

1.D.2. Evaluation Data BRIAR will utilize distinct test data to evaluate the performance of Performer subcomponents, modules, and systems against program goal, objectives, and metrics. Each BRIAR Test Set (BTS) will consist of diverse multi-modal biometric data, robust metadata annotations, and prescribed evaluation test protocols. The composition and characteristics of imagery within a BTS will be similar to BRC data but constructed to evaluate program metrics in a balanced and intentional manner with subject identities different from any represented in other BRS data. Using different identities for algorithm training from those used in testing will ensure that research approaches are generalizable.

Intended uses of the BTS evaluation datasets include both use by the T&E Team for independent evaluation of program Deliverables against target metrics, and use by Performers to refine and improve their algorithms. The evaluation datasets will be provided to Performers to enable internal T&E and exploratory error analysis by Performers and to improve the consistency and communication between Performers and T&E. The evaluation datasets will be separate and distinct from the development datasets with respect to subject identities. No evaluation data or evaluation identities will be permitted in any aspect of algorithm training or functionality.

Additional sequestered or external datasets may be used to supplement performance evaluations at the discretion of the BRIAR PM.

BTS datasets will derive most of their imagery from data collections carried out by the T&E Team with data augmentation using GOTS simulation tools, as needed. Some BTS data will originate from the same data collections used to construct BRS development data but with different subject identities (see SECTION 1.D.1.1. Government Research Sets for BRS details). Additional collections will be carried out to supplement BTS evaluation gallery databases with distractor subject imagery. A summary of BTS dataset characteristics across BRIAR phases is shown in Table 3.

Table 3: BRIAR Evaluation Data

Phase9 Probes Gallery Subjects Range Pitch Noise Subjects Enrollment

1 (BTS3) 300 10 – 500 m OR10 0 – 30° Low – High 750 Constrained 2 (BTS6) 600 10 – 1000 m OR10 0 – 50° Low – High 1,500 Semi-constrained 3 (BTS8) 800 10 – 1000 m AND11 0 – 50° Low – High 2,000 Unconstrained

9 Interim BTS datasets during each phase are not listed, only end-of-phase evaluations sets are shown on this table.

10 Individual videos will possess either moderate to severe range OR pitch as the main challenging condition.

11 The dataset will include videos that possess both challenging range AND pitch conditions.

Evaluation Data Characteristics:

• Biometric Imagery: Visible-band videos of subjects presenting WB, face, and gait

• Subjects: The number of unique subject identities

• Range: The direct linear distance from the optical sensor to subject

• Pitch: The angle between the sensor view position and the subject on the ground plane

• Noise: The qualitative degree of noise, degradation and/or atmospheric turbulence

• Enrollment: The conditions under which the gallery imagery is captured.

1.D.3. Privacy Plan As part of their proposal, Offeror’s shall prepare a BRIAR Privacy Plan v1.0 that comprehensively describes the efforts the Offeror will take to protect personally identifiable information, and safeguard the security of any personal data collected and that of any devices, applications, networks, or services involved in collection, transmission, processing, and storage of such data.

Any claims that data are anonymous must be based on evidence and supported with sufficient information regarding how the data have been anonymized. Each Performer’s BRIAR Privacy Plan is to cover all data categories described in this BAA; relevant information on GFI datasets will be provided by BRIAR T&E in advance of data release.

The initial version of the BRIAR Privacy Plan shall be included in the Offeror’s proposal as a standalone section or appendix that covers all external datasets to be leveraged as part of the proposed research approaches. The BRIAR Privacy Plan shall be updated at the beginning of each Phase and whenever new sources of data or datasets are proposed for use within a Performer’s BRIAR research activities, to include data used for either development or evaluation purposes.

Details on BRS-1 and BRT-1 (see SECTION 1.D.1.1. Government Research Sets) will be provided by IARPA at the Phase 1 Kick-off Meeting for inclusion in the BRIAR Privacy Plan v1.1 Deliverable (see Table 6).

1.E. Test and Evaluation (T&E) T&E will be conducted by an independent team of Government and contractor staff carrying out evaluation and analyses of Performer research Deliverables using program test datasets and protocols. In addition to independent T&E, the program will regularly gauge interim progress of Performer research activities towards BRIAR objectives and target metrics using T&E results measured and reported by the Performer teams themselves. The BTS data and test protocols (see SECTION 1.D.2. Evaluation Data) will be the primary mechanism by which the T&E Team carries out their evaluations.

The BRIAR Program will pursue rigorous and comprehensive T&E to ensure that research outcomes are well characterized, deliverables are aligned with program objectives, and that algorithm performance is measured across the full range of demographic, sensor, and environmental conditions. Such T&E activities will not only inform IARPA and Government stakeholders on BRIAR research progress but will also serve as invaluable feedback to the Performers to improve their research approaches, algorithm training practices, and system development. The BRIAR Program will work closely with Government leaders in biometrics (e.g., National Institute of Standards and Technology) to continually refine and improve T&E methodologies.

The Government will provide Performers with a uniform test Evaluation Harness (EH) Software Development Kit (SDK) with relevant scripts to run program test protocols on program datasets.

The design of the EH will be informed by and consistent with NIST evaluation environments used in standardized testing, such as the Face Recognition Vendor Test. The evaluation environment will be the platform for independent government testing of Performer Deliverables.

Performers will have specific Deliverable Milestones at which all subcomponent and system algorithms and software will be delivered to IARPA and its designated T&E Team. The T&E Team will then conduct evaluations at the direction of the BRIAR PM and with the objective of characterizing the quality, functionality, and performance of the BRIAR Deliverables. In addition to quantitative measurements, T&E will be carried out to establish a thorough understanding of the progress, status, and limitations of the Performer’s research.

T&E results and feedback will be provided to Performers at regular intervals to keep them abreast of current independent performance measurements and to inform and improve their R&D approaches and methods. T&E results from all Performers will be shared with all teams to establish an understanding of the current state and progress of BRIAR research; T&E results will also be shared with USG external stakeholders, including their contractors for Government Purposes. For example, a PI Review Meeting will be held annually to share research ideas, progress, and results across the BRIAR program (see SECTION 1.I.1. Workshops).

IARPA may conduct other supplemental evaluations or measurements in its sole discretion to evaluate the Performers’ research and Deliverables.

1.F. Program Metrics Achievement of metrics is often a criterion for exercise of options to continue performance under IARPA research contracts. IARPA has defined BRIAR program metrics to evaluate effectiveness of the proposed solutions in achieving the stated program goal and objectives, and to determine whether satisfactory progress is being made to warrant continued funding of the Performers.