VideoLINCS_BAA_TechSection_Draft_2024_01_25_C.pdf

Version Date: January 19, 2024

IARPA

DRAFT

BROAD AGENCY ANNOUNCEMENT

FUNDING OPPORTUNITY DESCRIPTION

IARPA-BAA-24-04

Video LINCS

Video Linking and Intelligence from Non-

Collaborative Sensors

IARPA-BAA-24-04

Release Date: TBD

INSTRUCTIONS

This Draft Broad Agency Announcement (BAA) is NOT a formal request for proposals.

Proposals are NOT being requested or evaluated at this time. A Final BAA will be released at a later date for which interested parties may submit proposals.

The Intelligence Advance Research Projects Activity (IARPA) is seeking interested parties to thoroughly review the Draft BAA Funding Opportunity Description and provide comments, questions, suggested changes, and feedback.

IARPA does not anticipate posting responses to any comments, questions, suggested changes, and/or feedback received; however, all input will be considered in developing the Final BAA.

Additionally, the Draft BAA Funding Opportunity Description is subject to change in the development of the Final BAA as a result of IARPA’s consideration of the input received from interested parties in response to the Draft BAA.

Please submit all comments, questions, suggested changes, and feedback to the designated procurement email address: dni-iarpa-baa-24-04@iarpa.gov.

Interested parties will be given the opportunity to submit questions and comments in writing for

IARPA response during Proposers’ Day and once the Final BAA is issued.

OVERVIEW

SECTION 1: FUNDING OPPORTUNITY DESCRIPTION

The Intelligence Advanced Research Projects Activity (IARPA) often selects its research efforts through the Broad Agency Announcement (BAA) process. The use of a BAA solicitation allows a wide range of innovative ideas and concepts. The BAA shall appear first under Contract

Opportunities on https://sam.gov, then the IARPA website at http://www.iarpa.gov/. The following information is for those wishing to respond to this Program BAA.

This BAA (IARPA-BAA-24-04) is for the Video Linking and Intelligence from Non-Collaborative

Sensors (Video LINCS) Program. IARPA is seeking innovative solutions for the Video LINCS

Program in this BAA, which is envisioned to be a 48-month effort across all focus areas, beginning approximately August 2024.

1.A Program Overview

In today’s world, video is a ubiquitous sensor. Voluminous video data is collected (e.g., security, closed circuit television (CCTV), aerial, etc.), but the manpower and available attention to provide desired security is limited. Operators are overloaded with many video feeds with limited attention for the available data volumes or ability to discern threats. Tragic incidents trigger forensic analyses that require significant manpower to pore through troves of imagery.

Diverse video sensors, even when installed at a single facility, often do not collaborate with each other. Some automated analytics have been developed, ranging from motion detection to object classification to summarization, providing valuable tools for filtering video content. These capabilities assist operators in focusing their attention. Ultimately, the burden on operators is high, requiring significant expertise, recall and resources to associate content across footage and discern threats.

The goal of the Video LINCS program, is to develop re-identification (reID) algorithms to autonomously associate objects across diverse, non-collaborative, video sensor footage and map re-identified objects to a unified coordinate system (geo-localization). The reID and geo-localization algorithms will distill raw pixel data into spatio-temporal motion vectors, providing the ability to analyze these patterns for anomalies and threats. While the ultimate goal will be to re-identify general objects, the program will start with person reID, progress to vehicle reID, and culminate with reID of generic objects across a video collection.

The overall program design is depicted in Figure 1, where a diverse collection of video imagery from different sensors is input into the Video LINCS system that reidentifies objects across the collection and geo-localizes those objects, providing tracks for each object in both the initial camera coordinate and common geo-coordinate reference frames. Offerors to the Video LINCS program shall propose solutions for both the re-identification and geo-localization objectives.

Video Imagery

Video LINCS

Display all object locations on a single map

Object Locations (World Coordinates)

Person 1

Vehicle 1

Object Tracks (Camera Coordinates)

Camera 1 Camera 2

Person 1

Vehicle 1

Vehicle 1

Person 1

Associate same object across all camera feeds

Camera 1 Camera 2

Object 1

Object 1

Object 1

TA-1

ReID

TA-2

Object

Geo-localization

Figure 1: Video LINCS Program Design

It is anticipated that annotated videos of people, vehicles and generic objects over a range of sensor types and collection conditions will be central to both Research and Development (R&D) and Test and Evaluation (T&E) objectives. Offerors who propose use of such R&D data will be required to collect (and/or simulate) and annotate R&D datasets throughout the program lifecycle and share them across the program with other performers. The Government will collect and annotate data primarily for T&E objectives and may potentially provide some data for R&D purposes. All data collections with potential Human Subject Research (HSR) will require Institutional Review Board

(IRB) approval, and all performers will be required to deliver a privacy plan that comprehensively describes how any potential personally identifiable information (PII) will be safeguarded.

The Video LINCS program will pursue rigorous and comprehensive independent T&E to ensure that research outcomes are well characterized, deliverables are aligned with program objectives, and performance is measured across a wide range of conditions. T&E activities will not only inform Government stakeholders on Video LINCS research progress but will also serve as valuable feedback to the performers to improve their research approaches, algorithm training practices, and system development. The Video LINCS program will continually refine and improve T&E methodologies throughout the lifecycle of the program.

1.A.1 Technical Challenges and Objectives

The Video LINCS program consists of two Technical Areas (TAs):

• TA-1 Re-identification (ReID): Autonomously and automatically associating the same object (person, vehicle, or generic object) across a video corpus.

• TA-2 Object Geo-localization: Geo-locating objects to provide positions for all objects in a common world reference frame.

The objective is for Video LINCS to be effective in the open-world setting where there is no advance knowledge of the sensor types, sensor collection geometries, scenes, objects to be re-identified, etc. There will be no a priori set of people, vehicles or objects to be queried or gallery/library to serve as a reference for matching. The goal will be to re-identify the content of the video set self-consistently; to determine presence and location of each object throughout the video collection, without any auxiliary or prior knowledge of that object. This presents multiple challenges for both reID and object geo-localization.

1.A.1.a TA-1 – ReID Challenges

The Video LINCS reID objective is to reidentify objects across an arbitrary video collection without any prior access to representative video or the objects to be reidentified. This presents a number of challenges.

Video in a collection may come from many different sources and collection geometries, containing significantly different characteristics that are difficult to associate. The variations in appearance within and across video sensor footage, which may range across many collection types such as stationary and mobile; ground-based, building/roof/pole mounted and aerial; outdoor and indoor;

day and night; varying weather conditions; temporally proximate and distant;

noise/artifacts/corruption, etc., present significant challenges to reidentifying objects across such collections.

The reID objective varies significantly from identification (ID) where specific identities are known a priori and queries are matched to a gallery/library of candidates. For reID on an arbitrary video collection, there are no a priori galleries. The lack of an a priori gallery of objects to be re-identified will require systems to autonomously determine when to expand system generated galleries to include additional objects vs. expanding matches to existing objects. For people, recognizable faces will predominantly be unavailable throughout the video collection. For vehicles, license plates will not be available for association.

While general categories of video will be known (e.g. aerial drone, building mounted security and

CCTV, etc.) no representative data will be provided for system development and training.

Furthermore, Video LINCS will focus on correct association/reID across all appearances of the same object in collection – not merely retrieving a single match such as that measured by the

Cumulative Match Characteristic (CMC) [1].

Video LINCS is seeking an end-to-end reID solution, where video is input into the system and reidentified object locations are output. The system needs to automatically locate the objects and associate them - across scale, aspect, density, crowding, obscuration, etc. - without introducing false detections and false matches.

Video LINCS will take a staged approach, beginning with cases where the class of the object to be re-identified is known, such as people and vehicles, and ultimately extend to all objects in a video collection when there is no a priori knowledge of the object class.

1.A.1.b TA-2 – Object Geo-localization Challenges

The ultimate objective is to provide spatio-temporal tracks of re-identified objects, which requires mapping from camera coordinates to a common reference frame – nominally, a geo reference frame. This mapping will be performed under TA2 – Object Geo-localization.

Object geo-localization presents a number of challenges. Homographies are often used to transform from image-plane to ground-plane coordinates, but the homographies are often computed manually, and are based on the assumption/approximation of a planar scene. Geometric inverse mapping using camera pose and auxiliary information, but camera pose is not always available and scale must be resolved. Even when camera pose is available, it may not be sufficiently accurate to facilitate inverse mapping.

Video LINCS TA-2 objectives will include autonomous coordinate remapping for a wide variety of video collections and conditions, to facilitate autonomous operation under broad conditions where different levels of metadata and reference imagery are available.

1.A.2 Program Phases

The Video LINCS program is planned as a 48-month Research and Development (R&D) effort and is divided into three phases. In an 18-month Phase 1, Video LINCS teams will demonstrate the feasibility of reID for people in a video corpus and object geo-localization to provide all subject motion in a common reference frame. During this phase, each person’s clothing will remain the same (short-term / temporally proximate reID) and provided metadata (e.g. time-stamps and camera pose, to the extent available) will be noise free. During Phase 2, reID will expand to include people with clothing changes (long-term / temporally distant reID), include vehicles, require functionality on generic objects, additional sensor types and collection geometries will be included in the evaluations, and noise will be introduced into provided metadata. During Phase 3, evaluation will have a stronger focus on reID of generic objects, temporally distant reID of vehicles will be performed, additional sensor types and collection geometries will be included, and there will be greater uncertainty in camera pose. The progression of objectives for both TAs across the program phases is shown in Figure 1.

18 Months 18 Months 12 Months

Phase 1

Feasible

18 Months

TA1 - ReID

• People

• Temporally proximate (people - same clothing)

• Within collection type

• Accurate time-stamps

TA2 - Geo-localization

• Camera pose provided (location & orientation)

Phase 2

Robust

18 Months

TA1 - ReID

• People, vehicles, generic objects

• Temporally distanced (people - attribute changes)

• Across collection types

• Time-stamp errors

TA2 - Geo-localization

• Camera location provided (no orientation)

Phase 3

Generalize

12 Months

TA1 - ReID

• Generic objects

• Temporally distanced (vehicles)

• Across broader collection types

TA2 - Geo-localization

• Camera general vicinity provided

Figure 1: Phase Objectives

Offerors are required to propose to all phases (Phases 1, 2, & 3) and all Technical Areas (TA-1 &

TA-2) of the Program under this BAA. Proposals that omit any of phases or Technical Areas will be considered non-compliant.

Test set conditions will expand throughout the program phases to increase the number of people/vehicles to be re-identified, increase the number of distinct identities within the datasets, increase the number of sensor collection classes/types, and lower the level of auxiliary metadata available. A listing of nominal test-set conditions throughout the phases is provided in Table 1.

The exact conditions are subject to potential update during program execution.

Table 1: Nominal test set conditions throughout program phases.

Phase 3Phase 2Phase 1

2,5001,500750# Distinct truthed peoplePeople

≥ 50% all phases% Small (< 150 pixels high)

< 30% all phases% Medium (150 < pixels high < 250)

< 20% all phases% Large (250 < pixels high)

10,0005,0001,000# Additional confusers

≥ 5≥ 5N/A# Attribute changes per person

653# Sensor collection types

2,0001,000N/A# Distinct truthed vehiclesVehicles

Approximate location within 100 m

Location, no orientation

Full pose provided

Collection sensor location and orientation

Video sensor pose

Leaderboards are planned to provide ongoing measurement of performance with rankings for all performers. A self-test dataset will be provided for performers to run algorithms on their own hardware with results submitted to a T&E hosted server for scoring and leaderboard posting. A separate sequestered data leaderboard will be used for independent evaluation where performer algorithms will be run by the T&E team on a sequestered dataset hosted on T&E hardware.

Monthly leaderboard submissions (initially to the self-test leaderboard, and subsequently to the sequestered data leaderboard) will be required to provide ongoing insight into performer progress.

Semi-annual waypoints will be incorporated where performance will be reported to IARPA management and software will be delivered to T&E to facilitate independent builds and runs from source code. Performer data collections and annotations must be shared with other performers after each semi-annual performance reporting waypoint. Key program activities and milestones are shown in Figure 2.

3 6 9 12 15 18 21 24 27 30 33 36 39 42 45 48

Data Collection & Annotation

TA1: ReID

TA2: Geolocalization

Leaderboards

Self-Test

Sequestered

Evaluation Reporting Waypoints

Cross Program Data Sharing

Kickoff Meetings and PI Workshops

Site Visits

Software Deliverables

Phase 1 - 18 Months Phase 2 - 18 Months Phase 3 - 12 Months

TASK

Figure 2: Video LINCS performer key activities and milestones throughout the program

Performer teams will be responsible for obtaining (collecting, curating, simulating) and annotating data in support of proposed R&D objectives. The Government will collect, curate and annotate data for evaluation. IARPA will provide access to self-test datasets for performers to run algorithms on their own hardware, and performers will submit containerized (nominally Docker) software using/conforming to the framework and APIs specified by the T&E team for independent runs on sequestered evaluation datasets.

1.A.1.a Phase 1

During an 18-month Phase 1, Video LINCS performers will research and develop the capability and demonstrate the feasibility for reID of people in a video corpus and geo-localization of those people to provide all subject motion in a common reference frame. Monthly submissions will be required to the program leaderboard throughout the program (initially to the self-test leaderboard and subsequently to the sequestered data leaderboard, as shown in Figure 2), with runtimes that are no longer than five times the test video length (and quicker in subsequent phases as shown in

Table 2) when run on the T&E hardware for sequestered dataset runs.

TA-1 – ReID

During this phase, each person’s clothing will remain the same (short-term / temporally-proximate reID) and provided metadata (e.g. time-stamps and camera parameters, to the extent available) will be noise free. ReID will be performed across videos within the same collection type (e.g. drone, CCTV, etc.). Multiple collection types will be included, but reID will only be performed within each collection type and not across the different types during this phase. Accurate time stamps will be provided for all video data. Performer systems will be required to re-identify people across all videos in the test set from the same collection type.

The Phase 1 test set will have approximately 750 distinct people which will be measured for reID performance (see Table 2 for metrics), and there will be an additional 1000 people present in the imagery to serve as potential confusers. People to be reidentified will largely have common external attributes (e.g. clothing) throughout the collection, but some attributes that could change over short periods (e.g. donning or removal of glasses or a cap, lifting or setting down a purse or bag, etc.) may be present.

TA-2 – Object Geo-localization

All people in the test set will need to be mapped to a common geo-coordinate frame and accuracy of geo-localizations will be measured. Full sensor pose will be available to performer systems during test runs. Geo-localization will be tested both as an independent algorithm as well as for the end-to-end reID system. For independent algorithm testing, camera coordinates of subjects in the test video will be provided at test time and performer systems will remap to geo-coordinates which will be measured for accuracy. For end-to-end reID, geolocation accuracy of subjects identified by the system will be measured.

1.A.1.b Phase 2

During an 18-month Phase 2, Video LINCS teams will demonstrate the feasibility of reID for vehicles, extend and develop new reID capabilities for people, and initiate reID for generic objects.

For people, reID will extend to temporally distant intervals where temporary attributes (e.g clothing) may vary and metadata (e.g. time-stamps and camera parameters) may have errors and might not be available. ReID will be performed across videos from different collection types (e.g.

drone, CCTV, etc.), and the number of collection types will expand beyond those used in Phase 1.

The number of distinct people to be re-identified in the test collection will increase to 1500, and there will be 5000 additional people present in the imagery to serve as potential confusers.

For vehicles, the reID conditions will be similar to those addressed for people in phase 1. During this phase, vehicle reID will be performed over short-term / temporally-proximate intervals and provided metadata (e.g. time-stamps and camera parameters, to the extent available) will be noise free. ReID will be performed across videos within the same collection type (e.g. drone, CCTV, etc.). Multiple collection types will be included, but reID will only be performed within each collection type and not across the different types during this phase. Accurate time stamps will be provided for all video. Performer systems will be required to re-identify vehicles across all videos in the test set from the same collection type. The Phase 2 vehicle test set will have approximately

1000 distinct vehicles which will be measured for reID performance, and there will be additional vehicles present in the imagery to serve as potential confusers.

For generic objects, systems will need to autonomously detect and reID objects throughout the collection, without any prior information regarding the types or characteristics of objects to be re-identified. To demonstrate the concept, consider video at a port of entry that contains ships. The system should autonomously reID ships in such a collection, without any a priori knowledge that ships are present. Similarly, if the system is provided videos of marbles, the system should autonomously reID the marbles throughout the collection. During Phase 2, performers will need to develop and implement a generic object reID capability which will be applied to test data and performance will be measured on a self-test dataset.

For vehicle reID, all vehicles in the test set will need to be mapped to a common geo-coordinate frame and accuracy of geo-localizations will be measured. Full sensor pose will be available to performer systems during test runs. Geo-localization will be tested both as an independent algorithm as well as for the end-to-end reID system. For independent algorithm testing, camera coordinates of vehicles in the test video will be provided at test time and performer systems will remap to geo-coordinates which will be measured for accuracy. For end-to-end reID, geolocation accuracy of vehicles identified by the system will be measured.

For person reID, all people in the test set will need to be mapped to a common geo-coordinate frame and accuracy of geo-localizations will be measured. Camera location will be available to performer systems during test runs but orientation will not be available. Geo-localization will be tested both as an independent algorithm as well as for the end-to-end reID system. For independent algorithm testing, camera coordinates of subjects in the test video will be provided at test time and performer systems will remap to geo-coordinates which will be measured for accuracy. For end-to-end reID, geolocation accuracy of people reidentified by the system will be measured.

1.A.1.c Phase 3

During a 12-month Phase 3, Video LINCS teams will demonstrate the feasibility of reID for generic objects and extend and develop new reID capabilities for people and vehicles.

For people and vehicles, reID will be applied to temporally distant intervals where temporary attributes may vary and metadata may have errors and/or might not be available. ReID will be performed across videos from different collection types, and the number of collection types will expand beyond those used in Phase 2. The number of distinct people to be re-identified in the test collection will increase to 2,500, and there will be 10,000 additional people present in the imagery to serve as potential confusers. The Phase 3 vehicle test set will have approximately 2000 distinct vehicles which will be measured for reID performance, and there will be additional vehicles present in the imagery to serve as potential confusers.

For generic objects, systems will need to autonomously detect and reID objects throughout the collection, without any prior information regarding the types or characteristics of objects to be re-identified. Objectives will remain consistent to those in Phase 2 with evaluation progressing from the self-test leaderboard to that on sequestered datasets with independent T&E system runs (the sequestered data leaderboard).

All people and vehicles in the test set will need to be mapped to a common geo-coordinate frame and accuracy of geo-localizations will be measured. Approximate camera location will be available to performer systems during test runs providing the general vicinity and no orientation information will be provided. Geo-localization will be tested both as an independent algorithm as well as for the end-to-end reID system. For independent algorithm testing, camera coordinates of people, vehicles and objects in the test video will be provided at test time and performer systems will remap to geo-coordinates which will be measured for accuracy. For end-to-end reID, geolocation accuracy of people, vehicles and objects identified by the system will be measured.

1.B Team Expertise

Collaborative efforts and teaming across candidate Offerors are highly encouraged. It is anticipated that teams will be multidisciplinary and may include expertise in one or more of the disciplines listed below. This list is included only to provide notional potential expertise areas for the Offerors;

satisfying all the areas of technical expertise below is not a requirement for selection and unconventional or innovative team expertise may be needed based on the proposed research.

Proposals shall include a description and the mix of skills and staffing that the Offeror determines will be necessary to carry out the proposed research and achieve program metrics.

• Artificial Intelligence

• Computer vision, including object detection, tracking, person/vehicle/object modeling, generic vision learning

• Deep learning

• Geometric camera projections and inverse projections

• Image and video geo-localization

• Machine learning

• Modeling and simulation

• Open set classification

• Re-identification

• Soft biometrics

• Software engineering

• Software integration

• Systems integration

• Truthed video data collection, annotation (truthing in both anonymized identities and geo-location), including potential HSR

• Vehicle fingerprinting

• Video data generation (including simulation, generative modeling)

1.C Program Scope and Limitations

Proposals shall explicitly address:

• Underlying theory: Proposed strategies to meet program-specified metrics must have firm theoretical bases that are described with enough detail that reviewers will be able to assess the viability of the approaches. Proposals shall properly describe and reference previous work upon which their approach is founded.

• Research & Development approach: Proposals shall describe the technical approach to meeting program objectives and metrics.

• Technical risks: Proposals shall identify technical risks and proposed mitigation strategies for each identified risk.

• Software development: Proposals shall describe the approach to software architecture, modularization, and integration.

The following areas of research are out of scope for the Video LINCS program:

• Research that does not have strong theoretical and experimental foundations

• Approaches that rely solely on integration of existing technologies without experimental or scientific improvements

• Development of hardware or sensors

• Approaches that consist merely of integrating currently existing software systems

• Research of non-visible image/video modalities (e.g. audio, emissive infrared)

• Research on face recognition. Existing face recognition technologies may be employed, but there should be no expectation that recognizable faces will be present in the video.

• Research on license plate recognition. Existing license plate recognition technologies may be employed, but there should be no expectation that recognizable license plates will be present in the video.

• Video from space-based platforms

• Methods that require a human in the loop

• Research of technologies that cannot be delivered with full US Government rights

Performers are permitted to use data from external sources provided that it is consistent with an approved privacy plan, may be shared across all performer and T&E teams, and is not proprietary.

Please include a listing of any datasets that you wish to share within your proposal to the effort.

Any data used for training shall be made available to all selected performer and T&E teams on

Video LINCS and be provided with Government Purpose Rights. Datasets need not be made public beyond the program teams.

1.D Program Data

Video data of a wide range of objects (people, vehicles and generic objects) from many video sensor types and collection conditions with accurate ground truth will be necessary for characterizing performance of Video LINCS developments (T&E). Similar data will likely be used by performers for their R&D as they develop Video LINCS solutions.

The Government will provide a list of nominal video types to be included in test video collections at program kickoff. Nominal video types to be included in evaluations include stationary and mobile sensors that are air, building, and ground based (e.g. drone/UAV, security/CCTV, ground mobile).

Performer teams will be responsible for obtaining (collecting, curating, simulating) and annotating data in support of R&D objectives. All data collected and/or annotated by performer teams under

Video LINCS funding, as well as any data used by performer teams for Video LINCS R&D must be shared program wide with other performer and T&E teams. Such data may be used exclusively by the team who collected and curated the data for the initial evaluation reporting period in which it was used, and must be shared program wide after performance results for that period are reported to IARPA management.

1.D.1 Development Data

Offerors have the freedom to propose any solution to the Video LINCS challenges that will meet program objectives. There is no requirement that such solutions must require training or validation data, but truthed data collection/curation and annotation is highly recommended. For solutions that propose the use of training or validation data, the following is required:

1. A Privacy Protection Plan (PPP) that comprehensively describes the efforts the teams will take to protect personally identifiable information and safeguard the security of any personal data collected or services involved in collection, transmission, processing, and storage of this data. Any claims that data are anonymous must be based on evidence and supported with sufficient information regarding how the data have been anonymized.

Version 1.0 of the Video LINCS Privacy Plan shall be included in the Offeror’s proposal as an appendix that covers all datasets to be leveraged as part of the proposed research approaches. This appendix will not count against page count limits. The Video LINCS

Privacy Plan shall be updated at the beginning of each Phase and when new sources of data or datasets are proposed for use within a Performer’s Video LINCS research activities.

2. A detailed plan for collection/curation/annotation to demonstrate scope and anticipated contribution to the proposed research.

3. All data collections that may potentially contain personally identifiable information must receive Institutional Review Board (IRB) approval.

All data collected, curated, annotated and/or simulated by performer teams under Video LINCS funding, as well as any data used by performer teams for Video LINCS R&D must be shared program wide with other performer and T&E teams. Such data may be used exclusively by the team who collected, curated, annotated and/or simulated the data for the initial evaluation period in which it was used, and must be shared program wide after each evaluation waypoint

(nominally 6 month waypoints throughout the program - see Figure 2 and

4. Table 5).

5. If a proposed solutions includes data collection, curation, annotation and/or simulation, a minimum of two collections/curation/annotation/simulation events must be performed within each phase (aside from Phase 3 which requires at least one such data generation event), with program-wide sharing of at least one of those datasets before the phase is 2/3 complete.

1.D.2 Evaluation Data

The T&E team will collect, curate and annotate data for evaluation. The T&E team will provide access to self-test datasets for performers to run algorithms on their own hardware throughout the program lifecycle, and performers will submit containerized (nominally Docker) software using/conforming to the framework and APIs (Section 1.H.1) specified by the T&E team for independent runs on sequestered evaluation datasets.

1.E Testing & Evaluation

T&E will be conducted by an independent team of Government and contractor staff carrying out evaluation and analyses of Performer research Deliverables using program test datasets and protocols. The Video LINCS Program will pursue rigorous and comprehensive T&E to ensure that research outcomes are well characterized, deliverables are aligned with program objectives, and that algorithm performance is measured across diverse conditions. T&E activities will inform

IARPA and Government stakeholders on Video LINCS research progress and serve as invaluable feedback to the Performers to improve their research approaches, algorithm training practices, and system development. Figure 3 shows the nominal Video LINCS T&E workflow.

T&E Data Collection

Performer Self-Test

T&E Infrastructure

Instrumented Data Collection, Curation

& Annotation

Common Framework

& APIs

Sequestered Data Runs

Manual Annotation

Baseline

Visualization Framework

Performer TA-1

Performer TA-2 Self-Test Dataset

Sequestered Dataset

Leaderboards

TA1 Test Harness

TA2 Test Harness

Baseline TA-1

Baseline TA-2

Performer TA-1

Performer TA-2

Performance Analysis

T&EPerformer

Figure 3: Test and Evaluation. The T&E team will collect datasets for performer self-test and T&E sequestered evaluation. Performers will develop systems conforming to the program common framework and APIs and submit containerized software for independent

T&E runs on sequestered data. Scores will be posted to program-wide leaderboards.

Leaderboards are planned to facilitate ongoing evaluation and ranking of performer systems against T&E datasets. Both self-test and sequestered data leaderboards will be instantiated by

T&E. For the self-test leaderboard, performers will be provided test video datasets to facilitate system runs on performer hardware. System output from performer runs on the self-test dataset will be submitted to a server hosted by the T&E team which will score the submission and performance results will be posted on a system wide leaderboard. Performers will not be permitted to view or probe the self-test data. The self-test leaderboard will provide rapid evaluation cycles to assist with development and provide an unofficial ranking across performers. Monthly submissions to the self-test leaderboard will be required in the period before the sequestered data leaderboard is operational (see Figure 2).

For official program evaluations, performer systems will be submitted to run independently on

T&E hardware which will contain sequestered video test data. Monthly submissions will be required, with semi-annual evaluation period waypoints where the systems will be officially ranked and reported to IARPA management. T&E results from all Performers will be shared with all teams to establish an understanding of the current state and progress of Video LINCS research.

T&E results will also be shared with USG external stakeholders, including their contractors, for

Government purposes. IARPA may conduct other supplemental evaluations or measurements at its sole discretion to evaluate the Performers’ research and Deliverables.

For both leaderboards, the T&E team will post finer grained performance measures beyond the official program metrics to provide insights into where systems require improvement. Such feedback from T&E will assist teams with determining conditions under which the systems require improvement without exposing the test set.

1.F Program Metrics

Achievement of metrics is a performance indicator under IARPA research programs. IARPA has defined Video LINCS program metrics to evaluate effectiveness of the proposed solutions in achieving the stated program goals and objectives, and to determine whether satisfactory progress is being made. The metrics described in this BAA are shared with the intent to scope the effort, while affording maximum flexibility, creativity, and innovation to Offerors proposing solutions to the stated problem.

The final Video LINCS T&E protocols and evaluation methodology will be provided at program kickoff. Program metrics may be refined during the various phases of the Video LINCS program;

if metrics change, revised metrics will be communicated in a timely manner to Performers. The evaluation methodology may be revised by the Government at any time during the program lifecycle to better meet program needs.

For TA-1 - ReID, the goal is to re-identify the same object throughout a video collection. Ideally, all appearances of the same object should be declared to be the same object by Video LINCS systems, without any misses and without matches to different objects (false positives). This is well encapsulated by the F1 measure as described by Ristani et al [2]

F1 = TPSys

(TPGT + (TP + FP)Sys) 2⁄

2 TPSys

2 TPSys + FPSys + FNSys where

TP = True Positive, FP = False Positive, FN = False Negative, GT = Ground Truth, Sys = System which is the number of correct system detections scaled by the average of the number of ground truth and system computed detections. Thus, this metric measures how often the correct object is re-identified, with even penalties for false positives and false negatives.

The processing time speedup for reID will also be measured, where the processing time for automated system reID will be computed relative that required for manual reID.

For TA-2 – Geo-localization, the ground-plane accuracy will be measured by computing the position error in meters

||(long, lat, alt)Sys - (long, lat, alt)GT||2 averaged over all object appearances.

Runtime for both TA-1 and TA-2 will be measured relative to test-set video length/runtime. The primary limiting factor for runtime requirements will be to ensure sufficient capacity for all performer submissions on the T&E server infrastructure.

Table 2 provides anticipated primary program performance milestones throughout the program phases.

Table 2: Primary program metrics and performance milestones by Phase.

Phase 3Phase 2Phase 1ConditionsObjective

0.90.750.6Temporally proximatePersonTA1: ReID accuracy: (F1)

0.750.6N/ATemporally distanced

0.750.6N/ATemporally proximateVehicle

0.5N/AN/ATemporally distanced

0.5N/AN/AGeneric Object

1000x100x10xvs. manual annotationTA1: ReID speedup

2510Full sensor pose provided

(position and orientation)

TA2: Geo-localization (m)

(accuracy in meters)

510N/ASensor position provided

(no orientation)

10N/AN/AGeneral vicinity provided

(w/in 100m, no orientation)

1x2x5xOn T&E servers (TBD)TA1 + TA2: Runtime

(relative to video runtime)

1.G Waypoints - Program Milestones and Deliverable timelines

Waypoints are the means by which the Performer clearly demonstrates the quantitative and timely progress that must be made for the overall concept to meet end-of-phase Milestones. In other words, the intent of Waypoints is to provide a clear measure of progress towards meeting the program Milestones so the PM and advisors can provide more effective guidance and assistance to the Performers. Performance against these Waypoints will be reviewed throughout the program, and the PM and advisors will use performance against the Waypoints to assess whether course corrections are needed to ensure program success. The Government has defined Waypoints for all

Performers. Progress against these Waypoints will be reviewed during site visits.

Table 3 shows the expected recurring events and deliverables that will occur across all phases of the program by all performers. Table 4 shows the expected events that will occur for all performers across both phases.

Table 5 shows the expected submissions for all performers across all phases of the program. Table

6 shows the expected deliverables for all performers across all phases of the program. Offerors may add other waypoints or deliverables in addition to the minimum set listed in the tables below.

Table 3. Recurring Events and deliverables expected by all Video LINCS performers.

Recurring Event / Deliverable Frequency Description

PI Virtual Meetings / Tagups Monthly

Monthly teleconference with Video LINCS

PM, SETA, and T&E teams.

Monthly Status Reports (MSR)

Monthly

Due on the 15th of each month; Technical and cost reports.

DELIVERABLE: MSR documentation

Table 4. Events that are expected to occur during the life of the program. Locations are

TBD; (most events anticipated in the Washington, D.C. metro area - WMA)

Event Month Description

Phase Kickoff

Meetings 1, 19, 37

Location: TBD. Discuss phase plans across program performers and T&E. Potentially virtual or combined with PI Workshop. Presentations shall be shared with the PM 5 calendar days prior to the kickoff meeting.

Site Visits 4, 10, 16, 22, 28, 33, 40

At performer sites. Discussions about ongoing work with PM, SETA, and T&E team.

PI Workshops 18, 35, 47

Location: TBD. Attendance in person or virtual.

Presentations about ongoing work to all performers, Government team and Government partners.

Table 5. Submissions expected by performers across the life of the program.

Submission Month Description

Leaderboard submissions

(TA-1 and TA-2)

Monthly

Leaderboard submission of system output (self-test leaderboard) to T&E scoring server or software system implementation (sequestered data leaderboard) to T&E hardware stack.

Submissions to T&E:

• Self-test leaderboard: System output for self-test dataset. Months 1-11

• Sequestered Data Leaderboard: Software meeting T&E API and specifications. Months

12-48

Evaluation

Waypoints

6, 12, 17, 24, 30, 35, 42, 47

Submissions included in “Leaderboard Submissions” with performance reported to IARPA management.

Data Sharing 6, 12, 17, 24, 30, 35, 42, 47

Program-wide sharing of data used in any submission in the current or prior evaluation periods

Table 6: Deliverables expected by performers across the life of the program.

Deliverable Month Description

Software

Deliverables

(TA-1 and TA-2)

Semi-annual, with final phase deliverables in months 17, 35, 47

Source code, build scripts and documentation for TA-1 and TA-2 facilitating reproduction of system operation by an independent user.

DELIVERABLE: Software for TA-1 and TA-2, including source code, build scripts, software containers and documentation.

End of Phase

Reports (Draft) 17, 35, 47

End of phase Final Reports highlighting work completed, metrics achieved, challenges, recommendations for future work and conclusions.

Submission: Draft Report for IARPA PMO

End of Phase

Reports (Final) 18, 36, 48

End of phase Final Reports highlighting work completed, metrics achieved, challenges, recommendations for future work and conclusions.

DELIVERABLE: Final Report

1.H Software Deliverable Formatting

Performers will be required to provide algorithms, software deliverables, design documentation, and operating instructions to execute the automated software in a manner that conforms to a standardized industrial method or methods that will be provided at program Kickoff. To facilitate planning, Offerors may assume that the standardized configuration will require the use of software containerization technology (e.g., Docker and a REST API). This means that the entirety of a

Performer’s system, including pre- and post-processing, must be included within the delivered software container. For models that require training, the expectation is for the initial model training to occur on Performer systems, with the ability for the T&E Team to re-train and test the model with the same and/or other data. Performers shall also provide source code to the government and

T&E team to enable modification of Performer containers to suit deployment constraints. Offeror teams that do not include the requisite expertise to conduct such software development shall include costs in their proposal to obtain software development support.

Each team is required to include among their key personnel a Lead System Integrator (LSI) who shall be responsible for preparing software Deliverable subcomponents, modules, and systems, performing quality control of Deliverable, and integrating key components into the primary Video

LINCS system(s). The LSI will also oversee communication and coordination across a Performer’s research teams including subcontractors, if applicable, to ensure research products are functional and following software coding best practices (e.g., inline comments, documentation). Additional team members and roles are dependent on the proposed research, as such, there is no predetermined or required skill mix.

1.H.1 Program API and Framework

The Video LINCS Program will utilize a standardized Application Programming Interface (API) for all software Deliverables and evaluations. The program will also require the use of a common processing framework, such as the Open Media Processing Framework (OpenMPF Error!

Reference source not found.) for development. The first version of the Video LINCS API and nominal framework will be provided to Performers at the beginning of the program and updated periodically thereafter. The API will define function calls, data structures, and display creation and management for operating and evaluating Video LINCS software in a standardized manner. All

Performer solutions must be compatible with the Video LINCS API.

1.I Meeting and Travel Requirements

Offerors are expected to assume responsibility for administration of their projects and to comply with contractual and program requirements for reporting, attendance at program workshops, and availability for site visits. The following paragraphs describe typical expectations for meetings and travel for IARPA programs as well as the contemplated frequency and locations of such meetings.

In addition to ensuring that all necessary details of developed software, algorithm, and operational instructions are clear and complete, each Performer will be required to be available for questions and troubleshooting from the T&E Team in Performer status meetings.

1.I.1 Technical Exchange Meetings and Workshops

All Performer teams are expected to attend Technical Exchange meetings and workshops, including key personnel from prime and subcontractor organizations.

The Video LINCS program intends to hold a program Kick-off Meeting in the first month of the program (in person) and the first month of each subsequent program phase. In addition, the program will hold PI Review Meetings / Workshops in the final month of each phase to discuss progress program-wide and with Government advisors and stakeholders. Kick-off Meetings and

PI Review Meetings may be combined for logistical convenience. The dates and locations of these meetings are to be specified at a later date by the Government, but for planning purposes, Offerors should use the approximate times and locations listed in Table 4. Both types of meetings are likely be held in the Washington, D.C. metropolitan area, but IARPA may opt to co-locate the meeting with a relevant external conference or workshop to increase synergy with stakeholders. IARPA reserves the right to hold the meeting virtually for logistical or health and safety reasons.

Kick-off Meetings will typically be two days in duration and will focus on plans for the coming

Phase, Performer planned research, and internal program discussions. PI Review Meetings will typically be two days in duration and will have a greater focus on communicating program progress and plans to USG stakeholders. These meetings will include additional time allocated to presentation and discussion of research accomplishments as well as interactive system demonstrations for Government stakeholders.

In both cases, the workshops will focus on technical aspects of the program and on facilitating open technical exchanges, interaction, and sharing among the various program participants.

Program participants will be expected to present the technical status and progress of their projects to other participants and invited guests. Individual sessions for each performer team with the Video

LINCS PM and T&E Team may be scheduled to coincide with these workshops. Non-proprietary information will be shared by performers in the open meeting sessions; proprietary information sharing shall occur during individual breakout sessions with the Video LINCS PM and the T&E

Team.

1.J Site Visits

Our expectation is that Site visits by the Government Team to all performers will take place semiannually during the life of the program. These visits will occur at the Performer’s facility.

Reports on technical progress, details of successes and issues, contributions to the program goals, and technology demonstrations will be expected at such site visits. Performers shall participate and provide final meeting documents, to include captured action items, within 15 calendar days following the meeting. Draft materials, for any presentations, are due 5 workdays prior to the meeting. IARPA reserves the right to conduct additional site visits on an as-needed basis.

1.K Period of Performance

The Video LINCS program is envisioned as a 48-month effort intended to kickoff in August 2024.

Phase 1 (Base Period) of the program will last 18 months, Phase 2 (Option Period 1) will last 18 months, and Phase 3 (Option Period 2) will last 12 months. Offerors are required to submit a proposal that addresses all three phases of the program.

1.L Place of Performance

Work for the Video LINCS program will be conducted at the Performers’ sites.

1.M REFERENCES

[1] M. Ye, J. Shen, G. Lin, T. Xiang, L. Shao and S. C. H. Hoi, “Deep Learning for Person Re-

Identification: A survey and Outlook." IEEE Trans. On Pattern Analysis and Machine

Intelligence, June 2022.

[2] E. Ristani, F. Solera, R. Zou, R. Cucchiara and C. Tomasi, “Performance Measures and a Data

Set for Multi-Target, Multi-Camera Tracking,” Proceedings of the European Conference on

Computer Vision, Amsterdam, the Netherlands, 2016.

[3] https://openmpf.github.io/