IARPA-BAA_SINTRA Amendment 002_v2_C.pdf

IARPA

BROAD AGENCY ANNOUNCEMENT

IARPA-BAA-22-02

Space debris Identification and Tracking (SINTRA) Program

IARPA-BAA-22-02

Release Date:

September 9, 2022

Amendment 002: October 14, 2022

ALL CHANGES FOR AMENDMENT 001 ARE HIGHLIGHTED IN YELLOW.

Amendment 002: October 21, 2022

ALL CHANGES FOR AMENDMENT 002 ARE HIGHLIGHTED IN GREEN.

i

BROAD AGENCY ANNOUNCEMENT: IARPA-BAA-22-02

SINTRA Program

TABLE OF CONTENTS

GENERAL INFORMATION

SECTION 1: FUNDING OPPORTUNITY DESCRIPTION

1.A. Program Overview

1.A.1. Technical Challenges and Objectives

1.A.2. Program Phases

1.B. Team Expertise

1.C. Program Scope and Limitations

1.D. Program Data

1.D.1. Development Data

1.D.1.1. Government Research and Test Sets

1.D.1.2. Researcher Collections

1.D.1.3. External Data Sources

1.D.2. Evaluation Data

1.E. Test and Evaluation (T&E)

1.F. Program Metrics

1.G. Program Waypoints, Milestones, and Deliverables

1.G.1. Program Waypoints

1.G.2. Program Milestone, Waypoint, and Deliverables Timeline

1.G.3. Software Deliverable Formatting

1.G.3.1. Program API

1.H. Meeting and Travel Requirements

1.H.1. Workshops

1.H.2. Site Visits

1.I. Period of Performance

1.J. Place of Performance

SECTION 2: AWARD INFORMATION

SECTION 3: ELIGIBILITY INFORMATION

3.A. Eligible Applicants

3.A.1 Organizational Conflicts of Interest (OCI)

3. A.2 Multiple Submissions to the BAA

3. B. U.S. Academic Institutions ii

3. C. Other Eligibility Criteria

4. A. Proposal Information

4. B. Proposal Format and Content

4. B.1 Volume 1: Technical and Management Proposal

4. B.1.a Section 1: Cover Sheet & Transmittal Letter

4. B.1.b Section 2: Summary of Proposal (see below for page limit)

4. B.1.c. Section 3: Detailed Proposal Information

4.B.1.d. Section 4: Attachments

4.B.2. Volume 2: Cost Proposal (No Page Limit)

4. B.2.a Section 1: Cover Sheet

4. B.2.b. Section 2: Estimated Cost Breakdown

4. B.2.c. Section 3: Supporting Information

4. C. Submission Details

4. C.1. Due Dates

4. C. 2 Proposal Delivery

4. D. Funding Restrictions

SECTION 5: PROPOSAL REVIEW INFORMATION

5.A. Technical and Funding Availability Evaluation Factors

5.A.1. Technical Evaluation Factor (technical criteria listed below)

5. A.1.a. Overall Scientific and Technical Merit

5. A.1.b. Effectiveness of Proposed Work Plan

5. A.1.c. Contribution and Relevance to the IARPA Mission and Program Goal

5. A.1.d Relevant Experience and Expertise

5. A.1.e Resource Realism

5.A.2. Funding Availability Factor

5. A.2.a. Budget Constraints

5. A.2.b. Program Balance

5.B. Method of Evaluation and Selection Process

5.C. Negotiation and Contract Award

5.D. Proposal Retention

SECTION 6: AWARD ADMINISTRATION INFORMATION

6.A. Award Notices

6.B. Administrative and National Policy Requirements

6.B.1. Proprietary Data iii

6.B.2. Intellectual Property

6.B.3. Human Use

6.B.4. Animal Use

6.B.5. Publication Approval

6.B.6. Export Control

6.B.7. Subcontracting

6.B.8. Reporting

6.B.9. System for Award Management (SAM)

6.B.10. Representations and Certifications

6.B.11. Lawful Use and Privacy Protection Measures

6.B.12. Public Access to Results

6.B.13. Other Contract Requirements

6.B.13.a. Provisions

6.B.13.b. Clauses

Appendix A: Templates for Volume 1: Technical Proposal

A.1 Cover Sheet for Volume 1: Technical Proposal

Appendix A.2 Academic Institution Acknowledgment Letter

Appendix A.3 Intellectual Property Rights

Appendix A.4 Organizational Conflicts of Interest Certification Letter

Appendix A.5 Three Chart Summary of the Proposal

Appendix A.6 Research Data Management Plan (RDMP) BAA 22-02

Appendix B: Templates for Volume 2: Cost Proposal

Appendix B.1 Cover Sheet for Volume 2: Cost Proposal

Appendix B.2 Prime Contractor/Subcontractor Cost Element Sheet for Volume 2: Cost Proposal

Appendix B.3 - Software and IP Costs

Appendix B.4 – Travel Costs Trip breakdown

Appendix B.5 – Contract Deliverables Table

Appendix B.6 – Technical References/Footnotes

GENERAL INFORMATION

This notice constitutes a Broad Agency Announcement (BAA) and sets forth research of interest in the area of space debris detection, tracking, and characterization. The solicitation process will follow Federal Acquisition Regulation (FAR) Part 35, Research and Development Contracting, as supplemented with additional information included in this notice. Awards based on responses to this BAA will be considered the result of full and open competition.

1. Federal Agency Name – Office of the Director of National Intelligence (ODNI)/Intelligence Advanced

Research Projects Activity (IARPA)

2. Funding Opportunity Title – Space debris Identification and Tracking (SINTRA) Program

3. Announcement Type – Initial

4. Funding Opportunity Number – IARPA-BAA-22-02

5. Catalog of Federal Domestic Assistance Numbers (CFDA) – Not applicable

6. Questions

Submit questions on administrative, technical, or contractual issues by email to DNI-IARPA-BAA-22- 02@iarpa.gov. All requests must include the full name and affiliation of a point of contact. Do not send questions with proprietary content. A consolidated Question and Answer response will be posted on SAM.gov for Contract Opportunities website (https://SAM.gov/) and linked from the IARPA website (https://www.iarpa.gov/index.php/research-programs/sintra). No answer will go directly to the submitter. IARPA will accept questions until October 21, 2022 @ 5:00 PM EST.

7. Dates

7.1 Posting Date: September 9, 2022

7.2 Proposal Due Date for Initial Round of Selections: November 14, 2022 @ 4:00 PM EST

7.3 BAA Closing Date: November 22, 2022, 4:00 PM EST

8. Anticipated individual awards – Multiple awards anticipated

9. Types of instruments that may be awarded – Procurement Contracts and Other Transactions 1

10. Agency Points of Contact

ATTN: IARPA-BAA-22-02

Office of the Director of National Intelligence Intelligence Advanced Research Projects Activity Washington, DC 20511 Electronic mail: DNI-IARPA-BAA-22-02@iarpa.gov

11. Program Manager (PM) – Dr. Alexis Truitt

12. Program Website – http://www.iarpa.gov/index.php/research-programs/sintra

13. BAA Summary –The SINTRA program seeks to develop novel, innovative techniques to detect, track, and characterize non-trackable orbital space debris. The detection of non-trackable orbital debris

1 Procurement Contract: This is a standard government contract that follows the processes, format and terms and conditions as outlined in the Federal Acquisition Regulations (FAR) and supplementing Agency specific regulations.

Other Transaction Authority: Agreements generally are not subject to the federal laws and regulations governing procurement contracts and thus are not required to comply with the Federal Acquisition Regulation (FAR), its supplements, or laws that are limited in applicability to procurement contracts. They may be used with non-traditional contractors under certain circumstances.

would enable safer operation of valuable government and commercial space assets. SINTRA performers will focus on two task areas (TAs): developing novel methods to identify debris signatures (TA1) and;

establishing techniques for the persistent monitoring of the debris distribution (TA2).

SECTION 1: FUNDING OPPORTUNITY DESCRIPTION

IARPA often selects its research efforts through the BAA process. The use of a BAA solicitation allows a wide range of innovative ideas and concepts. The BAA will appear under Contract Opportunities on https://sam.gov as well as the IARPA website at http://www.iarpa.gov. The following information is for those wishing to respond to this Program BAA.

This BAA (IARPA-BAA-22-02) is for the Space debris Identification and Tracking (SINTRA) Program.

IARPA seeks innovative solutions for the SINTRA Program, which is envisioned to be a 48-month effort, beginning approximately May 1, 2023, through April 30, 2027.

1.A. Program Overview

Orbital debris is defined as any human-made space object orbiting Earth that no longer serves any useful purpose.1 Orbital debris poses a risk to all space missions, including those of the Intelligence Community (IC). With an average low Earth orbit (LEO) impact velocity of 22,500 MPH, even the smallest of debris can cause significant damage, as demonstrated by the 3.8 mm diameter pit produced by the impact of a

0.2 mm paint chip on STS-71.2 Currently, there are over 100 million objects greater than 1 mm orbiting the Earth,3,4 however, less than 1 percent of debris that can cause mission-ending damage are currently tracked.5 Due to the dynamic nature of the near-Earth space environment, predicting the trajectory of the debris is extremely difficult, necessitating persistent monitoring.6 While debris larger than 10 cm can be detected and tracked, smaller debris cannot be tracked using current capabilities.7 Debris that is too small to track, often termed “lethal non-trackable debris” (LNT),8 can create significant damage to spacecraft and jeopardize space missions. The detection, tracking, and characterization of lethal non-trackable space debris would support the safe operation of valuable space assets worldwide.9

SINTRA is planned as a 48-month multi-phase program which will drive the state-of-the-art for exploitation of data capable of detecting orbital debris signatures, demonstrate signature detection, and develop automated methods for signature analysis, debris tracking, and debris characterization. If successful, SINTRA would enable the first tracking capability for the small debris population, reducing risk to space operations.

SINTRA will aim to develop innovative technologies to: (a) improve orbital debris detection and tracking;

(b) reduce uncertainties of debris data in orbit propagation and prediction; and (c) characterize orbital debris size, density, and mass.

SINTRA will address the limitations in current assessments of the orbital debris population. NASA estimates that there are over 100 million particles greater than 1 mm orbiting the Earth.3 However, estimates for small debris are largely based on collisional detections during the US space shuttle mission, which has now been retired.10 These collisional detections were restricted to measuring debris strikes at altitudes below 600 km, the upper limit of space shuttle operations, and were then used to estimate the small debris population at a range of altitudes.11

While ground-based sensors continue to improve their detection capabilities, the ground sensor detection sensitivity rapidly decreases with increasing altitude and is limited to observing high latitudes. The ground sensors in the United States Space Surveillance Network (SSN) can detect 10 cm objects at 2,000 km altitude in LEO and 1 m objects at 35,786 km altitude in GEO. Unfortunately, ground-based sensors are not able to track small objects due to the debris’ relatively high angular velocity and must remain in staring mode to count the number of objects passing through their small fields of view. The largest source of uncertainty exists in the ability to interpret the signal strength to determine the size or mass of the object passing through the field of view.7

The orbital debris models developed at NASA are based on either impact data from on-orbit experiments or remote detection using a variety of optical telescopes and radar systems, as illustrated in Figure 1. There is, however, a gap between these measurement types that has yet to be constrained by actual data, a problem that SINTRA hopes to solve. Much of the contributions for radar measurements are the Haystack Ultrawideband Satellite Imaging Radar (HUSIR) and the Haystack Auxiliary radar (HAX) systems operated by MIT Lincoln Laboratory. Additional information regarding contributing sensors can be found on the NASA Orbital Debris Office website.12

Figure 1: Measurement data used by the NASA Orbital Debris Program Office to describe orbital debris about Earth. Figure from NASA/ODPO.12

Most of the small orbital debris population resides in high LEO, with a mean altitude of 850 km.3 More orbital debris is created than removed due to the diminished atmospheric drag at high altitudes, making current orbit characterization for small debris even more uncertain.5

An estimate from the SSN for the number of debris particles greater than 10 cm is displayed in Figure 2, published by the NASA Orbital Debris Program office. Prior to 2010, two events are responsible for adding 40% to the LEO debris estimates:13 the Chinese Anti-satellite Test (ASAT) in 200714 and the collision of the Cosmos-Iridium satellites in 2009.15 These two events significantly increased the challenges of operating in LEO. More recently, the ASAT test conducted by the Russian Federation in November 2021 has caused a substantial increase in catalogued debris.16

Figure 2: Space Surveillance Network estimate of the number of objects greater than 10 cm as a function of time, based on data available on 1 March 2022. The three annotated increases debris correspond to (1) the Chinese ASAT test conducted in 2007, (2) the accidental collision between Iridium 33 and Cosmos 2251 in 2009, and (3) the Russian ASAT test conducted in November 2021. Image from NASA/ODPO.16

Figure 3: Space Surveillance Network estimate of the number of objects greater than 10 cm as a function of altitude. Image from NASA/ODPO.17

Figure 3 shows the increase in catalogued objects as a function of altitude, also published by the NASA Orbital Debris Program Office. Again, the Cosmos-Iridium collision and Chinese ASAT test debris fragments dominate the increase in objects from 2000 to 2010. The increase objects from 2010 to 2020 is due to the initial build-up of the Starlink large constellation and by the proliferation of CubeSats below about 650 km altitude. The increase from 2020 to 2022 is due to the Russian ASAT test, and also continues to be driven by proliferation of large constellations by commercial and government entities.17

With the increase in the number of satellites launched over time, there is potential for rapid growth of the debris population. Therefore, active means are necessary for quickly characterizing the dynamic orbital debris environment, allowing for the ability to prevent satellite damage through orbit selection, and collision mitigation and remediation. A non-collisional capability to detect small debris from the ground, and in situ with existing sensor technology, would provide the most immediate contribution for understanding the small debris population.

The successful completion of the SINTRA program would provide the world with the first collision-free detection method for small orbital debris, supporting the safe operation of our valuable space assets.

A robust data collection effort will be executed as part of the SINTRA program to assemble diverse debris datasets for use in Research and Development (R&D) for algorithm training, and in Test and Evaluation (T&E) for comparing the algorithm performance against ground truth. As described in Section 1.E, the T&E Team will conduct dataset development activities throughout the life of the program. A portion of these data will be made available to Performers for R&D. In addition, Performers will have the ability to conduct their own supplemental dataset collections. At the conclusion of each program phase, Performers shall make all datasets used for system development available to the T&E Team for the purpose of distribution to other Performers. The sharing of datasets across Performers will help ensure that improvement in performance attributable to a team’s superior technical approach rather than a greater accrual of development data resources. Public release of proprietary datasets is not a requirement; however, release for use within the SINTRA program is required. Additional details on program data can be found in section 1.D. Program Data below.

The SINTRA program will pursue rigorous and comprehensive independent T&E to ensure that research outcomes are well characterized, deliverables are aligned with program objectives, and algorithm performance is measured across the full range of data conditions. T&E activities will not only inform Government stakeholders on SINTRA research progress but will also serve as valuable feedback to the Performers to improve their research approaches, algorithm training practices, and system development.

The SINTRA program will work closely with Government leaders in the fields of orbital debris and space situational awareness to continually refine and improve T&E methodologies.

1.A.1. Technical Challenges and Objectives

Offerors shall address two task areas (TAs) to meet the goal of SINTRA to exploit data for the detection, tracking, and characterization of space debris 0.1 – 10 cm in size. Offerors will determine the threshold at which to track debris clouds instead of individual debris objects. Offerors must propose novel approaches to each of these TAs, and if selected as a Performer, will be required to create an end-to-end technology that incorporates software components from each TA.

Debris Detection, Tracking, and Characterization (TA1): The goal of TA1 is to research and develop novel, innovative techniques to detect, track, and characterize the size, mass, and density of space debris

0.1 – 10 cm in size, traveling in any orbital plane around the Earth.

During Task Area 1.A (TA1.A), the Performer will be assessed against known tracked debris objects 10 – 40 cm in size in Low Earth Orbit (LEO), with orbital trajectories on publicly accessible databases (e.g., space-track.org).

During Task Area 1.B (TA1.B), the program metrics will extend to debris 0.1 – 40 cm in size, from LEO to Geosynchronous Orbit (GEO). Offerors must propose novel, explainable, techniques to establish accurate detection, tracking, and characterization of currently untracked debris and/or debris clouds to address this goal. It is assumed that there is an increased number of smaller debris traveling with the larger tracked debris, due to the power law size distribution. It is also assumed that larger debris can collide and create smaller debris objects. Additionally, smaller debris can travel independently of larger debris objects. Since debris less than 10 cm in size is not currently tracked and characterized with existing capabilities, Offerors must propose methods to evaluate this approach that are potentially generalizable to other Performer approaches. T&E Teams will finalize protocol for evaluating explanations in Phase 1 and implement this protocol in evaluations during subsequent phases.

Persistent Monitoring of the Debris Population (TA2): The goal of TA2 is to demonstrate the capability to persistently monitor the orbital debris population for objects 0.1 – 10 cm in size. The TA2 deliverable must incorporate the variations in orbital trajectory due to gravitational perturbations from the Sun and Moon, and non-gravitational forces, including solar radiation and atmospheric drag.

Similar to TA1, the Performer will be assessed against known tracked debris objects 10 – 40 cm in size in LEO during Task Area 2.A (TA2.A), and debris 0.1 – 40 cm in size from LEO – GEO during Task Area

2.B (TA2.B).

1.A.2. Program Phases

The SINTRA program is anticipated to be a 4-year (48 month) effort, comprised of two (2) Phases. Both Phases are being solicited under this BAA. Each Phase will be 24 months in duration. Phase 1, Contract Line Item (CLIN) 0001, will encompass Task Area 1, and Phase 2 (CLIN 0002) will encompass Task Area 2, as described above.

Offerors are required to propose to both Phases and Task Areas of the Program under this BAA.

Proposals that submit to one Task Area or Phase will be considered non-compliant.

SINTRA aims to develop new techniques to detect, track, and characterize debris and debris clouds of varying physical characteristics that influence the risk assessment decisions for potential impact damage, including debris size, shape, density, and orbital regime. Program phases are designed to test Performer systems against increasingly challenging risk scenarios.

Within each program phase, there are three evaluation milestones at which Performers submit software deliverables to the T&E team for evaluation against sequestered datasets. The Phase 1 milestones are: (1) debris signature detection; (2) debris tracking; and (3) debris characterization. The Phase 2 milestones are:

(1) debris detection time; (2) detection rate; and (3) revisit rate. Additionally, regression testing will be conducted during Phase 2 to ensure that the Phase 1 performance of Performer software deliverables is maintained during Phase 2 development.

The contract will have two contract line items (CLINs): CLIN 0001 for the Base period – Phase 1 TA1.

CLIN 0002 (option) for Phase 2 TA2. Each CLIN will have a separate period of performance. However, as outlined in Table 1, a low level of work is anticipated for TA2 during Phase 1. For example, the debris detection time metric is aligned with TA2, and will be evaluated during Phase 2 Milestone 1, but a relaxed metric is provided in Phase 1 to allow for investigation. Therefore, minor exploration of TA2 work and associated costs for this minor work are acceptable during Phase 1 (CLIN 0001). Please ensure this work is clearly identified in the proposal.

Each evaluation will be performed on sequestered datasets using Performer-submitted containerized software components. Aspects of evaluation that involve human adjudication or feedback will be performed by the T&E Team.

In Phase 1 and Phase 2, IARPA will provide development datasets and an evaluation harness developed by the T&E team. The same harness will be used for testing and evaluation of Performer-submitted software components at each subsequent evaluation milestone.2 The purpose of the evaluation harness is to enable Performer-site training, testing, and competition between debris algorithm components. IARPA will also furnish to Performers documentation of the program Application Programming Interface (API) developed by the SINTRA T&E Team. This Government Furnished Information (GFI) will facilitate productive and useful research in a consistent and standardized manner. The first version of the SINTRA API will be provided to Performers at the Phase 1 Kick-off Meeting and updated periodically thereafter.

Datasets developed by the T&E Team for each program phase will be partitioned into development and evaluation datasets. In addition to the T&E development data, Performers are expected to create their own development datasets aggregated from open source or self-generated and using creative techniques and heuristics to infer debris population distributions. These datasets must be submitted to the T&E Team at the conclusion of each phase. The T&E team will combine all Performer datasets into a single dataset, such that it is releasable to all Performers as discussed in section 1.D.1.2.

Evaluation of Performer progress towards program metrics will primarily utilize evaluation datasets that are sequestered. A portion of these datasets will be shared with Performers after each evaluation to enable Performers to conduct error analyses. The baseline API provided in Phase 1 will standardize system and subcomponent design and streamline comparative evaluations. More details on the datasets are available in section 1.D. and more details on the API are available in section 1.G.3.1. Program API

In addition to the development datasets and evaluation harness provided by the T&E team, IARPA will also furnish to Performers the use of the Naval Research Laboratory (NRL) Space Physics Simulation Chamber (SPSC). Scaled near-Earth space-like plasmas are created in the SPSC’s 5-m long by 1.8-m diameter main chamber and 2-m long by 0.55-m diameter source chamber. Independently controllable electromagnets allow for control of the shape of the axial magnetic field. As requested, each Performer will have the opportunity to conduct two separate two-week experiments per Program Phase. An overview of the NRL

2 Systems developed by one performer will not be shared with other performers, only submitted to T&E for official evaluations.

SPSC facility will be provided at the SINTRA Program Kick-off. Performers must coordinate the scheduling of experiments with NRL.

Documentation from the Army Research Laboratory describing the current and predicted sensitivity spectrum of Rydberg sensors will be provided as GFI. Sensitivity spectra is provided in units of electric field (volts/meter) and/or effective noise temperature (kelvin) as a function of frequency from 0 Hz to 1 THz. Further, ARL has developed a Rydberg sensor-modeling suite (called Rydiqule). This suite is expected to be publicly released by the end of CY 2022. The ARL information will include a brief document describing potential use cases of Rydiqule for the SINTRA program, and instructions for accessing the code and finding the documentation.

Additional GFI provided by IARPA to SINTRA Performers will include several weeks of curated high-resolution ionospheric measurements made over the southern portion of the Mountain Time Zone and one year of lower resolution ionospheric measurements made along the East Coast of the Continental United States during the IARPA HFGeo Program. The HFGeo data will be included in the development datasets provided by the T&E team.

1.B. Team Expertise

Collaborative efforts and teaming among Offerors are highly encouraged. It is anticipated that teams will be multidisciplinary and should include expertise in one or more of the disciplines listed below. This list is included only to provide guidance for the Offerors; satisfying all the areas of technical expertise below is not a requirement for selection and unconventional or innovative team expertise may be needed based on the proposed research. Proposals should include a description and the mix of skills and staffing that the Offeror determines will be necessary to carry out the proposed research and achieve program metrics.

Adaptive optics Aerodynamic drag Astrodynamics Classical mechanics Data visualization Distributed processing Machine learning Materials science Modeling and simulation Multi-scale analysis Plasma physics Radar theory and operation Signal processing Software engineering Systems engineering Systems integration Thermodynamics

1.C. Program Scope and Limitations

Proposals shall explicitly address all the following:

Underlying theory: Proposed strategies to meet program-specified metrics must have firm theoretical bases that are described with enough detail that reviewers will be able to assess the viability of the approaches. Proposals shall properly describe and reference previous work upon which their approach is founded.

Research & Development approach: Proposals shall describe the technical approach to meeting program metrics.

Technical risks: Proposals shall identify technical risks and proposed mitigation strategies for each identified risk.

Software development: Proposals shall describe the approach to software architecture, modularization, and integration.

The following areas of research are out of scope for the SINTRA program:

Research that does not have strong theoretical and experimental foundations.

Approaches that do not pursue the detection, tracking, and characterization of debris.

Approaches that aim at debris removal or debris damage mitigation.

Approaches dependent on the currently unfunded development and launch of new satellites with debris detection sensors.

Trade studies for the development of new satellites with debris detection sensors.

Research that utilizes proprietary data that are not made available to other Performers.

US-based Offerors proposing to use existing space-based sensors must adhere to USG regulation. IARPA requires that all US-based Offerors proposing to use existing space-based sensors contact the NOAA Commercial Remote Sensing Regulatory Affairs (CRSRA) Office to determine if a license is required. If selected, a Performer must obtain the required license. The Performer must provide IARPA a copy of the NOAA CRSRA determination before the end of Year 1. The NOAA CRSRA Office contact form can be found here: https://www.nesdis.noaa.gov/about/our-offices/commercial-remote-sensing-regulatory-affairs.

Delivered software will be evaluated by the T&E Team on sequestered evaluation datasets. During each performance period, Performers will build software systems that will be run and evaluated by the T&E Team. Testing protocols do not allow for expert operators, but limited human-in-the-loop operation is permitted and will be carried out by the T&E Team. Notwithstanding testing and evaluation tasks that explicitly require human feedback, systems must be fully automated. Systems or algorithms that have been trained using human-in-the-loop methods may be submitted, provided they run autonomously.

1.D. Program Data

The SINTRA dataset will contain data from multiple sensor types, sensor locations, data formats, and data fidelity. To ensure sufficient data are provided for system development and statistically reliable evaluations, the program T&E Team will implement a robust and explicit data collection workflow.

The program will include two types of data – Development Data and Evaluation Data. Development data will be utilized by Performers, as they see fit, to conduct research, development, error analysis, and algorithm training. Evaluation data will be explicitly excluded from any algorithm training approaches and will be used by the T&E Team for performance evaluations of subcomponents, modules, and systems measured against the SINTRA program metrics.

Offerors shall include in their proposals the cost and resources to acquire, implement, and maintain the computing infrastructure necessary to carry out all work described in their proposals, including dataset development.

1.D.1. Development Data

Development data will be utilized by Performers, as they see fit, to conduct internal research, development, error analysis, and algorithm training. Each Performer is anticipated to have a unique technical solution to the SINTRA challenge, and accordingly may require different development data depending on their approach.

Proposals must specify the development data needed to carry out the proposed research and what data characteristics are necessary for the Offeror’s approach(es) to be successful at meeting program objectives.

Proposals must specify how the Offer plans to obtain the development data.

Additionally, as part of Volume 1 (Technical & Management Proposal) Section 3 (Detailed Proposal), Offerors should present a dataset development plan detailing how the Offeror intends to obtain these data.

Offerors’ unique data needs in a particular technical area should not be interpreted as requirements input into the program’s T&E data collections, but part of a comprehensive research approach that communicates a possible path towards meeting SINTRA program objectives.

There will be three types of development data used in the SINTRA program, as illustrated in Figure 4:

Government Research and Test Sets – Datasets collected, annotated, and curated by the T&E Team. Most of these data will be sequestered, but a small portion will be provided to Performers for R&D (prior to evaluation) and error analysis (following evaluation).

Researcher Collections – Datasets collected, annotated, and curated by R&D Performers. These datasets shall be delivered with a minimum of Government Purpose Rights, in accordance with

FAR 52.227-14.

External Data Sources – Data obtained by Performers that are available from third parties or that have been collected by a Performer outside of SINTRA shall be delivered with sufficient rights to allow the United States Government (USG) and the T&E Team to share this data with all Performers for their use in connection with the SINTRA Program.

Figure 4: Types of development data used in the SINTRA program.

1.D.1.1. Government Research and Test Sets

The T&E Team will develop SINTRA Research Sets (SRS) and SINTRA Test Sets (STS) throughout the program. The Government will release SRS to Performers to facilitate system development. STS will be sequestered for the purpose of blind testing and evaluation, though small portions of this dataset may be released for the purpose of error analysis following official evaluation milestones. The volume of data in the STS is anticipated to be much larger than the SRS.

The SRS datasets will include new collections from the radar and optical systems currently used to develop debris models, as detailed in Figure 1, to validate measurements by proposed systems. The new collections will be dependent on proposed solutions and may be coordinated across multiple sensors to ensure maximum utility to the performers.

The SRS datasets will also include collections of sensor data publicly accessible on the OpenMadrigal website. Led by the MIT Haystack Observatory since 1980, the OpenMadrigal project maintains and develops the Madrigal database, which is an open resource for geospace data. Madrigal is a robust, World Wide Web based system capable of managing and serving archival and real-time data, in a variety of formats, from a wide range of ground-based and some space-based instruments. An overview of Madrigal data can be found in material from the Boston University Incoherent Scatter Radar Summer School, publicly accessible at https://wikis.mit.edu/confluence/display/ASGScience/2022+ISR+School.

The T&E Team will augment the data collected with auxiliary sensors not included in the Madrigal database. The T&E Team will vet the quality of the auxiliary sensor data to verify contributions are appropriate to and not otherwise aberrant to Madrigal data. There will be no overlap between known tracked debris objects in the SRS and STS datasets.

The T&E Team will continuously develop SRS and STS database throughout the program, curating the database they develop to meet evaluation goals. The specifics of these collections will be determined by the T&E Team during the program.

1.D.1.2. Researcher Collections

As indicated in the previous section, the Government will provide limited datasets (SRS) to Performers for system development. Each Performer is required to plan and carry out debris dataset development efforts.

To minimize the risk of system biases, Performers must plan and implement steps to ensure that datasets include debris objects of varying size, shape, mass, density, and surface charge, and orbital regimes of varying altitude and inclination. Datasets assembled by Performers are referred to as SINTRA Researcher Collections (SRC). Performers are required to submit their most current SRC to the Government in a standard format at each evaluation milestone. Performers are permitted to withhold their internal SRC from other Performers until the end of each program milestone. At the conclusion of each milestone, the T&E Team will combine each Performer’s SRC into a unified collection and will release this collection to all Performers. Sharing of the combined SRC helps ensure that improvement in performance on evaluation tasks is attributable to a team’s superior technical approach rather than a greater accrual of development data resources.

As part of their proposal, each Offeror should provide a data collection plan and describe how their SRC will be used in R&D. Performers cannot control the quality and depth of data from other teams’ collections, but they can ensure their own collections meet or exceed their own research needs.

Performers will be responsible for annotating all SRC data for their R&D needs. All annotation data are to be delivered to the Government regardless of whether they are developed after the collection or after initial delivery of the SRC dataset. Annotation methodologies should be consistent with current research best practices to facilitate algorithm development.

All data collected, curated, annotated, synthesized, or generated by Performers created under the SINTRA program for R&D or any other purpose will be provided to the Government as Deliverables. Software tools or algorithms created for the purpose of data annotation, synthesis, or generation under the SINTRA program for R&D purposes or otherwise shall also be provided to the Government as Deliverables with Government Purpose Rights in accordance with FAR 52.227-14, to allow use by the USG. IARPA reserves the right to require delivery of incomplete or partially annotated versions of SRC datasets with 30 days’ advance notice.

1.D.1.3. External Data Sources

Performers may utilize external datasets in addition to SRS and SRC datasets. External data are data obtained by Performers that are available from third parties or that have been collected by the Performer outside of the SINTRA program. An example of a third-party dataset would be sensor data from the amateur astronomy community or a university-sponsored space-based platform that has been approved by the dataset owner for release to the research community. Data collected by a Performer under a different program are considered external data, even if the other program’s data collection was Government-sponsored.

All external datasets must be approved for use in the SINTRA program by IARPA during contract performance, in accordance with applicable policies, statutes, and regulations.

Performers may not use proprietary datasets unless these datasets are made available to all R&D Performers and the T&E Team in the SINTRA program. Public release of proprietary datasets is not a requirement;

however, release for use within the SINTRA program is required. Moreover, for any dataset not collected under the scope of the SINTRA program, Performers must provide the Government with an accounting of all resources used and sources from which data are drawn and describe how the data will be used for development, testing, and training of algorithms.

1.D.2. Evaluation Data

SINTRA will utilize distinct test data to evaluate Performer subcomponents, modules, and systems against program goals, objectives, and metrics. Each SINTRA Test Set (STS) will consist of diverse debris object physical characteristics and orbital regimes. These data are constructed to evaluate program metrics in a balanced and intentional manner. The set of known tracked debris objects appearing in this collection will not overlap with that of the SRS dataset. This ensures that no known debris objects used for system development will be included in the testing set, and that systems performance generalizes.

Intended uses of the STS evaluation datasets include both use by the T&E Team for independent evaluation of program Deliverables against target metrics and use by Performers to refine and improve their algorithms. Small portions of the evaluation datasets will be provided to Performers to enable internal T&E and exploratory error analysis by Performers and to improve the consistency and communication between Performers and the T&E Team. Additional sequestered or external datasets may be used to supplement performance evaluations at the discretion of the SINTRA PM.

1.E. Test and Evaluation (T&E)

T&E will be conducted by an independent team of Government and contractor staff carrying out evaluation and analyses of Performer research Deliverables using program test datasets and protocols. The STS data and test protocols (see 1.D.2. Evaluation Data) will be the primary mechanism by which the T&E Team carries out their evaluations.

The SINTRA Program will pursue rigorous and comprehensive independent T&E to ensure that research outcomes are well characterized, deliverables are aligned with program objectives, and that algorithm performance is measured across the full range of conditions. Such T&E activities will not only inform IARPA and Government stakeholders on SINTRA research progress but will also serve as invaluable feedback to the Performers to improve their research approaches, algorithm training practices, and system development. The SINTRA Program will work closely with Government leaders in space situational awareness (SSA) and orbital debris to continually refine and improve T&E methodologies.

Performers will have specific Deliverable Milestones at which all subcomponent and system algorithms and software will be delivered to IARPA and its designated T&E Team. The T&E Team will then conduct evaluations at the direction of the SINTRA Program Manager (PM) and with the objective of characterizing the quality, functionality, and performance of the SINTRA Deliverables. In addition to quantitative measurements, T&E will be carried out to establish a thorough understanding of the progress, status, and limitations of the Performer’s research.

T&E results and feedback will be provided to Performers at regular intervals to keep them abreast of current independent performance measurements and to inform and improve their R&D approaches and methods.

T&E results from all Performers will be shared with all teams to establish an understanding of the current state and progress of SINTRA research; T&E results will also be shared with USG external stakeholders, including their contractors, for Government purposes (see 4. A. Proposal Information). For example, a Principal Investigator (PI) Review Meeting will be held annually to share research ideas, progress, and results across the SINTRA program (see 1.H.1. Workshops).

IARPA may conduct other supplemental evaluations or measurements in its sole discretion to evaluate the Performers’ research and Deliverables.

1.F. Program Metrics

Achievement of metrics is a performance indicator under IARPA research programs. IARPA has defined SINTRA program metrics to evaluate effectiveness of the proposed solutions in achieving the stated program goal and objectives, and to determine whether satisfactory progress is being made. The metrics described in this BAA are shared with the intent to scope the effort, while affording maximum flexibility, creativity, and innovation to Offerors proposing solutions to the stated problem.

The final SINTRA T&E protocols and evaluation methodology are currently under development; further details will be provided at program kickoff. Program metrics may be refined during the various phases of the SINTRA program; if metrics change, revised metrics will be communicated in a timely manner to Performers. The evaluation methodology may be revised by the Government at any time during the program lifecycle to better meet program needs.

SINTRA program target metrics were selected based on three factors:

1. What is technically achievable but challenging based on current state-of-the-art in orbital debris detection, tracking, and characterization;

2. What is statistically measurable based on the planned program evaluation data; and

3. What is useful to mission partners based on USG stakeholder needs and use cases.

The following is a description of primary program metrics and target scores. Score targets will increase for each successive program phase. Targets scores account for orbital regimes since the detection of debris size is a function of distance for existing capabilities.

SINTRA performance metrics are designed to evaluate the software components deliverables for each of the two program tasks areas: debris detection, tracking, and characterization (TA1), persistent monitoring of the debris population (TA2). There may exist other metrics that have not been contemplated by IARPA.

It is acceptable for Offerors to propose alternative approaches in addition to the required program metrics, so long as such approaches are described in their proposal and are of a similarly challenging nature to those proposed in this BAA. Such alternate approaches will be evaluated in tandem with proposed techniques.

A summary of metric targets by Phase is shown in Table 1; these metrics are subject to change over the course of the program. Accuracy targets are defined as aggregated statistical measures for a given phases’ STS evaluation datasets, which contain a diverse set of debris physical characteristics and orbital regimes.

These targets are estimates informed by current capabilities and use case requirements. To mitigate risk and better inform the Government regarding the appropriateness of phase-over-phase performance targets, the T&E Team will compare performance against baseline systems for each task approximating the current state-of-the-art. Further information about baseline models will be furnished to Performers at program kickoff.

SINTRA Program Target Metrics and associated units are defined as follows:

Debris detection (h): Time to detect new debris after a debris-generating event.

Detection rate (%): Percentage of positive detections.

Coverage (sr/km): The coverage metric is given in steradians per kilometer. A steradian (sr) is a solid angle unit. The surface area of a sphere is 4 pi steradians. Each orbital altitude, given in kilometers, can be represented by a sphere with a surface area of 4 pi steradians.

Revisit rate (h): Time to revisit a previously detected debris object.

False alarm rate (%): Percentage of false positive detections.

Debris cloud: Multiple debris objects. Offerors will determine the threshold at which to define and track debris clouds instead of individual debris objects.

Space sensor size (cm3): Maximum size for a new space-based sensor.

Space sensor power (W): Maximum power for a new space-based sensor.

Table 1: SINTRA Program Target Metrics

Phase 1 Phase 2

Task Area 1.A

(TA1.A):

Task Area 1.B

(TA1.B):

Task Area 2.A

(TA2.A):

Task Area 2.B

(TA2.B):

Debris detection, tracking, and characterization

Debris detection, tracking, and characterization

Persistent debris population monitoring

Persistent debris population monitoring

Range +/- Range +/- Range +/- Range +/-

Debris diameter (cm)

10-40 0.25 0.1-40 0.05 10-40 0.25 0.1-40 0.05

Phase 1 Phase 2

Task Area 1.A

(TA1.A):

Task Area 1.B

(TA1.B):

Task Area 2.A

(TA2.A):

Task Area 2.B

(TA2.B):

Debris detection, tracking, and characterization

Debris detection, tracking, and characterization

Persistent debris population monitoring

Persistent debris population monitoring

Range +/- Range +/- Range +/- Range +/-

Debris/cloud speed (km/s)

1.4-10.2 0.25 1.4-10.2 0.1 1.4-10.2 0.1 1.4-10.2 0.1

Debris/cloud position (km)

200- 4,000

5 (LEO)

200-

40,000

5 (LEO)

50 (GEO)

200-4,000 2 (LEO)

200- 40,000

2 (LEO)

10 (GEO)

Debris particle density (kg/m3)

0 - 22000 2000 0 - 22000 1000 0 - 22000 1000 0 - 22000 1000

Debris detection (h)

< 168 8 < 60 5 < 2 0.5 < 0.083 0.0167

Detection rate > 70% 5% > 80% 5% > 80% 5% > 95% 2%

False alarm rate

< 10% 2% < 5% 1% < 5% 1% < 2% 0.5%

Coverage (sr/km)

> 1 pi 0.5 pi > 2 pi 0.5 pi > 3 pi 0.5 pi > 4 pi 0.25 pi

Revisit rate (h) < 168 8 < 48 5 < 2 0.5 < 0.083 0.0167

Space Sensor Size (cm3)

N/A N/A < 700 50 N/A N/A < 500 50

Space Sensor Power (W)

N/A N/A < 150 10 N/A N/A < 50 5

1.G. Program Waypoints, Milestones, and Deliverables

Waypoints, Milestones, and Deliverables are established from the program’s onset to ensure alignment with SINTRA objectives, organize research activities in a logical and reportable manner, and facilitate consistent and efficient communication among all stakeholders – IARPA, the SINTRA T&E Team, USG Stakeholders, and Research Performers.

1.G.1. Program Waypoints

Waypoints are the means by which the Performer clearly demonstrates the quantitative and timely progress that must be made for the overall concept to meet end-of-phase Milestones. In other words, the intent of Waypoints is to provide a clear measure of progress towards meeting the program Milestones so the PM and advisors can provide more effective guidance and assistance to the Performers. Performance against these Waypoints will be reviewed throughout the program, and the PM and advisors will use performance against the Waypoints to assess whether course corrections are needed to ensure program success.

The Government has defined Waypoints for all Performers that are included in the Program Milestones and Deliverables (see 1.G.2. Program Milestone, Waypoint, and Deliverables Timeline). Progress against these Waypoints will be reviewed during site visits.

1.G.2. Program Milestone, Waypoint, and Deliverables Timeline

Table 2 and Table 3 show a timeline for the program with defined Milestones, Waypoints and Deliverables.

The Offeror may add other Waypoints in addition to the minimum set listed in the table.

Table 2: Program Phase 1 Milestone, Waypoint and Deliverables Timeline

Phase Month Event Description Comments Deliverables

1-2 All Waypoint Monthly

Status Report

(MSR)

Due on 10th of each month; Technical and cost

MSR

1-2 All Waypoint Progress and

Status Meeting Bi-weekly teleconference with SINTRA PM N/A

1 1 Waypoint SRS1 datasets Provided as GFI N/A

1 1 Waypoint Kick-off Meeting

Location TBD; information about GFI/GFE furnished to Performers; T&E presents guidelines performance evaluation

N/A

1 2 Waypoint Site Visit At Performer site N/A 1 8 Waypoint Site Visit At Performer site N/A

1 9 Deliverable System submission for Milestone 1.1

All subcomponents and systems

TA1 System; SRC; Software should be delivered at the beginning of evaluation.

Updated software may be requested if deficiencies are identified during evaluation

1 10-11 Milestone Evaluation

Milestone 1.1 T&E reports scores to

Performers N/A

1 12 Deliverable

Technical report describing methods for evaluating unknown debris

Delivered by Performers to T&E Team and other

Performers Document

1 14 Waypoint Site Visit At Performer site N/A

1 15 Deliverable System submission for Milestone 1.2

All subcomponents and systems

TA1 System; SRC (updated);

Software should be delivered at the beginning of evaluation.

Updated software may be requested if deficiencies are identified during evaluation

1 16-17 Milestone Evaluation

Milestone 1.2 T&E reports scores to

Performers N/A

1 18 Waypoint

Review of combined framework for evaluating unknown debris

T&E Team presents tentative evaluation framework to Performers;

Performers present work on evaluating unknown debris

Meeting

1 20 Waypoint Site Visit At Performer site N/A

Phase Month Event Description Comments Deliverables

1 21 Deliverable System submission for Milestone 1.3

All subcomponents and systems

TA1 System; SRC (updated);

Software should be delivered at the beginning of evaluation.

Updated software may be requested if deficiencies…