HR001121S0026-Amendment-01.pdf

HR001121S0026 QUANTUM BENCHMARKING 1

Broad Agency Announcement Quantum Benchmarking Defense Sciences Office

HR001121S0026

Amendment 1

April 1, 2021

HR001121S0026 QUANTUM BENCHMARKING 2

Table of Contents I. Funding Opportunity Description

A. Introduction B. Background C. Program Description/Scope D. Program Structure E. Technical Area Descriptions F. Schedule/Milestones G. Deliverables H. Government-furnished Property/Equipment/Information I. Other Program Objectives and Considerations

II. Award Information A. General Award Information B. Fundamental Research

III. Eligibility Information A. Eligible Applicants B. Organizational Conflicts of Interest C. Cost Sharing/Matching D. Ability to Receive Awards in Multiple Technical Areas - Conflicts of Interest

IV. Application and Submission Information A. Address to Request Application Package B. Content and Form of Application Submission C. Submission Dates and Times D. Funding Restrictions E. Other Submission Requirements

V. Application Review Information A. Evaluation Criteria B. Review and Selection Process C. Federal Awardee Performance and Integrity Information (FAPIIS)

VI. Award Administration Information A. Selection Notices B. Administrative and National Policy Requirements C. Reporting

VII. Agency Contacts VIII. Other Information

A. Proposers Day B. Frequently Asked Questions (FAQs) C. Collaborative Efforts/Teaming

BAA Attachments:

Attachment A: ABSTRACT SUMMARY SLIDE TEMPLATE Attachment B: ABSTRACT TEMPLATE Attachment C: PROPOSAL SUMMARY SLIDE TEMPLATE Attachment D: PROPOSAL TEMPLATE VOLUME 1: TECHNICAL & MANAGEMENT Attachment E: PROPOSAL TEMPLATE VOLUME 2: COST Attachment F: MS ExcelTM DARPA COST PROPOSAL SPREADSHEET Attachment G: PROPOSAL TEMPLATE VOLUME 3: ADMINISTRATIVE & NATIONAL POLICY REQUIREMENTS

HR001121S0026 QUANTUM BENCHMARKING 3

PART I: OVERVIEW INFORMATION

Federal Agency Name: Defense Advanced Research Projects Agency (DARPA), Defense Sciences Office (DSO)

Funding Opportunity Title: Quantum Benchmarking

Announcement Type: Amendment 1

Funding Opportunity Number: HR001121S0026

Catalog of Federal Domestic Assistance (CFDA) Number(s): 12.910 Research and Technology Development

Dates (All times listed herein are Eastern Time.)

o Posting Date: April 1, 2021 o Proposers Day: April 20, 2021. See Section VIII.A.

o Abstract Due Date: May 11, 2021, 4:00 p.m.

o FAQ Submission Deadline: June 8, 2021, 4:00 p.m. See Section VIII.B.

o Full Proposal Due Date: June 29, 2021, 4:00 p.m.

Anticipated Individual Awards: DARPA anticipates multiple awards in both Technical Area 1 (TA1) and Technical Area 2 (TA2).

Anticipated Funding Available for Award: To maximize the diversity of approaches considered using available resources, DARPA is limiting funding for TA2 awards to $1,450,000 for the entire 18 months of Phase 1 and $1,500,000 for the entire 18 months of Phase 2. Funding guidance is not provided for TA1.

Types of Instruments that May be Awarded: Procurement contracts, cooperative agreements, or Other Transactions. Award instruments will be limited to procurement contracts or Other Transactions for Proposers whose proposed solution includes Controlled Unclassified Information (CUI).

Agency contacts o Technical POC: Joseph Altepeter, Program Manager, DARPA/DSO o BAA Email: QuantumBenchmarking@darpa.mil o BAA Mailing Address:

DARPA/DSO

ATTN: HR001121S0026

675 North Randolph Street Arlington, VA 22203-2114 o DARPA/DSO Opportunities Website: http://www.darpa.mil/work-with-us/opportunities

Teaming Information: See Section VIII.C for information on teaming opportunities.

mailto:QuantumBenchmarking@darpa.mil https://www.darpa.mil/work-with-us/opportunities?oFilter=DSO https://www.darpa.mil/work-with-us/opportunities?oFilter=DSO

HR001121S0026 QUANTUM BENCHMARKING 4

Frequently Asked Questions (FAQ): FAQs for this solicitation may be viewed on the DARPA/DSO Opportunities Website. See Section VIII.B for further information.

Security: Quantum Benchmarking is an unclassified program. It is not anticipated that work in this program will generate controlled unclassified information (CUI).

HR001121S0026 QUANTUM BENCHMARKING 5

PART II: FULL TEXT OF ANNOUNCEMENT

I. Funding Opportunity Description

This Broad Agency Announcement (BAA) constitutes a public notice of a competitive funding opportunity as described in Federal Acquisition Regulation (FAR) 6.102(d)(2) and 35.016 as well as 2 C.F.R. § 200.203. Any resultant negotiations and/or awards will follow all laws and regulations applicable to the specific award instrument(s) available under this BAA, e.g., FAR

15.4 for procurement contracts.

A. Introduction

The Defense Sciences Office (DSO) at the Defense Advanced Research Projects Agency (DARPA) is soliciting innovative research proposals in the area of quantum benchmarking.

Proposed research should quantify the long-term utility of quantum computers. In particular, proposed research should center around either (1) the creation of application-specific, hardware-agnostic benchmarks for quantum computer utility or (2) hardware resource estimation for quantum computers. Specifically excluded is research that primarily results in evolutionary improvements to the existing state of practice.

B. Background

It has been credibly hypothesized that quantum computers will revolutionize multiple scientific and technical fields within the next few decades; examples include machine learning, quantum chemistry, materials discovery, molecular simulation, many-body physics, classification, nonlinear dynamics, supply chain optimization, drug discovery, battery catalysis, genomic analysis, fluid dynamics, protein structure prediction. For many of these examples, like quantum chemistry and protein structure prediction, quantum computers are hypothesized to be useful simulators because the target problem is inherently quantum mechanical. Other examples, like classification and nonlinear dynamics, center around problems that have nothing to do with quantum systems, but involve combinatorial complexity that is intractable for conventional computers.

For each of the fields listed above, it is unclear exactly what size, quality, and configuration of quantum computer – if any – will enable the hypothesized revolutionary advances. This lack of clarity may be the result of one or more of the following factors:

Where only the technical field has been identified, the specific application instances1 that would be solved by a hypothetical quantum computer – at specific scales, with specific identified values for key parameters, and with clearly identified impact if successful – have not been posed.

1 “Application instance” is defined below in Section II.F; it refers to a particular computational problem to be solved within a specific application domain, e.g., simulating the ground state energy of one particular molecule. An “application domain” is defined as defined as a particular class of applications for which quantum computers may be useful, e.g., molecular simulation or materials discovery.

HR001121S0026 QUANTUM BENCHMARKING 6

Where application instances have been posed, the new core computational capability that would enable success is not understood. This often contributes to a lack of understanding about the gap between existing classical, state-of-the-art solutions and hypothesized quantum solutions.

The appropriate metrics and testing procedures for quantifying progress towards critical new quantum computing capabilities are not known. This is especially problematic for problems where the testing procedures themselves may be classically intractable.

Where benchmarks for quantum utility have been proposed, they are often distilled down to a single parameter that gives limited insight into the ability of a system under test to succeed at specific application instances. In almost all cases, it is not known how to measure hardware progress towards a specific application at a specific scale, especially using robust multi-dimensional metrics suitable for driving research and development into special-purpose hardware.

The full-system-hardware resources required to solve particular problems at specific scales have not been estimated. This is particularly true where large, fault-tolerant quantum computers are expected to be required. When quantum hardware resources have been estimated, only the exponential scaling term(s) have been quantified and not the constant and polynomial scaling terms. The ancillary classical resources and low-level hardware configurations (e.g., connectivity requirements) that are required are either unaddressed or cursorily addressed.

In the past two years, two different groups have claimed to have achieved “quantum supremacy”

– the ability to repeatably perform a computation that is unrealistic for classical systems to replicate. In addition, multiple commercial companies have published roadmaps showing that they will create universal, fault-tolerant quantum computers in the next decade. The extent to which these roadmaps, if realized, will represent significant and important new computational capabilities is not currently understood.

C. Program Description/Scope

Program Description The Quantum Benchmarking program will create new benchmarks that quantitatively measure progress towards specific, transformational computational challenges. In parallel, the program will estimate the hardware-specific resources required to achieve different levels of benchmark performance.

The benchmarks will be hardware agnostic. This is essential for benchmarks that are focused on measuring utility, since a novel classical solution to an urgent problem is just as valuable as a novel quantum solution to an urgent problem. However, work in this program will focus on creating hardware-agnostic benchmarks for problems where quantum approaches are most likely to be needed.

The Quantum Benchmarking program will quantify the long-term utility of quantum computers by solving a series of hard problems:

HR001121S0026 QUANTUM BENCHMARKING 7

Compiling a list of specific utility-driven application instances from a variety of application domains

Grouping these application instances according to common core enabling computational capabilities

Developing novel test procedures for quantifying progress towards these core enabling computational capabilities

Using all of the above to create scalable, robust multi-dimensional benchmarks that can act as guidestars for research and development aimed at long-term, real-world utility for a variety of application domains

Creating tools for estimating the primary quantum hardware resources and ancillary classical hardware resources needed to achieve a specific level of benchmark performance

Each of these program goals is described in more detail below.

Compiling a list of application instances. The Quantum Benchmarking program will compile a list of specific application instances from across as many application domains as possible.

These application instances are the answers to the question: “If you had a large, perfect quantum computer today, what would you ask it to do?” Each application instance will be a specific problem at a specific scale. For example, one application instance could be estimating the ground state energy of a particular molecule in a particular configuration.

Grouping application instances. The Quantum Benchmarking program will group application instances according to their core enabling computational capabilities. Because the primary goal of the program is to estimate the long-term utility of quantum computers, it is crucial to uncover the core enabling computational capabilities for the application instances that have been compiled. After grouping instances, performers will determine the key metrics that can be used to quantify these core enabling computational capabilities, e.g., the precision or accuracy of a specific class of matrix operation.

Developing test procedures. The Quantum Benchmarking program will discover novel methods for testing and predicting performance against the key metrics that quantify core enabling computational capabilities. The Quantum Benchmarking program recognizes that some metrics for measuring quantum computational capability may not be testable using finite classical resources. Instead, a new type of quantum device, referred to here as a quantum benchmarking testbed (QBT), may be needed to test certain metrics. A QBT would act as a synthetic problem with tunable size, complexity, and key parameters and serve as a simulation target. If a quantum computer under test can correctly simulate the behavior of the QBT, it passes that benchmarking challenge. Note that if all key metrics associated with a particular grouping of application instances can be tested using existing or realizable classical resources, then those classical resources may be the best means for testing progress toward realizing the core enabling computational capabilities.

Creating benchmarks. The Quantum Benchmarking program will create benchmarks that can act as guidestars for research and development into quantum computation. More specifically, the Quantum Benchmarking program will create scalable and predictive benchmarks that can not only make it clear when a particular performance threshold has been reached, but also quantify

HR001121S0026 QUANTUM BENCHMARKING 8

progress towards important performance thresholds. The Quantum Benchmarking program will create robust and multi-dimensional benchmarks that embrace problem complexity by having many input parameters to define problem scope and scale, and by having even more output parameters that provide rich debugging information about not just if a system under test succeeded or failed but exactly how it succeeded or failed along as many relevant axes as possible. If successful, the Quantum Benchmarking program will provide benchmarks that provide rich debugging information for quantum computer developers.

Estimating hardware resources. The Quantum Benchmarking program will create tools for estimating the computational-paradigm-specific hardware resources needed to achieve specific benchmark performance thresholds. Existing estimates of resource scaling with problem size are often limited to the leading term in an asymptotic expansion of problem complexity. In some cases, quantum computers are assumed to be useful if quantum advantage scales exponentially with problem size. Of course, if the constant and polynomial scaling terms outweigh the exponential scaling terms for problem sizes of interest, quantum advantage may not exist. The Quantum Benchmarking program will provide estimates of these additional scaling terms by predicting not just the quantum resources needed to achieve a new computational capability but the ancillary classical resources (for example, from decoders and schedulers) needed to support the proposed quantum system. Estimates will necessarily be tied to a particular quantum computing technology, e.g., superconducting quantum computers or photonic quantum computers, because the hardware resources being estimated will vary dramatically with different hardware paradigms.

Program Scope In scope. The following proposal elements are explicitly in-scope for the Quantum Benchmarking program:

Analysis of applications that require large-scale, universal, fault-tolerant quantum computers to solve

Estimates of the classical and quantum resources required to execute quantum algorithms on large-scale, universal, fault-tolerant quantum computers

Applications of fault tolerance and error correction that are relevant to the benchmarks under study

Nontraditional quantum computing paradigms Out of scope. The following proposal elements are explicitly out-of-scope for the Quantum Benchmarking program:

Approaches that focus exclusively on noisy intermediate-scale quantum computers

Approaches to cryptanalysis and/or Shor’s algorithm

Development of new quantum computing hardware

D. Program Structure

Technical Areas (TAs)

HR001121S0026 QUANTUM BENCHMARKING 9

Performer teams for the Quantum Benchmarking program are divided into hardware-agnostic teams (TA1) and hardware-specific teams (TA2). A performer may propose to both TAs if they wish, but this requires submitting two separate proposals. A single proposal must address only one of the two TAs.

TA1: Hardware-agnostic benchmark creation. TA1 will comprise interdisciplinary performer teams that combine application-domain experts with hardware-agnostic quantum computing experts. TA1 teams will lead the benchmark creation process and quantify computational utility as a function of benchmark performance.

TA2: Hardware-specific resource estimation. TA2 will comprise teams focusing on one or more hardware-specific quantum computing paradigms. Each team will determine the hardware-paradigm-specific resources – both classical and quantum – required to achieve a given benchmark performance level.

Government Test and Evaluation (T&E) General T&E. As part of the Quantum Benchmarking program, the Government will create a robust Test & Evaluation effort designed to quantify and qualify progress in both TA1 and TA2.

In particular, the T&E team will evaluate tools for both utility estimation and resource estimation. This BAA is not soliciting proposals for T&E.

QBT Prototype Development Team. A specific subset of the T&E team will be funded to evaluate efforts to design quantum benchmarking testbeds, or QBTs. It is an open question if QBTs will be a necessary tool for benchmarking, and if they are, what different QBT specifications will be required for different benchmarks. The QBT Prototype Development Team will create and test hardware prototypes relevant to basic research into QBTs, in order to test and evaluate performer designs as applicable. This BAA is not soliciting proposals for T&E, including proposals for the QBT Prototype Development Team.

Program Phases The Quantum Benchmarking program will be structured into two phases, each 18 months long.

Proposals should structure Phase 1 as the base effort, with Phase 2 included as an option.

Participation in Phase 1 does not guarantee funding in Phase 2. Progression to Phase 2 is based on the availability of funds and on DARPA’s estimate of the overall value to the Government from each team’s continued participation.

To maximize the diversity of approaches considered using available resources, DARPA is limiting funding for TA2 awards to $1,450,000 for the entire 18 months of Phase 1 and $1,500,000 for the entire 18 months of Phase 2. Funding guidance is not provided for TA1.

Phase 1: Benchmark selection and tool development In Phase 1, teams will propose utility-driven benchmarks to be studied in Phase 2, while at the same time developing the utility-estimation and resource-estimation tools that will be required to succeed in Phase 2.

TA1 teams will lead the benchmark selection process by compiling application instances, grouping them by candidate metrics for core enabling computational capabilities, and proposing candidate benchmarks. In parallel, they will develop utility estimation tools and investigate methods for scalable test procedures, possibly including the use of QBTs.

HR001121S0026 QUANTUM BENCHMARKING 10

TA2 teams will assist in the benchmark selection process by compiling application instances with specific emphasis applicable to their chosen hardware paradigm. In parallel, they will develop tools for estimating the hardware resources required to reach specific performance thresholds.

Phase 2: Benchmark creation and resource estimation In Phase 2, the Government will select specific candidate benchmarks for detailed study.

TA1 teams will expand the Phase 2 benchmarks to include test procedures that provide multi-dimensional output data, making the benchmarks applicable to a broad range of input problems and all scales, and making the benchmarks capable of quantifying progress towards critical performance thresholds. In parallel, teams will use their utility estimation tools to quantify utility as a function of benchmark performance.

TA2 teams will improve their hardware resource estimation tools and use those tools to estimate the required resources for a variety of benchmark performance levels.

E. Technical Area Descriptions

Technical Area 1: Hardware-agnostic benchmark creation In order to understand the long-term utility of quantum computers for specific computational tasks, it is necessary to quantify the gap between proposed quantum approaches and the existing best classical approach. This requires a collaboration between multiple distinct communities:

quantum computing experts who understand the former and application-domain experts who understand the latter. Quantum experts are required to ensure that the benchmarks that are proposed do not implicitly assume classical limitations and approaches; application-domain experts are required to assess the real utility of a new computational capability on specific unsolved application instances.

TA1 proposals should:

Explicitly propose at least four application domains2 that their team will specialize in.

Clearly articulate why their proposed areas of application-domain expertise contain application instances3, which, if solved satisfactorily using a novel computational approach, would result in transformational utility for Government and/or commercial applications.

Include personnel who have application-domain expertise in state-of-the-art classical approaches in each of the proposed application domains.

2 An “application domain” is defined as a particular class of applications for which quantum computers may be useful, e.g., molecular simulation or materials discovery.

3 “Application instance” is defined in Section II.F; it refers to a particular computational problem to be solved within a specific application domain, e.g., simulating the ground state energy of one particular molecule.

HR001121S0026 QUANTUM BENCHMARKING 11

Include personnel with expertise in quantum computation, especially in the application of quantum algorithms to a wide variety of application domains.

Propose a compelling approach to compiling application instances; clearly articulate why the approach will produce many distinct and specific application instances, which, if solved, would result in transformational utility.

Propose a compelling approach to grouping application instances by core enabling computational capability.

Offer an innovative approach for testing the metrics associated with core enabling mathematical operations, especially when these tests may themselves be classically intractable.

Propose innovative methods for quantifying utility as a function of achieved benchmark performance.

Propose to develop all program materials and research as publicly-releasable material open to robust scrutiny from the academic and commercial community both during and after the program; proprietary or restricted research is not appropriate for TA1.

TA1 proposals should NOT:

Propose mathematically interesting application domains with little or no quantifiable utility to Government and/or commercial applications.

Propose application domains where existing classical solutions are entirely satisfactory or where the proposal cannot quantify the application scale at which existing classical solutions become unsatisfactory.

Propose research related to cryptographic application domains or application instances, including any applications of factoring or Shor’s algorithm.

Technical Area 2: Hardware-specific resource estimation Performers in TA2 are required to quantify the hardware-paradigm-specific quantum and classical resources to solve program-developed problems at scales of interest. In addition, TA2 teams will calculate the resources required for their proposed hardware paradigm(s) to act as suitable QBTs, if the Phase 2 benchmarks selected by the Government require QBTs for effective testing.

TA2 proposals should:

Propose one or more specific quantum computing hardware paradigms that their team will specialize in.

Include personnel with expertise in one or more specific quantum computing hardware paradigm(s).

Propose a compelling approach to compiling application instances relevant to their chosen hardware paradigm(s); clearly articulate why the approach will produce several distinct and specific application instances, which, if solved, would result in transformational utility.

HR001121S0026 QUANTUM BENCHMARKING 12

Propose innovative methods for calculating both quantum and classical hardware resource requirements for multiple problem scales and target metrics.

Offer innovative approaches for characterizing the hardware requirements and configurations needed to achieve specific levels of benchmark performance; these should include resources for classical control, scheduling, etc., in addition to the quantum resources.

Offer innovative approaches to develop automated and scalable tools for estimating resource requirements, as successful TA2 teams will be required to provide estimates of resources for dozens of distinct, specific problem instances.

Offer innovative methods for calculating how resource requirements scale, not just in the exponential limit, but also including polynomial and constant scaling terms.

Clearly articulate the approach to justifying hardware resource estimates. Ideally, the approach will be in a manner that will allow a quantum computing expert who is not familiar with the team’s code to verify and validate the team’s claims.

TA2 proposals should NOT:

Propose to develop quantum computing hardware as part of this effort.

F. Schedule/Milestones

All proposals should support the following list of program milestones and the associated program schedules for both TAs. Performers are encouraged to include in their proposals additional milestones and an expanded schedule that supports their specific proposed course of research, but all efforts should support the program-wide schedule laid out below.

Throughout the course of this basic research effort, program-wide and performer-specific milestones and metrics are expected to be updated as progress is made, but the following structure should be used to estimate costs and other programmatic dependencies for proposing teams.

Definitions of Phase 1 Milestone Terms The terms below are grouped by their relevance to the program goals described in Section II.C above.

Application instance. A specific example of one problem to be solved with a quantum computer. Examples include finding the ground state energy of one particular molecule or optimizing one particular logistics problem. The Quantum Benchmarking program will compile a large list of specific application instances from a variety of application domains. Teams are encouraged to take these instances from the vast existing literature on the potential uses of quantum computers as well as the literature describing the state of the art in other application domains. Teams are also encouraged to collaborate with all other teams on the creation of lists of application instances. Note that not all proposed application instances will be carried forward into Phase 2 of the program.

Instance group. A group of instances that share the same core enabling computations – the critical mathematical operations that underpin a successful solution to the application instances.

For example, a specific class of matrix operation might enable a wide variety of disparate

HR001121S0026 QUANTUM BENCHMARKING 13

application instances. One of the primary goals of Phase 1 is to create these instance groups, as the identification of common core enabling computations is the first step to creating useful, general benchmarks that quantify a candidate quantum computer’s utility for a particular type of task. The following sub-definitions are relevant to an instance group.

Candidate metric. A measure that quantifies the ability of a computer to successfully perform the core enabling computation for an instance group. Some candidate metrics might apply to all instance groups (like time to solution) while others will be instance-group-specific (like the precision or accuracy of a specific class of matrix operation).

Utility threshold. Specific values of a candidate metric or metrics that correspond to the performance needed to solve a particular application instance. A computer capable of exceeding these thresholds should therefore be quantifiably more useful than existing state-of-the-art solutions to the same problem. Note that passing one utility threshold will likely require minimum performance levels on multiple candidate metrics.

Utility estimation tools. Performer-developed tools for estimating the impact of passing a given utility threshold. These will be very application-domain-specific.

Utility estimate. The quantitative benefit to a user if a utility threshold is surpassed. The default unit of utility should be “dollars” and should be compared directly against the best state-of-the-art solution to the application instance.

Testing procedure. A procedure for testing all of the metrics associated with a particular benchmark. Note that metrics that are not testable are not useful as benchmarks. Moreover, testing some metrics may be computationally intractable for classical systems due to the exponential scaling in degrees of freedom present in some problems. However, it is also possible that some classically intractable testing procedures may be implementable using specially designed quantum systems capable of generating challenge problems for the computer under test. This type of system is referred to in this BAA as “quantum benchmarking testbed,” and is defined below.

Quantum benchmarking testbed (QBT) (As applicable). A scalable, multi-dimensional challenge problem generator that may be needed to provide intermediate computational targets toward a candidate metric or metrics. Teams are not required to produce such testbeds experimentally, but they are required to submit notional specifications for QBTs if their proposed testing procedures require them.

Candidate benchmark. The combination of all of the following: a candidate metric or metrics, a list of constituent application instances and associated utility thresholds, and a proposed testing procedure for measuring performance against the candidate metric or metrics (this testing procedure could require a quantum benchmarking testbed, but not necessarily). The quantum benchmarking program is seeking to create benchmarks that are useful as guidestars for research and development and that can provide robust, detailed debugging information. Good candidate benchmarks should have many “inputs” that can be used to parameterize the problem space for which a benchmark measures computational ability. Good candidate benchmarks likewise have many “outputs” that provide rich debugging information about a system under test. Key parameters relevant to candidate benchmarks are defined below.

Number of benchmark inputs. The number of distinct degrees of freedom used to parameterize the problem space under test. Each of these inputs should be an

HR001121S0026 QUANTUM BENCHMARKING 14

independent variable that affects the testing procedure to be performed.

Number of benchmark primary outputs. The number of distinct degrees of freedom that describe the output of a benchmark. These outputs should allow a user to determine exactly which utility thresholds were surpassed by a computer under test.

Resource estimation tools. Performer-developed tools for estimating the hardware-specific resources needed to achieve a given utility threshold for a given metric or metrics. These tools will be hardware-paradigm-specific. The simplest tools will be first-order estimates only, of the type that can show an exponential quantum advantage over classical tools in the limit of infinite problem size. Intermediate versions should include additional terms, for example capturing polynomial scaling of resources. By the end of the program, estimates should include all scaling terms, including constant terms. The following sub-definition is the output of a resource estimation tool.

Resource estimate. The hardware resources needed to achieve a given utility threshold for a given metric or metrics. The estimate should not only include quantum resources but also ancillary classical resources needed for control, decoding, and scheduling. Resource estimates should address not only the number of resources required but also their required specifications and configuration. A key parameter for the resource estimate – the number of output parameters – is defined below.

Number of Output Parameters. The number of types of resources tracked by a particular resource estimate. The types of resources tracked should include the number of each type of primitive hardware element that is needed in a particular configuration or role, the amount and type of required ancillary classical computing resources, the energy to solution, the time to solution, etc.

Schedule of Phase 1 Milestones

Program Month Technical Area 1 Technical Area 2

3 3+ Application Instances

1+ Candidate Metric

Report on strategy for quantifying utility, including specifying units for utility

3+ Application Instances to be used as test cases for developing resource estimation tools

1+ Candidate Metric

Report on strategy for estimating hardware resources as a function of problem scale, including constant, polynomial, and exponential terms.

6 50+ Application Instances (Teams are encouraged to collaborate with other teams from both TAs; overlap between team lists is expected.)

20+ Application Instances (Teams are encouraged to collaborate with other teams from both TAs. Teams in TA2 should either choose TA1-proposed instances or propose their own distinct instances; in either case TA2-

HR001121S0026 QUANTUM BENCHMARKING 15

proposed instances should be suitable for their specific hardware paradigm.)

9 3+ Utility Estimates (of 3+ Application Instances)

1+ Initial Testing Procedure (of 1+ Candidate Metric)

3+ Resource Estimates (of 3+ Application Instances)

12 5+ initial Instance Groups (for 50+ Application Instances) o Associated Candidate Metrics o Associated Utility Thresholds

(Teams are encouraged to collaborate with other teams from both TAs;

overlap between team lists is expected.)

10+ proposed Resource Estimate Output Parameters

15 Initial Utility Estimates (for mo. 12 Instance Groups and Candidate Metrics)

Initial Test Procedures (for mo. 12 Instance Groups) o Notional QBT designs, if required

10+ Resource Estimates (for mo. 6 Application Instances) o Include at least polynomial scaling terms

18 5+ Candidate Benchmarks o Calculation of the computational complexity of each associated Test Procedure

20+ Hardware Resource Estimates (for TA1 mo. 12 Utility Thresholds)

Definitions of Phase 2 Milestone Terms Phase 2 benchmark. The Candidate Benchmarks from the Phase 1 Month 18 milestones that DARPA will select to carry forward into phase 2 of Quantum Benchmarking. Each phase 2 benchmark will be developed into one final benchmark.

Final benchmark. Final output of the program includes improved and expanded metrics, more robust list of constituent application instances and utility thresholds, and a more robust testing procedure that is both scalable and verbose. The following metrics and sub-definitions are relevant to the final benchmarks.

Associated metrics Application instances, impact thresholds, benchmark inputs, benchmark primary outputs, testing procedure, and QBT (see above).

Associated metrics Number of benchmark ancillary outputs. The number of distinct degrees of freedom that describe the output of a benchmark. These outputs should not directly determine which utility thresholds were surpassed by a computer under test but rather provide valuable ancillary information about the

HR001121S0026 QUANTUM BENCHMARKING 16

operation of the computer under test that can be used for either debugging a faulty system or understanding the limitations of a particular system.

Definition of a “Scalable” benchmark A scalable benchmark shows progress towards a utility threshold, not just a test of when the threshold has been passed.

Scalable benchmarks are predictive tools that can be used as research guidestars, not just verifiers.

Definition of a “Verbose” benchmark Here “verbose” is used in the context of a computer debugger; a “verbose” debugging output mode provides rich, detailed information about a computation’s operation and context, not just information about the computation’s success or failure.

Benchmark waypoints. Specific values of a candidate metric or metrics that correspond to the performance needed to make substantive progress towards a solution to a particular application instance. Benchmark waypoints are very similar to a utility threshold but are used to measure partial progress towards a solution. These waypoints are needed to provide specific test cases for TA2 resource estimation techniques used in the near-term and intermediate-term before a utility threshold is actually achieved.

Resource estimation tools, resource estimate, utility estimation tools, utility estimate, testing procedure. See Phase 1 definition.

Schedule of Phase 2 Milestones

Program Month Technical Area 1 Technical Area 2

24 Per Phase 2 Benchmark:

Identify 10+ Problem Instances o Associated Utility Thresholds

Per Phase 2 Benchmark:

Appropriate Benchmark Waypoints

27 Per Phase 2 Benchmark:

Preliminary scalable and verbose benchmark improvements, including:

o 10+ inputs o 10+ primary outputs, 10+ ancillary outputs

Intermediate Testing Procedures

Improved Resource Estimation Tools including:

o 50+ Output Parameters o All scaling terms estimated, including constant terms

30 Per Phase 2 Benchmark:

Improvement to 20+ ancillary outputs

No milestone

33 Deliver a complete example of each of the Final Benchmarks

Resource Estimates for all benchmark waypoints

HR001121S0026 QUANTUM BENCHMARKING 17

o 50+ ancillary outputs

Utility Estimates for all Application Instances for all Final Benchmarks

Final Testing Procedures for all Final Benchmarks

36 Final Benchmark delivery o 100+ ancillary outputs

Final Resource Estimates o 100+ output parameters

Summary Tables for Milestones and Metrics The following table summarizes the schedule and milestones for both phases of the program.

Additional Information Related to Schedule and Milestones

Proposers should provide a technical and programmatic strategy that conforms to the entire program schedule and presents an aggressive plan to fully address all program goals, metrics, milestones, and deliverables.

The task structure must be consistent across the proposed schedule, Statement of Work, and cost volume.

A target start date of December 2021 may be assumed for planning purposes.

Schedules will be synchronized across performers, as required, and monitored/revised as necessary throughout the program.

HR001121S0026 QUANTUM BENCHMARKING 18

All proposals must include the following meetings and travel in the proposed schedule and costs:

o To continue integration and development between TAs, foster collaboration between teams and disseminate program developments, a two-day Principal Investigator (PI) meeting will be held approximately every six months with locations split between the East and West Coasts of the United States. For budgeting purposes, plan for seven two-day meetings over the course of 36 months: four meetings in the Washington, D.C. area and three meetings in the San Francisco, CA area.

o Regular teleconference meetings will be scheduled with the Government team for progress reporting as well as problem identification and mitigation. Proposers should anticipate at least one site visit per phase by the DARPA Program Manager during which they will have the opportunity to demonstrate progress towards agreed-upon milestones.

G. Deliverables

Performer(s) will be expected to provide at a minimum the following deliverables:

Comprehensive quarterly technical reports due within ten days of the end of the given quarter, describing progress made on the specific milestones as laid out in the SOW, including those milestones listed in Section F.

A phase completion report submitted within 30 days of the end of each phase, summarizing the research done.

Source code and other appropriate media for the utility estimation tools developed by TA1 over the course of the program. Source code should be well-documented and follow industry best practices for readability.

Source code and other appropriate media for the resource estimation tools developed by TA2 over the course of the program. Source code should be well-documented and follow industry best practices for readability.

Other negotiated deliverables specific to the objectives of the individual efforts.

These may include registered reports; experimental protocols; publications; data management plan; intermediate and final versions of software libraries, code, and APIs, including documentation and user manuals; and/or a comprehensive assemblage of design documents, models, modeling data and results, and model validation data.

Reporting as outlined in Section VI.C.

H. Government-furnished Property/Equipment/Information

No Government-furnished Property/Equipment/Information is anticipated.

I. Other Program Objectives and Considerations

1. Collaboration

HR001121S0026 QUANTUM BENCHMARKING 19

Throughout the course of the program, it is likely to be necessary for all performers—regardless of category—to share relevant information regarding their research and development to support the larger program goals. DARPA expects all program performers to work collaboratively with one another to realize the program objectives outlined herein, so proposers should carefully review the goals for the entire program in order to fully understand the context of each program objective, performer category, and TA within the overall program structure. All proposals should describe plans for ensuring transparency of their processes to enable interactions with other program performers. Proposals that fail to include these plans may be deemed non-conforming and be removed from consideration.

2. Intellectual Property

The Quantum Benchmarking program emphasizes creating and leveraging open source technologies and architectures, making data sharing and collaboration key aspects of this program. Therefore, intellectual property rights asserted by proposers are strongly encouraged to be aligned with open source regimes. See Section VI.B.4 for more information related to intellectual property.

Because a key goal of the Quantum Benchmarking program is the creation of new benchmarks for quantum computers and because it is intended that these benchmarks be widely disseminated and adopted, all required milestones listed in Section F, except for Resource Estimation Tools, should be marked as publicly releasable, with no intellectual property restrictions of any kind.

The following guidance refers to delivered Resource Estimates and Resource Estimation Tools developed in TA2. All Resource Estimates should be marked as publicly releasable, with sufficient supporting information that would allow an expert in the field to verify and validate the estimates. Resource Estimation Tools, including all noncommercial software (including source code), software documentation, and hardware designs and documentation, should be provided as deliverables to the Government with a minimum of Government Purpose Rights (GPR).

II. Award Information

A. General Award Information

DARPA anticipates multiple awards.

The level of funding for individual awards made under this BAA will depend on the quality of the proposals received and the availability of funds. Awards will be made to proposers4 whose proposals are determined to be the most advantageous to the Government, all evaluation factors

4 As used throughout this BAA, “proposer” refers to the lead organization on a submission to this BAA. The proposer is responsible for ensuring that all information required by a BAA--from all team members--is submitted in accordance with the BAA. “Awardee” refers to anyone who might receive a prime award from the Government, including recipients of procurement contracts, cooperative agreements, or Other Transactions. “Subawardee” refers to anyone who might receive a subaward from a prime awardee (e.g., subawardee, consultant, etc.).

HR001121S0026 QUANTUM BENCHMARKING 20

considered. See Section V for further information.

The Government reserves the right to:

select for negotiation all, some, one, or none of the proposals received in response to this solicitation;

make awards without discussions with proposers;

conduct discussions with proposers if it is later determined to be necessary;

segregate portions of resulting awards into pre-priced options;

accept proposals in their entirety or select only portions of proposals for award;

fund awards in increments with options for continued work at the end of one or more phases;

request additional documentation once the award instrument has been determined (e.g., representations and certifications); and remove proposers from award consideration should the parties fail to reach agreement on award terms within a reasonable time or the proposer fails to provide requested additional information in a timely manner.

Proposals identified for negotiation may result in a procurement contract, cooperative agreement, or Other Transaction (OT), depending upon the nature of the work proposed, the required degree of interaction between parties, and other factors.

Proposers looking for innovative, commercial-like contractual arrangements are encouraged to consider requesting Other Transactions. To understand the flexibility and options associated with Other Transactions, consult http://www.darpa.mil/work-with-us/contract-management#OtherTransactions.

In accordance with 10 U.S.C. § 2371b(f), the Government may award a follow-on production contract or Other Transaction (OT) for any OT awarded under this solicitation if: (1) that participant in the OT, or a recognized successor in interest to the OT, successfully completed the entire prototype project provided for in the OT, as modified; and (2) the OT provides for the award of a follow-on production contract or OT to the participant, or a recognized successor in interest to the OT.

In all cases, the Government contracting officer shall have sole discretion to select award instrument type, regardless of instrument type proposed, and to negotiate all instrument terms and conditions with selectees. DARPA will apply publication or other restrictions, as necessary, if it determines that the research resulting from the proposed effort will present a high likelihood of disclosing performance characteristics of military systems or manufacturing technologies that are unique and critical to defense. Any award resulting from such a determination will include a requirement for DARPA permission before publishing any information or results on the program. For more information on publication restrictions, see the section below on Fundamental Research.

B. Fundamental Research http://www.darpa.mil/work-with-us/contract-management#OtherTransactions

HR001121S0026 QUANTUM BENCHMARKING 21

It is DoD policy that the publication of products of fundamental research will remain unrestricted to the maximum extent possible. National Security Decision Directive (NSDD) 189 defines fundamental research as follows:

‘Fundamental research’ means basic and applied research in science and engineering, the results of which ordinarily are published and shared broadly within the scientific community, as distinguished from proprietary research and from industrial development, design, production, and product utilization, the results of which ordinarily are restricted for proprietary or national security reasons.

As of the date of publication of this solicitation, the Government expects that program goals as described herein may be met by proposers intending to perform fundamental research and does not anticipate applying publication restrictions of any kind to individual awards for fundamental research that may result from this solicitation. Notwithstanding this statement of expectation, the Government is not prohibited from considering and selecting research proposals that, while perhaps not qualifying as fundamental research under the foregoing definition, still meet the solicitation criteria for submissions. If proposals are selected for award that offer other than a fundamental research solution, the Government will either work with the proposer to modify the proposed statement of work to bring the research back into line with fundamental research or else the proposer will agree to restrictions in order to receive an award.

Proposers should indicate in their proposal whether they believe the scope of the research included in their proposal is fundamental or not. While proposers should clearly explain the intended results of their research, the Government shall have sole discretion to determine whether the proposed research shall be considered fundamental and to select the award instrument type. Appropriate language will be included in resultant awards for non-fundamental research to prescribe publication requirements and other restrictions, as appropriate. This language can be found at http://www.darpa.mil/work-with-us/additional-baa.

For certain research projects, it may be possible that although the research to be performed by a potential awardee is non-fundamental research, its proposed subawardee’s effort may be fundamental research. It is also possible that the research performed by a potential awardee is fundamental research while its proposed subawardee’s effort may be non-fundamental research.

In all cases, it is the potential awardee’s responsibility to explain in its proposal which proposed efforts are fundamental research and why the proposed efforts should be considered fundamental research.

III. Eligibility Information

A. Eligible Applicants

All responsible sources capable of satisfying the Government's needs may submit a proposal…