Draft EES2 Roadmap AMMTO_August 2024.pdf

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE

Energy Efficiency Scaling for Two Decades Research and Development Roadmap—Version 1.0 i

N’scir

Energy Efficiency Scaling for Two Decades (EES2) R&D Roadmap

Version 1.0

August 2024

DRAFT REPORT

Energy Efficiency Scaling for Two Decades Research and Development Roadmap

Version 1.0

August 2024

DRAFT REPORT

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE ii

Disclaimer This work was prepared as an account of work sponsored by an agency of the United States

Government. Neither the United States Government nor any agency thereof, nor any of their -employees, nor any of their contractors, subcontractors or their employees, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or any third party’s use or the results of such use of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof or its contractors or subcontractors. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof, its contractors or subcontractors.

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE iii

Authors The initial draft of this report was prepared by Energetics, Inc., for the U.S. Department of

Energy Office of Energy Efficiency and Renewable Energy, Advanced Materials and

Manufacturing Technologies Office (AMMTO) based on inputs from a year-long roadmapping effort involving the participating organizations below documented at https://live-slac-microelectronics-d9.pantheonsite.io/sustainable-computing/energy-efficient-and-sustainable-computing.

The primary AMMTO author and Energy Efficiency Scaling for Two Decades (EES2) lead is

Tina Kaarsberg, AMMTO.

The primary EES2 analysis lead and author is Sadasivan Shankar, SLAC National Accelerator

Laboratory.

The primary Energetics, Inc., authors of this report are I-Hsi Daniel Lu, Nick R. Johnson, Kenta

Shimizu, Emmanuel Taylor, Russ Jones, Harper Alerion, and Ken Weaver.

The authors acknowledge Matt Roney, Dwight Tanner, and Jessica Blackburn from AMMTO’s technical writing team who provided invaluable editorial support.

They also acknowledge Thomas Finamore, the Energetics, Inc., producer who provided graphic editorial support.

https://live-slac-microelectronics-d9.pantheonsite.io/sustainable-computing/energy-efficient-and-sustainable-computing https://live-slac-microelectronics-d9.pantheonsite.io/sustainable-computing/energy-efficient-and-sustainable-computing https://live-slac-microelectronics-d9.pantheonsite.io/sustainable-computing/energy-efficient-and-sustainable-computing

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE iv

Participating Organizations1 Government

U.S. Department of Energy AMMTO

U.S. Department of Commerce National Institute of Standards and Technology

National Laboratories

Argonne National Laboratory

Brookhaven National Laboratory

Fermi National Laboratory

Lawrence Berkeley National Laboratory

Lawrence Livermore National Laboratory

National Renewable Energy Laboratory

Oak Ridge National Laboratory

Pacific Northwest National Laboratory

Sandia National Laboratories

SLAC National Accelerator Laboratory

Industry

3D Epitaxial Technologies

Advanced Micro Devices

Aligned Carbon

America's Frontier Fund

America's Frontier Fund

Applied Materials

ARM

BRDG Bridge to Connect

Cadence

Carbice

Carbon Technology, Inc.

Dedalo AI

Dexmat

Energetics

1 Except for the National Institute of Science and Technology, all participants’ organizations have formally signed the EES2 pledge.

Finwave Semiconductor

General Electric Vernova

Google Great Lake Crystal

Technologies

Hyperion

IBM

IEEE- USA

Infineon Technologies

Intel

Iris Light

Liqid

Metis Microsystems

Micron Technology

Microsoft

Multi3D

NanoSonic Inc.

Nantero

Nhanced Semiconductors, Inc

Paragraf

PseudolithIC

Quantum Silicon

Radiation Monitoring

Devices, Inc.

SAP

SEMI

Semiconductor Research

Corporation

Siemens

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE v

Sixline Semiconductor

Synopsys

TechSearch International

Tetramer

Tiptek

VIEE

Wolley Tech

Zyvex Labs

Universities

Duke University

Florida Semiconductor Institute

Hasso Plattner Institute

Los Angeles Trade-Technical College

Stony Brook University

University of Colorado at Boulder

University of Nebraska at Lincoln

University of Texas at Dallas

University of Texas at San Antonio

Working Group Co-Chairs Algorithms and Software

Brian Hirano, Micron Technology

Tapan Shah, General Electric Vernova

Advanced Packaging and Heterogeneous Integration

Moinuddin Ahmed, Argonne National Laboratory

Na Li, Carbice

Circuits and Architectures

Azeez Bhavnagarwala, Metis Microsystems

Emre Salman, Stony Brook University

Education and Workforce Development

Russell Harrison, IEEE-USA

David Shahoulian, Intel

Manufacturing Efficiency and Environmental Sustainability

Josh Ballard, Zyvex Labs

Prashant Nagapurkar

Steve Putna, Oak Ridge National Laboratory

Texas A&M University

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE vi

Materials and Devices

Jayasimha Atulasimha, Virginia Commonwealth University

John Baniecki, SLAC National Accelerator Laboratory

Paul Fischer, Intel

Shashank Misra, Sandia National Laboratories

Metrology and Benchmarking

Craig Green, Carbice

James Booth, National Institute of Science and Technology

Power and Control Electronics

Paul Sharps, Sandia National Laboratories

Tim McDonald, Infineon Technologies

Acknowledgments The authors would like to acknowledge the additional significant contributions of the following working group members and Energetics staff members:

Aaron Fisher, Lawrence

Livermore National

Laboratory

Alexander Paramonov, Argonne National Laboratory

Amanda Petford-Long, Argonne National Laboratory

Amir Ziabari, Oak Ridge

National Laboratory

Angel Yanguas-Gil, Argonne

National Laboratory

Anil Mane, Argonne National

Laboratory

Antonino Tumeo, Pacific

Northwest National

Laboratory

Ashfia Huq, Sandia National

Laboratories

Ben Tang, Infineon

Technologies

Bill Gervasi, Nantero

Brian Rowden, Oak Ridge

National Laboratory

Caecilia Gotama, BRDG

Can Bayram, University of

Illinois

Carlos Gutierrez, Sandia

National Laboratories

Chad Husko, Iris Light

Conrad James, Sandia

National Laboratories

Dan Green, PseudolithIC

Daniel Gopman, NIST

Danny Clavette, Infinenon

Technologies

David Gilmer, Nantero

David Gotthold, Pacific

Northwest National

Laboratory

David Voss, Energetics

Dhanushkodidurai

Mariappan, GE Vernova

Dhireesha Kudithipudi, University of Texas - San

Antonio

Elizabeth Neville Reyes, Applied Materials

Eungsan Cho, Infineon

Technologies

Francesco Musso, Dedalo AI

Godwin Maben, Synopsys

Hal Finkel, DOE/ASCR

Harish Bhandari, RMDI

Hsien-Hsin Sean Lee, Intel

Hyunim Chung, Sandia

National Laboratory

Jack Kotovsky, Lawrence

Livermore National Lab

James Provine, Aligned

Carbon

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE vii

Jan Vardaman, TechSearch

International

Jigesh Patel, Synopsys

Jo Luo, Nantero

Joe Kline, NIST

Joel Varley, Lawrence

Livermore National

Laboratory

John Leung, Intel

Josh Fryman, Intel

Julia Deitz, Sandia National

Laboratories

Keita Teranishi, Oak Ridge

National Laboratory

Keith Lanier, Synopsys

Maria DiGiulian, Infineon

Technologies

Martin Frank, IBM

Matthew Weimer, Forge

Nano

May Gokhale, Lawrence

Livermore National Lab

Muralidharan Govindarajan, Oak Ridge National Lab

Nick Lalena, DOE

Pamela Klabbers, Fermilab

Percy Kawas, Infineon

Technologies

Peter Dowben, University of

Nebraska - Lincoln

Petr Sushko, Pacific

Northwest National

Laboratory

Radislav Potyrailo, GE

Vernova

Rahul Sen, Nantero

Ricardo Ruiz, Berkeley Lab

Richard Kainradl, SAP

Rick Ridgley, Nantero

Sam Salama, Hyperion

Sameh Khalil, Infineon

Technologies

Sean Shaheen, University of

Colorado - Boulder

Shari Liss, SEMI Foundation

Srilatha Manne, AMD

Steffen McKernan, Carbon

Technology

Steve Buffat, Nantero

Steve Pawlowski, Micron

Sushanta Pal, Nantero

Timothy Wei, University of

Nebraska-Lincoln

Tommy Finamore, Energetics

Victor Zhirnov, Semiconductor Research

Corporation

Vishal Saxena, University of

Delaware

Volker Sorger, University of

Florida

Walt Zalis, Energetics

Wayne Johnson, ICF

Wiley Kirk, 3DET

Yiran Chen, Duke University

Yuepeng Zhang, Argonne

National Laboratory.

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE viii

List of Abbreviations 3DHI 3-dimensional heterogeneous integration

AC Alternating current

ADK Assembly design kit

AI Artificial intelligence

ALU Arithmetic logic unit

AMD Advanced Micro Devices

ANL Argonne National Laboratory

ANN Artificial neural network

AP Advanced packaging

APHI Advanced packaging/heterogeneous integration

ASIC Application-specific integrated circuit

AST Abstract syntax tree

BEOL Back-end-of-line

CAD Computer-aided design

CDO Carbon-doped silicon oxide

CDR Clock and data recovery

CD-SAXS Critical-dimension small-angle X-ray scattering

CD-SEM Critical-dimension scanning electron microscope

CIM Compute-in-memory

CISC Complex instruction set computer

CMOS Complementary metal–oxide–semiconductor

CMP Chemical mechanical polishing

CNT Carbon nanotube

CNTFET Carbon nanotube field-effect transistor

CPO Co-packaged optics

CPU Central processing unit

Cu Copper

CVD Chemical vapor deposition

CXL Compute express link

D2W Die-to-wafer

DBI Direct bond interconnect

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE ix

DC Direct current

DDR Double data rate

DFT Density functional theory

DNN Deep neural network

DoD Department of Defense

DOE U.S. Department of Energy

DRAM Dynamic random-access memory

DSA Domain-specific architecture

DSL Domain-specific language

DTCO Design technology co-optimization

DTM Dynamic thermal management

EAM Electro-absorption modulator

EDA Electronic design automation

EES2 Energy Efficiency Scaling for Two Decades

EMI Electro-migration issues

EMIB Embedded multi-die interconnect bridge

EOM Electro-optic modulator

ESD Electrostatic discharge

FeFET Ferroelectric field-effect transistor

FEOL Front end of line

FET Field-effect transistor

FinFET Fin field-effect transistor

FMEA Failure mode effect analysis

FP16 Floating point 16 (16-bit representation)

FP32 Floating point 32 (32-bit representation)

FPGA Field programmable gate array

FSG Fluorosilicate glass

FTJ Ferroelectric tunnel junction

GDDR Graphics double data rate

GPU Graphics processing unit

GWP Gross World Product

HBM High bandwidth memory

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE x

HCI Hydrochloric acid

HDI High-density interconnects

HDI Hydrogen Deficiency Index

HF Hydrofluoric acid

HI Heterogeneous integration

HIST Heterogeneous interconnect stitching technology

HMC Hybrid memory cube

HNO3 Nitric acid

HPC High-performance computing

HVAC Heating, ventilation, and air conditioning

HVM High-volume manufacturing

I/O Input/output

IC Integrated circuit

IC Interconnect

ICT Information and communication technology

IEEE Institute of Electrical and Electronics Engineers

ILD Interlayer dielectric

IMEC Interuniversity Microelectronics Centre

InP Indium phosphide

Int32 Integer 32 (32-bit precision)

Int8 Integer 8 (8-bit precision)

IoT Internet of Things

IP Intellectual property

Ir Iridium

IRDS International Roadmap for Devices and Systems

ISA Instruction set architecture

JVM Java Virtual Machine

KOH Potassium hydroxide

LED Light-emitting diode

LMP Liquid metal paste

LPDDR Low-power double data rate

MAC Multiply-accumulate

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE xi

MAPT Microelectronics and Advanced Packaging Technologies

MBE Molecular-beam epitaxy

MIEC Mixed ion-electric conductor

MIV Monolithic inter-tier via

ML Machine learning

MLIR Multi-level intermediate representation

MOCVD Metalorganic chemical vapor deposition

MOSFET Metal–oxide–semiconductor field-effect transistor

MWNT Multi-walled carbon nanotube

MZM Mach-Zehnder modulator

NAND Not AND memory

NaOH Sodium hydroxide

NASA National Aeronautics and Space Agency

NDE Non-destructive evaluation

NIST National Institute of Standards and Technology

NRAM Nanotube random-access memory

NRAM Non-volatile random-access memory

NVM Non-volatile memory

OEM Original equipment manufacturer

OOK On/off keying

ORNL Oak Ridge National Laboratory

OSAT Outsourced semiconductor assembly and test companies

OSC Organic semiconductor

PACE Power and control electronics

PAM4 Pulse amplitude modulation 4-level

PCB Printed circuit board

PCIe Peripheral Component Interconnect Express

PCRAM Phase change random-access memory

PDK Process design kit

PDU Power distribution unit

PFC Power factor correction

PIM Process-in-memory

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE xii

PINN Physics-informed neural network pJ Picojoule

PUE Power use effectiveness

QD Quantum dot

QW Quantum well

RC Resistive-capacitive

R&D Research and development

RD&D Research, development, and demonstration

RDL Redistribution layer

ReRAM Resistive random-access memory

RF Radio frequency

Rh Rhodium

RISC-V educed instruction set computer 5

RM Ring modulator

Ru Ruthenium

SerDes Serializer/deserializer

Sandia Sandia National Laboratories

SIA Semiconductor Industry Association

SiC Silicon carbide

SiCN Silicon carbon nitride

SiGAA Silicon gate-all-around

SiGe Silicon germanium

SiO2 Silicon dioxide

SiP System in package

SixNy Silicon nitride

SLAC Stanford Linear Accelerator Center

SNN Spike neural network

SNR Signal-to-noise ratio

SOA Semiconductor optical amplifier

SoC System on Chip

SOIC Small outline integrated circuit

SOI-TFET Silicon-on-insulator tunnel field-effect transistor

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE xiii

SOP Small outline package

SRAM Static random-access memory

SRC Semiconductor Research Corporation

SSD Solid state drive

STCO System technology co-optimization

STEM Science, technology, engineering, and mathematics

STTRAM Spin transfer torque random-access memory

SWNT Single-walled carbon nanotube

TCB Thermocompression bonding

TCO Total cost of ownership

TEM Transmission electron microscopy

TFET Tunnel field-effect transistor

TIM Thermal interface material

TMDC Transition-metal dichalcogenide

TPU Tensor processing unit

TRL Technology readiness level

TSMC Taiwan Semiconductor Manufacturing Company

TSV Through silicon via

UCIe Universal Chiplet Interconnect Express

UMC United Microelectronics Corporation

UPS Uninterruptible power supply

VCache Vertical cache memory

W2W Wafer-to-wafer

WDM Wavelength division multiplexing

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE xiv

Executive Summary This R&D roadmap is part of the U.S. Department of Energy’s (DOE’s) pledge to increase microelectronics’ energy efficiency 1,000-fold in two decades. With rapidly emerging challenges such as the increase in electricity use of data centers, innovations that exponentially increase energy efficiency are urgently needed to put microelectronics’ electricity use on a more sustainable path (see Figure ES-1). Just as President Kennedy did with his moonshot goal 60 years ago, DOE pledged to achieve this energy efficiency goal not because it is easy but because it is hard.

Background

Since the invention of the integrated circuit or “chip” 65 years ago, semiconductor-based electronics, or microelectronics, have enabled growth of information technology (IT)— computing, communication, and other electronics applications. Chip manufacturers now layer billions of semiconductor-based switches (i.e., transistors, the foundational unit of electronic devices) onto silicon to make the microelectronics that are essential for modern life. IT growth in the last century was propelled forward by the biennial doubling of transistor density on chips, which led to greater performance and lower cost per function. This tradition of exponential performance improvements is why much of the semiconductor industry already sets exponential technical goals. For example, the initial pledge signer, Advanced Micro Devices (AMD), had already set the goal to increase the efficiency of its chips 30 times by 2025; since signing DOE’s pledge, AMD has increased its goal to 100 times by 2027.

As transistors were miniaturized, chip power density initially remained constant (Dennard scaling), leading to more than doubling energy efficiency biennially. By 2005, however, this biennial efficiency doubling began to slow markedly as it reached certain physical limits. The slowing of efficiency doubling coupled with the rapid rise in energy and computation-intensive IT applications, has led to sharp increases in global

IT energy consumption. According to the Semiconductor Research

Corporation (SRC), by 2010, global computing energy use began to double every 3 years (see Error! Reference source not found.).

Energy Efficiency Scaling for Two Decades

The U.S. Department of Energy’s Advanced Materials and Manufacturing Technologies Office

(AMMTO) launched a national initiative with industry partners, national labs, and academia, called Energy Efficiency Scaling for Two Decades (EES2) in 2022. This ambitious program aims to double the energy efficiency of microelectronics biennially, targeting a 1,000-fold

Figure ES 1. SRC energy consumption forecast and the EES2 efficiency goal in two energy consumption scenarios https://ees2.slac.stanford.edu/doe-ees2-pledge

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE xv improvement over two decades. A key component of EES2 is this version 1.0 roadmap, the first in a series of research, demonstration, and demonstration (RD&D) roadmaps. This document is a product of extensive literature review and energy analysis, collaboration meetings between nine working groups, and expert input during the writing process. The working groups met monthly, with the organizing committee engaging in literature review and analysis to prepare for the following meeting.

The EES2 roadmap focuses on the largest and fastest growing IT energy user, the “compute stack” (see Figure ES-2), which comprises everything from devices to software. The stack shown is from the seminal DOE report Basic Research Needs for

Microelectronics (DOE Office of Science, 2018), which extended the notion of co-design from simply designing hardware and software together to specifically co-designing adjacent layers of the hardware with adjacent layers of the software. This roadmap examines innovative technologies co-designed by experts on different parts of the stack that can exponentially increase computing energy efficiency. This roadmap is a first step in a multi-year research effort to develop and deploy portfolios of cutting-edge microelectronics technologies that are 10-, 100-, and even 1,000-times more energy efficient than the technologies they replace.

Alone, none of the technologies will achieve the industry-wide biennial efficiency doubling leading to the 21 1,000-times goal.

DOE’s Undersecretary for Energy and Science and the now sixty-five other external industry-based EES2 pledgers were inspired to work toward this goal and join the roadmap effort on the strength of

DOE’s 2021–2022 “Semiconductor R&D for Energy Efficiency” virtual workshop series and its 2022 sponsored assessment of computing energy use (Shankar and Reuther 2022), which contributed additional insight on how the stack could be co-designed with rigorous analysis of computing performance using the metrics of energy per bit, per instruction, and per application.

Strategies for Efficiency First

To directly support the goals of EES2, co-design strategies are prioritized to optimize for efficiency first. Simply put, this means that where multiple properties are desired for a given technology solution, energy efficiency should be the first property for optimization in the co-design. In addition, three sub strategies emerged related to the three different energy metrics for the near-, mid-, and long-term, as shown in Figure ES-2.

Near-Term: Optimization of Energy per Instruction/Operation

EES2-sponsored analysis by Stanford Linear Accelerator Center (SLAC) (Shankar and Reuther

2022) showed large variation in energy per instruction or operation for different types of computational tasks. This suggests that a chip design strategy that ensures instruction complexity for a given task is as low as possible is the correct “efficiency first” hardware co-design strategy. Graphic processing units (GPUs) for gaming and artificial intelligence (AI) are a successful example of an approach of this type.

Figure ES-1. Compute Stack. Source: DOE Office of Science, 2018

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE xvi

Mid-Term: Device-Level Innovations To Minimize Energy Use per Bit

Because they are so foundational, innovations at the device level, especially with transistors, are critical. In the near- and mid-term, the EES2 roadmap highlights innovations that sharpen the subthreshold swing slope and lower switching voltage, such as tunnel field-effect transistors

(TFETs). In the mid- to long-term, device level innovations from quantum and nature-inspired computing will be critical for widespread advances from 100x to 1,000x energy efficiency.

Long-Term: Full-Stack Software-Driven Co-design To Minimize Energy per Application

The goal of full-stack co-design has yet to be implemented. This strategy is accelerating toward this goal by focusing on a subset of full-stack co-design that is software-driven, requiring that hardware developers understand what the software needs to do, and software designers understand the needs of hardware. Full implementation would require a major change in pedagogy and curriculum since software and hardware engineers have become more and more specialized in recent decades. But in the meantime, steps in this direction include specifying in algorithms that do not require high precision to save energy. Figure ES-2 illustrates the interaction between different layers of the compute stack, the timeline for the innovation, and the correlated energy metric.

Figure ES-2 Relationship of compute stack elements to achieving energy efficiency goals with different time frames

Expanding Co-Design for the Compute Stack → Stack Working Groups

When the concept of co-design was first applied to microelectronics, it simply meant the integration of hardware and software design in computing. Compute is the largest microelectronics system energy user—hence the focus of this roadmap. As the complexity of the compute stack grew (see Figure ES-3 left side), numerous subcategories of hardware and software were developed. In order to achieve the benefits of co-design envisioned by DOE in its seminal Basic Research Needs for Microelectronics (2018) report, co-design for energy efficiency must ensure that adjacent elements of the stack work together. AMMTO’s DOE partners in SC defined co-design in this 2018 report as “where each of the technical abstraction layers in modern computer system design (the compute stack), from fundamental materials research through applications, inform and engage other abstraction layers.” Furthermore, “co-design activities largely occur between adjacent technology abstraction layers (e.g., between materials and devices or computer architects and software designers).” Therefore, the initial four https://science.osti.gov/-/media/bes/pdf/reports/2019/BRN_Microelectronics_rpt.pdf

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE xvii

EES2 working groups (WGs) were assembled, as shown in Figure ES-3, to ensure co-design among adjacent layers.

Pledger Experience Led to Inclusion of Enabling Layers of Co-Design

EES2 industry and laboratory pledging partners with experience in the rapidly growing data center sector also urged the inclusion of power in the WGs’ co-design approach. Thus, a Power and Control Electronics WG was added. In addition, since the National Institute of Standards and Technology (NIST) had been involved with pre-EES2 efficiency efforts—and the EES2 team knew the importance of metrology to keep track of efficiency goals—a Metrology and

Benchmarking WG was included from the beginning. Finally, the analysis EES2 was conducting in parallel with the WGs showed that manufacturing energy use, complexity, and chemical intensity also had begun to rise rapidly in recent years, so a Manufacturing Energy Efficiency and Sustainability WG was included.

Early in the process, the working groups realized that past efforts at co-design had not generally involved software for hardware, such as the proprietary electronic design automation (EDA) software used to design circuits. To rectify this issue, the Circuits and Architectures WG began to meet with other WGs. In 2018, the co-design was mainly between adjacent layers, but by

2023, it had become clear that every aspect of the compute stack, plus every aspect of the enablers, needed to be aligned to minimize energy use.

Figure ES-3. Organization of the EES2 working groups

With the help of cross cutting pledgers such as SRC and IEEE (the world’s largest professional society), an Education and Workforce Development WG was formed. A co-design focus between working group areas is essential to make the rapid efficiency progress needed for biennial efficiency doubling and to ensure that the effort is both technically feasible and commercially viable.

https://live-slac-microelectronics-d9.pantheonsite.io/sustainable-computing/energy-efficient-and-sustainable-computing

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE xviii

The general scope of each WG is described in the following section.

Compute Stack Co-Design Working Groups

The Materials and Devices group tackled energy efficiency through materials and devices, such as carbon nanotubes and spintronic memory. This included scalability, thermal management, and interface issues in current materials.

The Circuits and Architectures group worked to overcome the challenges of slowing planar geometric scaling of transistors and memory. This group pioneered alternative, energy-efficient designs in processors and memory systems, including compute-in-memory technologies.

The Advanced Packaging and Heterogeneous Integration group at the next level up in the compute stack, worked on advanced thermal management techniques, and optimizing data movement strategies such as optical interconnects.

The Algorithms and Software group emphasized software-driven co-design and were inspired by natural systems such as dragonflies and human brains to benchmark neuromorphic algorithms matched directly with accelerator hardware.

Table ES-1. Condensed Focus Areas for Energy Efficiency and their Manufacturing Challenges &

Solutionsa

Focus Areas for Energy Efficiency Manufacturing Challenges & Solutions

Materials and Devices (Mid-Term)

Innovate in materials such as 2D materials, carbon nanotubes (CNTs), and ferroelectric materials for future CMOS alternatives.

Address production and integration challenges by investing in scalable high-quality material manufacturing and creating industry-wide standards and protocols.

Circuits and Architectures (Near-Term)

Enhance energy efficiency in compute architectures and memory technologies.

Prioritize advanced electronic design automation (EDA) and new architectures integrated with algorithms to optimize power distribution and increase energy efficiency, backed by continued investment in novel device technologies.

Advanced Packaging and Heterogenous Integration (Near-Term)

Develop vertically integrated, energy-efficient

3D technology stacking.

Pair novel technologies with state-of-the-art processors/memories to show durability and enhance intra-chip energy efficiency, improving EDA for system-level cooling and interconnect scaling.

Algorithms and Software (All Time Scales but Especially Long-Term)

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE xix a For complete list, refer to Table 85.

Crosscutting Co-Design Working Groups (Also Known as Enablers)

The Power and Control Electronics group focused on enhancement and innovation of power delivery systems on chip as well as in energy intensive applications such as data centers.

The Manufacturing Energy Efficiency and Sustainability group looked at the correlation between less efficient products and less efficient manufacturing processes to make them. The group also explored other energy-related environmental impacts of manufacturing.

The Metrology and Benchmarking group defined measurement and benchmarking standards necessary to evaluate emerging microelectronic technologies.

The Education and Workforce Development group took advantage of the compelling EES2 benefits to the planet for efforts to convince policy makers and potential new industry employees.

Table ES-2. Condensed Focus Areas for Energy Efficiency and Their Grand Challenges and Solutionsa

Innovate in machine learning algorithms and software that efficiently support diverse computing architectures.

Develop machine learning optimization through meta-learning and exploit massively parallel computing systems more effectively, using advanced parallelization of code.

Focus Areas for Energy Efficiency Manufacturing Challenges and Solutions

Power and Control Electronics (Very Near-Term)

Enhance power delivery and control across microelectronics to data centers by migrating loads to higher-efficiency regions and utilizing renewable resources.

Develop resource-aware scheduling strategies and implement advanced co-design tools to optimize power provisioning and thermal management, reducing overall energy consumption.

Manufacturing Efficiency and Environmental Sustainability (Near-Term)

Improve manufacturing processes to lower greenhouse gas emissions and energy consumption.

Introduce alternative gases and processes with lower environmental impacts and invest in alternate lithography technologies like nanoimprint lithography (NIL) for energy-efficient manufacturing.

Metrology and Benchmarking (All Time Scales)

Advance metrology by integrating AI/ML in nondestructive, high-resolution techniques to evaluate complex structures and materials accurately.

Establish comprehensive benchmarking standards and develop advanced metrology tools for real-time analysis, bridging the gap between traditional methods and the needs of emerging technologies.

Education and Workforce Development (All Time Scales but Especially Long-Term)

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE xx a For complete list, refer to Table 86.

Key Technologies Identified

Figure ES-4 graphically illustrates some promising technology options identified in this roadmap by adjacent co-design approaches to the compute stack, sorted by working group.

Figure ES-4. Key energy-efficient technologies for industry researchers to beat in each epoch

The figure illustrates one of two major criteria used by the working groups in evaluating candidate EES2 technologies: factor of efficiency improvement based on energy metrics (e.g., energy per bit, energy per switch, memory access) compared to state-of-the-art technologies.

Note the semi-log tick marks where the efficiency factor increases logarithmically to the right.

Note that these technologies do not in any way represent a government plan for energy efficiency. Rather these technologies and the technology areas from which they spring are technologies with “energy efficiency to beat.” Rather than being a plan or even a forecast, the roadmap seeks to provide benchmarks that will inspire technology developers to apply the recommended efficiency first design principles and possibly prove wrong dire predictions for future computing energy use.

Next Steps

This version 1.0 of the EES2 roadmap is the end of the beginning of a two-decade effort to take energy efficiency scaling from historical fact to future reality. The demand for computing and the critical need to curb emissions require an acceleration and expansion of current initiatives.

Cultivate a skilled workforce attuned to the demands of energy-efficient microelectronics and sustainability.

Align educational outcomes with industry needs, implement targeted training programs, and promote inclusivity to build a diverse workforce capable of driving global innovation.

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE xxi

In addition to the roadmap, DOE intends for the EES2 partners to begin a cycle of challenging one another on increasingly ambitious energy efficiency goals. For example, AMD has already begun to challenge the industry on AI chip efficiency—and a response from other AI chip makers is hopefully forthcoming soon. Other non-roadmap EES2 activities planned for the pledgers include the establishment of a testing facility to measure the relative efficiency of highly energy-intensive software (e.g., AI training, Transformer) due to the emergence of many different AI chip architectures. Such a testing facility would also verify the orders of magnitude efficiency improvements of AMMTO and other government funded hardware, such as TFET and neuromorphic chips.

Finally, AMMTO hopes that the EES2 partners will continue to document and learn from microelectronics’ past and forecasted future ability to enable all sectors of the economy to become more energy efficient and sustainable. EES2 partners will also continue to identify and publicize the problems solved and the opportunities offered by the roadmap 1.0 and the analysis performed for EES2. A surge in energy use forecasted for data centers is the first of many challenges to which the EES2 community will forcefully respond. Future potential energy-use surges related to communications (such as those that may accompany 6G+) will also be identified, documented, learned from, and publicized by the EES2 community as it evolves from a government-led organization to one that is privately led.

While this report documents myriad potential efficiency improvements across 55 technologies, achieving their full benefits requires an integrated approach that emphasizes software-driven co-design across the entire technology stack. Ultimately, EES2 hopes to reboot the energy efficiency doubling pace of Dennard scaling doubling efficiency every two years—with the goal of reaching 1,000 times more in the next 20 years.

Plans for roadmap 2.0 are already underway. As DOE and its partners recruit more industrial, academic, and national laboratory members for the EES2 innovation ecosystem, the initiative will not only have more policy impact, but it will also boast even broader technical expertise among the WGs. Now that the first roadmap is published, EES2 will actively turn to broaden its recruiting into new microelectronics application sectors, such as communications. In addition, while EES2 started with electronics and electrons, it will also broaden to promising new information carriers, such the photons used in optoelectronics and photonics. EES2 already includes pledgers whose research includes long-term transformational technology areas such as quantum computing as well as the latest advances in nature-inspired architectures. EES2 will work with these pledgers to help recruit additional pledgers from their respective sectors and to attract more volunteers for the version 2.0 WGs.

Although much can change before the start of version 2.0 of the roadmap in spring 2025, future

WGs will continue to build upon a solid base of peer-reviewed research while continuing to work with EES2 pledgers to lower barriers toward immediate deployment of technologies for biennial microelectronics energy efficiency doubling. This dual R&D and deployment strategy ensures flexibility and responsiveness to emerging technologies and market shifts, thereby fostering a sustainable evolution of the microelectronics sector.

As the EES2 Initiative continues to grow and build momentum for massive improvements in computing energy efficiency, the EES2 team will work further with stakeholders in microelectronics and related applications to develop the technology base and to assess progress toward the goal every 2 years.

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE xxii

This roadmap is not intended to serve as a forecast or to pick winners and losers among technologies. Rather, it is the opening salvo in a new energy efficiency “space race,” where instead of outer space, the EES2 team explores the fascinating realm of increasingly tiny and ultra energy efficient information systems. The roadmap sets a high bar to challenge and motivate technology developers and to counteract grim forecasts that humanity cannot achieve the clean energy transition due to rising computing energy use trends. The semiconductor industry’s inspiring past successes in improving energy efficiency indicate that ambitious EES2 efficiency goals can be met as well. Let’s do it now.

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE xxiii

Table of Contents Executive Summary ................................................................................................................. xiv

1 Introduction

1.1 Background

1.1.1 Moore’s Law

1.1.2 Dennard Scaling and the Initiative

1.1.3 Why 20 Years? Why 1,000 Times?

1.1.4 Linear Versus Exponential Growth

1.1.5 History of the Pledge and Pledgers

1.1.6 Analysis Metrics: Energy per Bit, Instruction, and Application

1.2 Scope of the Problem

1.1.1 Scaling Problems and Innovations To Overcome Them

1.2.1 The S-Curve

1.2.2 Major Sources of Computing Energy Use

1.2.3 Estimation of Inefficiencies

1.2.4 Efficiency First

1.2.5 Energy Use and Sustainability in Semiconductor Manufacturing

1.3 Key Concepts for Microelectronic Energy Efficiency

1.3.1 Co-Design Process

1.4 Organization of the Work

1.4.1 Compute Stack

1.4.2 Microelectronic Enablers

1.4.3 Cross-Collaboration

1.5 Methodology

1.6 Related Work

1.6.1 International Roadmap for Devices and Systems

1.6.2 Heterogeneous Integration Roadmap

1.6.3 2030 Decadal Plan for Semiconductors

1.6.4 Microelectronic and Advanced Packaging Technologies Roadmap

1.6.5 National Strategy on Microelectronics Research

1.6.6 How This Complements Prior Roadmaps and Strategies

1.6.7 Related Work References

2 Technologies for the Compute Stack

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE xxiv

2.1 Materials and Devices

2.1.1 2D Semiconductor Materials

2.1.2 Carbon Nanotube-Based Devices

2.1.3 Spintronic Devices

2.1.4 Ferroelectric Memory/Ferroelectric Field-Effect Transistors

2.1.5 Tunnel Field-Effect Transistors

2.1.6 Silicon Gate-All-Around Transistors

2.1.7 Emerging Devices and Materials for Analog Computing

2.1.8 Novel Materials for Silicon Scaling

2.1.9 Conclusion for Materials and Devices

2.1.10 Materials and Devices References

2.2 Circuits and Architectures

2.2.1 Memory Access

2.2.2 Digital Compute-in-Memory

2.2.3 Analog Compute-in-Memory/Neuromorphic Computing

2.2.4 Nonvolatile Memory

2.2.5 Domain-Specific Architectures

2.2.6 Instruction Set Architecture

2.2.7 Electronic Design Automation for Circuits and Architectures

2.2.8 Conclusion for Circuits and Architectures

2.2.9 Circuits and Architectures References

2.3 Advanced Packaging and Heterogeneous Integration

2.3.1 Carbon Nanotube-Based Interconnects

2.3.2 Optical Interconnects

2.3.3 3D Hybrid Bonding

2.3.4 Vertical Integration (2.5D/3D)

2.3.5 Thermal Management

2.3.6 Packaging Electronic Design Automation/Process Design Kits/Assembly Design

Kits 162

2.3.7 Conclusion for Advanced Packaging and Heterogeneous Integration

2.3.8 Advanced Packaging and Heterogenous Integration References

2.4 Algorithms and Software

2.4.1 Algorithm-Specific Energy Efficiency Tooling and Benchmarks

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE xxv

2.4.2 Reduced Energy for Machine Learning Algorithms

2.4.3 Reduced Energy for Algorithms Used in Scientific Computing

2.4.4 Reduced Energy for Cryptocurrency Mining

2.4.5 Software for Conventional Architectures

2.4.6 Software for Domain-Specific and Emerging Architectures

2.4.7 Conclusion for Algorithms and Software

2.4.8 Algorithms and Software References

3 Enablers

3.1 Power and Control Electronics (PACE)

3.1.1 State of the Art and Benchmarks

3.1.2 PACE Approaches for Reducing Computing Energy Use

3.1.3 PACE Honorable Mentions

3.1.4 Conclusion for Power and Control Electronics

3.2 Manufacturing Energy Efficiency and Sustainability (MEES)

3.2.1 Lithography

3.2.2 Process Gas Abatement Systems

3.2.3 Conclusion for Manufacturing Energy Efficiency and Sustainability

3.3 Metrology and Benchmarking

3.3.1 Enhanced Metrology

3.3.2 Artificial Intelligence/Machine Learning in Metrology

3.3.3 Failure Analysis

3.3.4 More Samples

3.3.5 Benchmarking

3.3.6 Conclusion for Metrology and Benchmarking

3.3.7 Metrology and Benchmarking References

3.4 Education and Workforce Development (EWD)

3.4.1 Raise Public Awareness on the Crucial Role Energy-Efficient Semiconductors

Play in Global Sustainability

3.4.2 Engage Students and Workforce in Microelectronics Research

3.4.3 Empower a Future-Ready Microelectronics Workforce Through Multidisciplinary

Education, Training, and Continuous Support for Educators and Learners

3.4.4 Navigate Demographic Shifts and Engage Diverse Talent

3.4.5 Conclusion for Education and Workforce Development

3.4.6 Education and Workforce Development References

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE xxvi

4 Conclusion

4.1 A New Moonshot and Space Race

4.2 EES2: Putting People and Their Organizations First

4.3 Technology Results and Co-Design for Efficiency First

4.4 The Future

5 Bibliography

5.1 High-Level Views

5.2 Computing Energy Use Cases

5.3 Materials and Devices

5.4 Advanced Packaging and Heterogeneous Integration

5.5 Photonics

5.6 Circuits and Architectures

5.7 Metrology and Benchmarking

5.8 Education and Workforce Development

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE xxvii

List of Figures Figure ES 1. SRC energy consumption forecast and the EES2 efficiency goal in two energy consumption scenarios ............................................................................................................ xiv Figure ES-2. Compute Stack. Source: DOE Office of Science, 2018 ........................................ xv Figure ES-3 Relationship of compute stack elements to achieving energy efficiency goals with different time frames ................................................................................................................ xvi Figure ES-4. Organization of the EES2 working groups .......................................................... xvii Figure ES-5. Key energy-efficient technologies for industry researchers to beat in each epoch xx Figure 2. The 2023 version of the SRC computing energy use forecast Figure 4. Analysis of the opportunity space for energy efficiency (from bits to bitcoin) according to the bits, instructions and application metrics. Source: Shankar 2022 Figure 5. 50 years of microprocessor trend data. Original data up to the year 2010 collected and plotted by Horowitz et al; new plot and data collected for 2010–2021 by Rupp 2022 Figure 6. The gap between processor performance and DRAM latency is measured as the difference in the time between processor memory requests. Source: Hennessey and Patterson

Figure 7. S-curve model Figure 8. Scale of energy use from bits to applications. Source: Jouppi et al. 2021 for INT8, FP64, SRAM and DRAM access; Shankar 2023 for the remaining values Figure 9. Energy cost for various operations Figure 10. The compute stack. Source: DOE Office of Science 2018 Figure 11. 2022–2023 organization of the EES2 working groups Figure 12. Definitions for technology readiness levels for the microelectronics industry as used in this report Figure 13. Technology options for new information processing technologies. Source: Hiramoto

Figure 14. Potential efficiency improvement factor and timeline for selected technologies proposed by the Materials and Devices working group Figure 15. Selected 2D materials and their bandgap. Source: Chaves et al. 2020 Figure 16. Si NS-FET versus 2D-FET: (a) Drain current improvement and (b) capacitance improvement. Source: Ahmed et al. 2020 Figure 17. Overview of synthesis techniques of single to few layer TMDC flakes. Source: Briggs et al. 2019 Figure 18. Graphene lattice with chiral angle and vector for CNT, along with two dominant CNT configurations (armchair and zigzag). Source: Ávila and Lacerda 2008 Figure 19. Comparison of energy and delay of a 32-bit adder among various charge- and spin-based devices with benchmarks showing performance and switching energy versus delay time.

Source: Pan and Naeemi 2018 Figure 20. The magnetoelectric FET with performance shown for a channel with a spin-orbit splitting of only 100 meV. With a channel of 0.5 eV spin-orbit splitting the on/off ratio might approach 104 (Dowben et al. 2018) Figure 21. Write energy vs. write delay for various types of spintronic memory cells. Star shows the desired target (Pan and Naeemi 2017)

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE xxviii

Figure 22. STT-MRAM device structure. Source: MRAM-info 2023 Figure 23. Operating principles of ferroelectric memory. Source: Mikolajick et al. 2021 Figure 24. Subthreshold Slope of I/V on/off curve for typical FET and subthreshold sharpening tech (e.g. TFET) Figure 25. Device operation of a tunnel field-effect transistor (TFET): (a) the schematic of a

TFET cross section and band diagram along the channel in on and off states and (b) comparison between TEFET and MOSFET operation principle. Source: Agha et al. 2021 Figure 26. Enhanced ON current and subthreshold slope with atomically precise advanced manufacturing (APAM). Source: Kaarsberg, Misra, and Shimizu 2023 Figure 27. Typical source, drain, and gate arrangements for planar, FinFET, and GAA transistors. Source: Semiconductor Engineering 2023 Figure 28. Basic schematic of memristors in crossbar arrays. Source: Yadav et al. 2023 Figure 29. Novel encapsulation strategy Figure 30. Schematic of pore stuffing method Figure 31. Logarithmic comparison of the damascene line resistance vs. the total conductor cross-sectional area of Ru, Co, and Cu nanowires Figure 32. Potential efficiency improvement factor and timeline for selected technologies of the

Circuits and Architectures working group Figure 33. Classical von Neumann computer architecture Figure 34. Typical memory hierarchy sizes and access times (c. 2019) Figure 35 CXL Native DRAM 8-lane PCIe Gen5 vs. LPDDR 2x LPDDR-6400 latency vs.

bandwidth comparison Figure 36. Potential efficiency improvement factor and timeline for selected technologies of the

APHI working group Figure 37. Interconnect figure of merit benchmarks (circa 2018) with 2023 commercial and R&D optical interconnect benchmark references. Source: Stojanovic 2020; Original source: Gordon

Keeler, DARPA MTO, ERI Summit 2019 Figure 38. Optoelectronic modulator device scaling laws Figure 39. Comparative images and size scales for solder, microbump, and 3D hybrid bonding interconnects. Source: Jani 2019 Figure 40. 3D hybrid bonding process with Cu vias and SiO2 films. Source: Ong et al. 2022 .. 145 Figure 41. Digital signal channel paths and associated capacitance, including (left) schematics of digital signal channels: (a) bridge-chip 2.5D, (b) interposer and HIST 2.5D, and (c) 3D integrations; and (right) capacitance breakdowns for these signal channels (ESD = electrostatic discharge capacitor). Source: Zhang, Zhang, and Bakir 2018 Figure 42. Relative sizes of typical NAND gates, MIVs, and TSVs. Source: Samal et al. 2016151 Figure 43. Algorithms and Software working group potential efficiency improvement factor and timeline initial assessment Figure 44. Interaction of software and CPU architecture Figure 45. EES2 proposed approach to evaluation of computer system energy performance and progress toward long-term improvement goals Figure 46. Market growth worldwide for machine learning and artificial intelligence through 2030.

Figure 47. Neural network with one hidden layer

U.S. DEPARTMENT OF ENERGY OFFICE OF ENERGY EFFICIENCY & RENEWABLE ENERGY | ADVANCED MATERIALS & MANUFACTURING TECHNOLOGIES OFFICE xxix

Figure 48. Complexity of machine learning models Figure 49. Physics-informed neural network differential equation solver Figure 50: Energy/Instruction based on HPL benchmarks Rmax. The X-axis consist of the top

500 supercomputers with the fastest on the left and slowest on…