Technical Requirements Document for

ESIF-HPC-4

Status: Draft v1.0

Table of Contents Table of Contents

1.0 Introduction

1.1 Guidance for O:erors

1.2 Requirements Definitions

1.3 Schedule

1.4 Collaboration and Partnership

2.0 Workload Overview

3.0 System Architecture Requirements

3.1 System Description

3.2 Requested Options

4.0 High-Performance Compute Zone

4.1 Compute Nodes

4.2 High Speed Interconnect

5.0 Access Zone

5.1 Access Zone Architecture

5.2 Access Nodes

5.3 User Environment

5.4 Performance Analysis Tools and Debuggers

6.0 Storage Zone

6.1 Parallel File System

6.2 AI-Optimized Storage

7.0 Management Zone

7.1 System Management Architecture

7.2 Operating System and Workload Management

7.3 System Monitoring

7.4 Power and Energy

7.5 Security

8.0 Benchmarks

8.1 Benchmark Categories

8.2 Application Benchmarks

8.3 Microbenchmarks

8.4 I/O Benchmarks

8.5 Benchmark Requirements

9.0 Facilities and Site Integration

9.1 Power and Cooling

9.2 Delivery and Installation

10.0 Maintenance and Support

10.1 Maintenance and Support

10.2 Resilience, Reliability, and Availability Metrics

10.3 Media Retention

11.0 Project Management

11.1 Project Management and Organization

11.2 Project Planning, Schedule, and Risk

11.3 Technical Coordination and Acceptance

12.0 Training and Documentation

12.1 System Administration Training

12.2 End User Training

12.3 Documentation

Appendix A: Benchmarking Application Throughput Improvement Metric

Appendix B: Facilities & Site Integration

Data Center Floor Layout

1.0 Introduction

The Alliance for Energy Innovation (“The Alliance”), on behalf of the National Laboratory of the Rockies (NLR), has released a Request for Proposal (RFP) for the next-generation High Performance Computing (HPC) system to be installed at NLR’s Energy System Integration Facility (ESIF). This new system, which we refer to as ESIF-HPC-4, is expected to be delivered in mid-2027 and ready for production in 2028. The primary objective of the ESIF-HPC-4 procurement is to provide a computational system that will support the mission computing needs of the Department of Energy's Office of Critical Minerals and Energy Innovation (CMEI).

This document contains the technical requirements for ESIF-HPC-4. Each proposed solution in response to this document should clearly describe the role of the major lower-tier technology subcontractors and third-party software partners and the value added that these subcontractors provide.

These draft requirements describe specific technical requirements related to both the hardware and software capabilities of the desired system as well as application benchmark requirements.

NLR aims to procure a highly productive, robust system that handles a diverse workload comprised of a mix of traditional modeling and simulations workloads as well as workloads in artificial intelligence (AI) and machine learning (ML).

The delivery timing provided in Section 1.3 represents current outlook and alignment of programmatic requirements and funding. “Alliance” reserves the right to revise the above schedule based upon their and/or DOE’s needs.

1.1 Guidance for O0erors

These requirements are structured with the objective of allowing o:erors to propose a wide range of solutions, including an on-premise or o:-premise system. Specific guidance for preparing responses is given below. O:erors may provide one or more proposals for the scenarios.

O:erors should respond to as many requirements and options as are applicable to the proposal and are practicable but should not feel obligated to respond to all. O:erors should mark all requirements, and label requirements not o:ered as Not O:ered.

In Section 10, The Alliance requires maintenance for the first five years post-acceptance with options for maintenance in years 6 and 7. If o:eror is bidding a service for which maintenance is included in the service price, describe the maintenance as requested in the requirements and indicate in the separate Pricing Sheet that the cost is included.

1.1.1 Guidance for On-Premise Proposals

NLR has a data center at the Energy Systems Integration Facility with power delivery capability of 10MW. The requirements section 9 provides information on the ESIF facility. Appendix B: Facilities section has additional information on the location and capabilities.

1.1.2 Guidance for O=-Premise Proposals

NLR intends for the language in these Technical Requirements to be inclusive of o:-premises system proposals. If proposing an o:-premises solution, the requirements of Section 9:

Facilities do not apply and require no response.

NLR expects as much detail in the System Description regarding processors, nodes, interconnect, and storage as would be provided in an On-Premises proposal. As part of the Software response, indicate the level of service being proposed and how software will be provided (e.g., base operating system, programming environment, workload manager, storage) to NLR to deploy on the system.

1.2 Requirements Definitions

Requirements in this document have priority designations, which are defined as follows:

• Mandatory Requirements (MR) are features that are essential to NLR’s requirements, and an O:eror must satisfactorily propose all Mandatory Requirements in order to have its proposal considered responsive.

• Target requirements designated as (TR-1, TR-2, or TR-3)

Target Requirements (designated as TR-1, TR-2, or TR-3) are features, components, performance characteristics, or other properties that are important to NLR. Target Requirements are prioritized by dash number, where TR-1 is most desirable to NLR, while TR-2 is more desirable than TR-3. Target Requirement responses will be evaluated as part of the proposal evaluation process.

• Technical Option Requirements (designated as TO-1, TO-2, or TO-3)

Technical Option Requirements (designated TO-1, TO-2, or TO-3) are features, components, performance characteristics, or upgrades that are important to NLR. Technical Options add value to a proposal. Technical Options are prioritized by dash number. TO-1 is most desirable to NLR, while TO-2 is more desirable than TO-3. Technical Option responses will be evaluated as part of the proposal evaluation process; however, NLR may or may not elect to include Technical Options in the resulting subcontract(s). Each proposed TO should appear as a separately identifiable item in an O:eror’s proposal response.

1.3 Schedule

The following is the tentative schedule for the HPC-4 system acquisition:

Table1

HPC-4 Schedule Request for Proposals release Q4 FY26 O6eror questions due Q4 FY26 Proposals due Q1 FY27 Early access hardware delivery FY27 Full system delivery Q1 FY28 System acceptance Q2 FY28

1.4 Collaboration and Partnership

The NLR Energy System Integration Facility (ESIF) is collaborative research hub for future-ready, secure, reliable, and a:ordable energy technologies. As the home of NLR’s high-performance computing systems, the ESIF HPC Data Center serves as a living laboratory where advanced computing, AI workloads, and physical energy systems converge. Within this environment. NLR’s HPC systems are tightly integrated with experimental facilities to enable rapid materials screening, digital twins, hardware-in-the-loop experimentation, and real-time experimental steering. The HPC platform functions as a computational backbone for close-loop workflows in which modeling, simulation, and machine learning operate in concert with laboratory instrumentation – accelerating translation from discovery to scale-up and commercialization.

Areas of particular interest for collaboration include partnerships fostering innovation across the full stack from advanced semiconductor technologies and system architectures to facility integration and grid-responsive operations. NLR is interested in partnerships that explore new approaches to performance-per-watt optimization, composable and heterogeneous architectures, advanced interconnect and storage technologies, and hardware–software co-design under real operational energy constraints. Additional areas include digital twin frameworks and workload management strategies that enhance data center flexibility and grid interactivity, as well as AI-driven autonomous experimentation enabled by tightly coupled compute and experimental systems. These collaborations may span core HPC platforms, edge-to-core integration, real-time data pipelines, and emerging multi-modal computing paradigms that accelerate applied energy research and deployment.

Looking beyond the installation of new computational capabilities, NLR welcomes proposals that include substantial future collaboration and partnership with the successful O:eror. These could take the form of joint institutes, a new center of excellence, co-development of AI-optimized and power-e:icient computing architecture, product testing and validation within a grid-connected research environment, or coordinated product announcements. Proposed opportunities should be mutually beneficial to both organizations and be consistent with NLR’s leadership role in applied energy research and AI-enabled energy innovation.

2.0 Workload Overview

NLR’s systems have historically supported a heterogeneous mix of traditional modeling and simulation workloads, as well as high-throughput ensembles of smaller jobs. AI/ML-driven workloads are increasingly becoming a larger percentage of compute, as are hybrid and data intensive workloads. Specifically:

• Artificial Intelligence: AI and machine learning (AI/ML) is a growing workload on NLR systems.

As researchers and application developers explore and integrate AI/ML technologies into their workflows, a broad range of use cases is anticipated. These workloads are comprised primarily of training and inference jobs ranging from single-node to multi-node scale requiring high memory bandwidth and intra-node GPU-GPU communication. Large language models (LLMs) and generative AI workloads are also experiencing growth. Many workloads take the form of workflows that couple AI/ML with traditional simulation codes, requiring double-precision data types and fast data movement between CPUs, GPUs and storage. These AI/ML-powered workflows will possibly require an optimized storage layer that can handle high I/O, i.e., frequent reads of large number of training files and model artifact writes.

• Modeling and Simulation: Characterized by large-scale, tightly coupled jobs spanning hundreds of nodes with both strong and weak scaling requirements, and requiring high-precision data types (i.e. FP64). The largest runs scale to ~50% of the system, though the majority typically consume 5-10% of compute resources per job. In addition, this category includes high throughput ensembles of smaller jobs using fewer than 100 nodes, but are often single node only. Simulation codes are increasingly migrating to accelerated paradigms, and many require scaling to multiple GPU nodes. A minority of the modeling and simulation workloads will continue to require CPU-only nodes. The interconnect needs to provide su:icient support for all modeling and simulation workloads, provide application scalability, and balance node-level computation performance.

• Hybrid and Data Intensive: Includes digital twins, distributed data analytics, visualization, experimental interoperability, and edge compute capabilities, often requiring container-based deployments or orchestrated workflows. Some use cases require real-time on-demand access to compute resources rather than the traditional HPC queue, to enable operational digital twin research, AI-controlled experimental facilities, and hardware-in-the-loop experiments.

3.0 System Architecture Requirements

The system requirements have been split into four zones: high performance compute zone, access zone, storage zone, and management zone. These zones are based on NIST SP 800-223 and are a logical segmentation of the HPC system.

Figure1

3.1 System Description

The O:eror shall provide a detailed architectural description of the proposed compute system, including text, diagrams, tables, etc. as needed. Provide descriptions of each processor type, node type (which and how many processors, connectivity between processors, memory, NICs, node-local storage, if any), blade/chassis type (if applicable), rack/cabinet and interconnect. The o:eror should include quantities and define any minimum scalable unit sizing to maintain optimal performance and productivity across the system.

[MR] Provide a high-level overview of the system by describing the items below, if applicable:

• Component architecture – details of all processor(s), memory technologies, storage technologies (if used on the compute nodes), network interconnect(s) and any other applicable components.

• Compute node architecture(s) – details of how components are combined into the node architecture(s). Details should include bandwidth and latency specifications (or projections) between components. Details should be provided for each compute node type.

• Board and/or blade architecture(s) – details of how the node architecture(s) is integrated at the board and/or blade level. Details should include all inter-node and inter-board/blade communication paths and any additional board/blade level components.

• Rack and/or cabinet architecture(s) – details of how board and/or blades are organized and integrated into racks and/or cabinets. Details should include all inter rack/cabinet communication paths and any additional rack/cabinet level components.

• Interconnect - details of the system’s high speed network topology and connectivity across all system components (compute nodes, login nodes, management system).

• System architecture – details of how rack or cabinets are combined to produce system architecture, including the high-speed interconnects and network topologies (if multiple).

• Management node(s) - details of hardware to support management and services to operate the ESIF-HPC-4 system. Describe the management node types (e.g., admin, leader, scheduler/resource manager, fabric manager). Multiple node types may be needed to optimize for di:erent uses. Management nodes should be accessible even if the compute system is unavailable.

• Access zone nodes- details of hardware to support user access and potentially user-driven workflow activities. A pool of access zone nodes will be needed to address the requirements described in Section 7. The access zone will collectively support 400 interactive users, 60 data analysis front-end applications,150 integrated development environments and 100 simultaneous compilations of software of equivalent complexity to the latest GNU Compiler Suite. The access zone should be accessible even if the high performance compute zone is unavailable.

3.2 Requested Options

The O:eror will provide Technical Options to enable NLR to adjust various components of the system. Only provide descriptions of the options in this document; do not include prices. Prices should be made available separately in the Price Breakdown Table.

Technical Options may include architectural choices such as additional racks, changes to memory capacity or interconnect topology. The O:eror should describe which options remain viable after deployment (e.g. adding racks of CPU-only nodes, adding AI nodes, incremental storage adds) and for how long after deployment.

3.2.1 Hardware Options

3.2.1.1 [TO-1] Scale the System Size The O:eror should provide an option in the Price Breakdown Table to increase and decrease the compute partition. Provide options separately for each node type proposed. Describe here the scalable unit (e.g., rack, N racks with CDU, row). The O:eror should identify any thresholds requiring increased component infrastructure (e.g., extra switches, extra cables), any technical challenges foreseen with respect to scaling and any other production issues.

3.2.1.2 [TO-1] Scale Processor Memory For each processor type, the O:eror should provide an option to increase and decrease, if possible, the memory capacity. Describe the scaling units. The price should be per scaling unit (e.g., rack, N racks).

3.2.1.3 [TO-2] Scale the Interconnect The O:eror should provide options to increase and decrease the interconnect bandwidth. This option may include additional NICs per node, additional switches, additional cables, etc. Describe the expandability of the interconnect and identify any thresholds requiring increased component infrastructure (e.g., extra switches, extra cables), any technical challenges foreseen with respect to scaling and any other production issues.

3.2.1.4 [TO-1] Additional Maintenance Periods The System Price should include five years maintenance. The O:eror should provide separate options for year 6 and year 7 maintenance and support for the system. If the O:eror is a cloud provider, provide the price to continue the service for each year.

3.2.1.5 [TO-3] Mid-Life Upgrades The O:eror should describe and separately price any future options for adding node types not available during the proposal period to the proposed ESIF-HPC-4 system over its five-year lifetime.

3.2.1.6 [TO-3] Data, Analysis and Visualization Nodes In order to provide additional visualization/rendering capabilities, the O:eror may propose a price for a scalable unit (e.g., rack, N racks with CDU) of data, analysis and visualization/rendering nodes that connect to the high speed interconnect. Describe the node architecture including processor architecture, number of processors, memory, connectivity within the node, connectivity to the high speed interconnect, and node-local storage, if any. The O:eror may choose to provide multiple options and prices for di:erent architectures.

3.2.1.7 [TO-3] Node-Local Storage For each node type, the O:eror should provide an option to add NVMe drives with a target size of at least triple the node memory capacity. Describe the NVMe performance characteristics and system software requirements.

3.2.1.8 [TO-3] Advanced Architectures Option NLR is interested in optional advanced or novel architectures that can accelerate AI, data-driven analytics, modeling and simulation, and other emerging scientific workflows. These may include novel AI/ML acceleration technologies, heterogeneous or domain-specific processors, dataflow or reconfigurable architectures, neuromorphic approaches, or other innovative designs capable of integrating into the HPC ecosystem.

Proposed options should describe how the architecture would enhance performance, scalability, and workflow efficiency for workloads characterized by high-throughput simulation, AI-driven model development and tight integration with experimental facilities. The option may be delivered with the system or later and may consist of scalable units defined by the Offeror. Multiple options may be proposed and should be described and priced separately. Offerors may highlight architectural features that provide clear benefits for integrated AI/HPC workflows, rapid iteration cycles, real-time or streaming data pipelines, or acceleration of domain-specific numerical kernels.

For each proposed Advanced Architectures option, the Offeror shall provide:

• An architectural description: an overall architectural diagram showing hardware, interconnect(s), compilation infrastructure, and I/O subsystems; and confirmation that the partition can mount NLR’s file system(s) at the required performance.

• A software environment description: an overview of the software stack, including libraries, SDKs, supported frameworks, usability/programmability characteristics, and all associated licensing and support terms.

• A performance characterization: performance results (measured or projected) for relevant benchmarks listed in Section 8, with the benchmark version, submission status, and any rule modifications clearly specified.

3.2.3 Early Access Technology

[TO-3] The O:eror should describe options to provide NLR with early access hardware that enables hardware and software testing prior to delivery of the production system. Early-access systems may use processor generations one or two generations older than the processors proposed for the production system (i.e., N-1 or N-2). Early access to other key technologies, such as interconnects, memory, or accelerators, is also encouraged. If multiple early-access configurations are o:ered, each should be described and priced separately.

3.2.4 Alternatives to Stand-Alone Test and Development System

NLR seeks alternatives to a traditional stand-alone Test and Development System (TDS). Because such systems are often difficult to maintain in lockstep with the production system, NLR is interested in approaches that leverage the early-access resources described in Section 2.2.3 or that make use of the delivered system itself.

Prior to substantial system delivery, NLR is interested in mechanisms that provide developers and user-support staff with early system exposure to accelerate application readiness and reduce time-to-production. These mechanisms may include access to pre-delivery hardware or early access hardware as described in Section 3.2.3 that mirrors the configuration of the proposed system; or off-premises resources comparable to the proposed system for application porting, training, or workflow development.

After substantial delivery, NLR is interested in how the production system can be partitioned or segmented to support development activities, such as testing operating system updates, firmware revisions, system software changes, or application-stack upgrades, without impacting broader system availability.

3.2.4.1 [TO-1] Pre-delivery options for Test and Development The O:eror shall describe available options for providing pre-delivery access or resources suitable for early development and training.

3.2.4.2 [TO-1] Post-delivery options for Test and Development The O:eror shall describe how the delivered system can be partitioned, isolated, or otherwise configured to support testing and development while maintaining production operations. Describe any additional hardware, software, or architectural features that enable these test and development workflows.

3.2.5 O=-premise Options

NLR is interested in augmenting on-premises compute capacity with off-premise, consumption-based (cloud) resources to support bursting or time-sensitive workloads. Cloud resources may be used for overflow demand, urgent mission-driven work, or workloads that require hardware capabilities not available on-premises.

The Offeror should provide options for providing a guaranteed pool of compute capacity (e.g., node-hours or equivalent metrics) on a monthly or quarterly basis for the life of the production system. Proposed solutions should include mechanisms for securely accessing cloud resources, expected performance characteristics, and any data-movement considerations.

For Off-premise options, the Offeror shall provide:

3.2.5.1 [TO-1] Security All proposed cloud resources must be FedRAMP-authorized at a level suitable for NLR workloads.

3.2.5.2 [TO-2] Capacity and pricing Descriptions of guaranteed capacity options and associated pricing models.

3.2.5.3 [TO-3] Integration Any mechanisms for integrating or extending on-prem workflows to cloud environments (e.g., bursting, federated schedulers, or containerized execution).

3.2.5.4 [TO-3] Other considerations Any constraints, data-transfer requirements, or architectural considerations relevant to the proposed solution.

4.0 High-Performance Compute Zone

The High-Performance Compute Zone contains the compute nodes and the high-performance interconnect which connects the compute nodes to the other zones. NLR's anticipated workload will be diverse, and serve AI, high-throughput, high-performance, and data-centric computing models. NLR anticipates the majority of the compute cycles will be provided by accelerated nodes.

4.1 Compute Nodes

4.1.1 Compute node architecture

[TR-1] O:eror shall provide a detailed description of each proposed compute node type, including:

• Processor or accelerator architecture

• Memory types, configuration, and maximum capacity

• Network interface type(s) and link topolog

• Per-network-interface bandwidth and latency, and the e:ective per-node network bandwidth

• Per-node peak computational performance and sustained memory bandwidth

• High-bandwidth memory (HBM) capacity provided on accelerated node types, per accelerator. It is highly desirable that the system provide at minimum double the HBM capacity per accelerator than Kestrel

4.1.2 Compute node ratio

[TR-2] The O:eror shall describe the compute node architecture ratio of the proposed system. A CPU-only (Arm or X86_64) partition shall represent no more than 10% of the system’s total peak FLOP capacity.

4.1.3 Compute Processor Energy E=iciency

[TR-2] It is highly desirable that the system support operation at the optimal point on the performance-per-watt curve during computationally intense phases. The Offeror shall describe any user-controllable power or frequency features available on compute processors, accelerators, NICs, and switches.

4.2 High Speed Interconnect

4.2.1 Network Description

[TR-1] The O:eror shall describe the high-speed network including:

• A high-level description of topology and routing protocol

• The per-node and per-link network bandwidth and latency characteristics

• The number and type of high-speed network interfaces on each compute node

• Any hardware-accelerated communication features, such as GPUDirect, GPUDirect RDMA, or equivalent technologies

• Any network-level congestion management features relevant for accelerated workloads.

4.2.2 Resilience

[TR-1] The Offeror shall describe the network’s resilience features, including link- and switch-failure scenarios, the number of failures tolerated while maintaining connectivity, and the expected impact on performance as components fail.

4.2.3 Management

[TR-1] The Offeror shall describe the out-of-band management network, including supported security mechanisms, routing and segmentation capabilities.

4.2.4 Quality of Service

[TR-2] The Offeror shall describe the interconnect QOS mechanisms, including congestion control, traffic classes/virtual channels, and prioritization options. The Offeror should describe any recommended QoS configurations for typical HPC workloads (messaging, collective operations, and storage IO) and any features that support complex or multi-tenant workflows.

4.2.5 Compute Network Integration

[TR-2] The Offeror shall describe the capability of the high-speed network to integrate storage systems, specialized nodes, or other vendor-supplied components into the fabric, including any limitations on interoperability or mixed-vendor configurations.

4.2.6 Protocol Support

[TR-1] The Offeror shall describe protocol support for both the high-speed and management networks, including:

• Supporting IP networking capabilities (IPv4/IPv6),

• Capabilities for traffic control and ACLs on management interfaces,

• Mechanisms for path selection, routing, and load balancing where applicable.

4.2.7 CXL

[TR-3] If CXL is o:ered, the O:eror shall describe supported CXL versions and capabilities for any CPU, accelerator, NIC, or storage components that utilize CXL.

4.2.8 MPI

[TR-1] The system shall provide MPI libraries supporting MPI 4.0 or higher, with accelerator-aware MPI where supported by the accelerator vendor. The O:eror shall describe any deviations from or extensions to the MPI standard.

4.2.9 PMI

[TR-1] The system shall support the Process Management Interface (PMI). The O:eror shall describe the supported PMI versions and any relevant integrations.

4.2.10 Other Communication Libraries

[TR-3] The O:eror shall describe any additional communication libraries provided (e.g., SHMEM, UCX-based PGAS libraries, etc).

4.2.11 Optimized Collective Libraries

[TR-3] The O:eror shall describe any optimized collective-communication libraries provided (e.g.

NCCL, RCCL, vendor-specific collective libraries) and any noted performance characteristics or limitations.

5.0 Access Zone

The Access Zone provides the primary user and administrative entry point to the system. It must support secure authentication, data ingress/egress, interactive workflows, and system management functions while maintaining compliance with DOE’s cybersecurity requirements. The Login and DAV Nodes should provide an architecture and programming environment compatible with the High-Performance Compute Nodes. NLR will provide dual 100G or dual 25G Ethernet switch connections for every system in the Access Zone to connect to the NLR data center network or Science DMZ. For user accessed resources like login nodes and data transfer nodes, 100G Ethernet is preferred.

5.1 Access Zone Architecture

5.1.1 Architecture

[TR-1] The O:eror shall describe how the proposed Access Zone architecture provides login, data transfer, and analysis/visualization services, supporting 400 interactive users, 60 data analysis front-end applications,150 integrated development environments and 100 simultaneous compilations of software of equivalent complexity to the latest GNU Compiler Suite.

5.1.2 Access Zone compatibility

5.1.2.1 [TR-1] Component Interoperability The O:eror shall describe how Access Zone components interoperate with the Compute Zone, including compatibility with the system software stack, storage systems, and cluster management, without requiring workflow modifications from users.

5.1.2.2 [TR-2] Login and DAV nodes Login and DAV nodes are listed separately by function in this document. Describe if the two node types can be provided by the same physical node type.

5.2 Access Nodes

5.2.1 Login Nodes

The system should support interactive access for command-line session, pre- and post-processing, and development workflows like Jupyter or VSCode.

5.2.1.1 [TR-1] Login node architecture The O:eror shall describe the login node architecture that will meet these requirements and provide su:icient compute, memory and I/O for concurrent multi-user workloads.

5.2.1.2 [TR-2] Login node connectivity Login nodes should connect to the Ethernet data center network with dual connections preferred.

5.2.2 Data, Analysis, and Visualization Nodes (DAV)

NLR uses Data, Analysis and Visualization nodes for pre and post processing as well as other interactive graphical work. Typically, these are deployed with a virtual desktop provided by NLR such as FastX.

5.2.2.1 [TR-1] DAV Nodes The system should support interactive analysis and remote visualization of simulation and experimental data. The O:eror shall describe a set of at least 15 DAV nodes that incorporate accelerators, memory capacity and software

5.2.2.2 [TR-1] OpenGL Describe the support for OpenGL to enable remote visualization and interactive exploration of large datasets.

5.2.3 Data Transfer Nodes

Data Transfer nodes will be used for secure, high-throughput data transfer between the system’s file system and external networks. Due to the sensitive location of these nodes, a fully compatible environment with the cluster is not necessary in order that these systems can be patched more aggressively than the rest of the cluster.

[TR-1] The O:eror shall describe the architecture of at least four data transfer nodes providing high-speed external network connectivity.

5.2.4 External Connectivity

[TR-1] The Offeror shall provide a high-bandwidth and resilient solution for connectivity between the high-speed compute fabric and the ESIF data center network. The solution shall support simultaneous transfers and provide aggregate throughput commensurate with the expected system IO and data-movement workloads.

5.2.5 O=-premise External Connectivity

[TR-3] If any system components or resources are deployed off-premises, the Offeror shall describe external connectivity between NLR and the off-premises resource. Any Offeror-provided fiber connection shall terminate in the ESIF Data Center.

5.3 User Environment

5.3.1 User Environment Software Management

5.3.1.1 [TR-3] Module Framework The O:eror shall describe the mechanisms by which the provided user software libraries (mathematical, I/O, and AI/ML; 5.2.2) can be installed for each provided toolchain supported on ESIF-HPC-4. Note how multiple versions of packages are managed and made accessible.

5.3.1.2 [TR-3] Container Build and Execution Capabilities Describe support for unprivileged users to build and run containerized applications while leveraging provided system hardware and software features. Note whether the proposed container technologies support, or comply with, Open Container Initiative (OCI) standards.

5.3.1.3 [TR-3] Containerized Programming Toolchain To support portable development workflows, NLR expects ESIF-HPC-4 to supply a containerized programming toolchain (i.e., compiler(s) and MPI) that mirrors the host environment and can be used both on and o: ESIF-HPC-4. Please describe the o:ering and its limitations.

5.3.2 Scientific Programming and Development

5.3.2.1 Compilers and Programming Languages

ESIF-HPC-4 must support a broad set of compiled and interpreted languages. The O:eror shall describe how:

5.3.2.1.1 [TR-3] The latest ISO standards for C and C++ are supported, noting any partial or missing features.

5.3.2.1.2 [TR-3] The O:eror shall describe how more than one programming toolchain to program all compute node types.

5.3.2.1.3 [TR-2] For compilers requiring licenses, provide separate pricing for at least 10 seats and temporary licensing su:icient for benchmarking and acceptance testing.

5.3.2.2 CPU threading

5.3.2.2.1 [TR-1] The system must support OpenMP threading. The O:eror shall describe any limitations in OpenMP threading support, if applicable.

5.3.2.3 Accelerator programming

5.3.2.3.1 [TR-1] The O:eror shall describe support for accelerator-based programming platforms and/or APIs, including any associated tools for conversion between di:erent stacks (if applicable).

5.3.2.3.2 [TR-1] Describe available programming models for directive-based accelerator o:loading.

5.3.2.4 AI/ML Libraries and Frameworks

[TR-3] The O:eror shall describe any optimized frameworks and/or libraries available to maximize performance for AI/ML tasks such as distributed training, inference, and hyperparameter tuning.

Describe how any provided frameworks and/or libraries integrate with and utilize the AI Optimized Storage (section 6.2).

5.3.3 Mathematical and I/O Libraries

5.3.3.1 Optimized Mathematical Libraries

[TR-1] The O:eror will provide optimized BLAS, LAPACK, SCALAPACK, and FFT libraries for each compute node type and supported precisions. Please describe the o:ering and its limitations.

5.3.3.2 Optimized I/O Libraries

[TR-1] The O:eror shall describe support for optimized I/O libraries (e.g., NETCDF, p-NETCDF, HDF5) on each compute node type, including any parallel I/O capabilities and limitations.

5.4 Performance Analysis Tools and Debuggers

NLR currently uses several third-party performance analysis and debugging tools (e.g., Linaro Forge) to support existing and emerging HPC workflows. ESIF-HPC-4 must be able to support these types of tools for ongoing code maintenance and support by NLR sta:.

5.4.1 Performance Profiling

5.4.1.1 [TR-2] Profiling Tools The O:eror shall describe how the system provides performance profiling and tracing tools that operate across all supported programming models (e.g. MPI, OpenMP, accelerator programming) and compute node types.

5.4.1.2 [TR-1] Hardware Counters and Multi-node profiling The O:eror shall describe access to hardware counters and support for multi-node profiling.

5.4.1.3 [TR-2] Storage and I/O Performance Profiling The O:eror shall describe available tools for profiling storage and I/O performance, including collection of throughput and metadata operation metrics relevant to user workflows.

5.4.2 Debugging

[TR-2] The O:eror shall describe the provided debugging environment, including graphical and command-line interfaces, and how it supports debugging of parallel applications across multiple nodes, threads, and accelerators.

6.0 Storage Zone

The Storage Zone must support a wide range of computational workloads, each placing di:erent demands on the storage environment. Traditional modeling and simulation require a high-bandwidth, write-optimized parallel file system capable of supporting large data outputs, checkpoint/restart operations, and mixed I/O patterns ranging from small metadata-intensive files to large streaming transfers. At the same time, emerging AI/ML workflows introduce access patterns that di:er substantially from conventional HPC usage. These workloads often rely on small, random reads and may require higher IOPS and lower latency characteristics. To accommodate these needs, NLR is willing to consider an additional storage tier optimized for AI and data-driven workflows, capable of serving as a shared read resource across many nodes within a job.

Both traditional and AI-focused storage use cases must provide consistent performance at varying scales, from single node jobs to workloads utilizing a substantial portion of the system, ensuring that users can rely on predictable behavior across ESIF-HPC-4’s diverse computational workloads.

All required file systems (e.g. Scratch, AI-Optimized Storage, Projects and Home) must be mountable across the Access and Compute Zones in a consistent manner.

6.1 Parallel File System

The system must include a high-performance parallel file system supporting modeling and simulation workloads with high bandwidth and mixed I/O patterns.

6.1.1 Architecture

[TR-1] The O:eror shall describe the architecture of the proposed parallel file system, including key hardware and software components, network connectivity, supported interfaces (e.g. POSIX), and any required client-side modules.

6.1.2 Capacity and Performance

[TR-1] The parallel file system should exceed a capacity of 50PB and a sustained read and write performance of 500Gb/s. The O:eror shall describe how the system meets or exceeds these minimum requirements, and how performance scales from single-node jobs to system-level workloads.

6.1.3 Resilience and Failure Behavior

[TR-1] The O:eror shall describe data-protection mechanisms, expected rebuild behavior, and performance impacts while operating in degraded or recovery modes.

6.1.4 Management, Monitoring and Diagnostics

[TR-1] The O:eror shall describe the administrative tools provided for performance monitoring, diagnostic analysis, quota enforcement, and usage reporting.

6.1.5 Expandability

[TR-1] The O:eror shall describe how additional storage or metadata capacity may be added after deployment, including any performance implications or downtime requirements.

6.1.6 Security Controls

[TR-1] The O:eror shall describe security features, including at-rest and in-transit encryption and any capabilities to restrict visibility of sensitive or project-scoped data.

6.2 AI-Optimized Storage

To support emerging AI, ML, and data-driven workloads, the system may include a second storage tier optimized for high-IOPS, low-latency access and shared dataset availability.

6.2.1 Architecture

[TR-1] The O:eror shall describe the architecture of the proposed AI-optimized storage tier, including hardware/software components, supported interfaces (e.g. POSIX, S3/object, APIs), and any client-side software requirements.

6.2.2 Capacity and Performance

[TR-1] The AI-optimized storage tier should exceed a capacity of 50PB and a sustained read and write performance of 100 GB/s. The O:eror shall describe how the system meets or exceeds these minimum requirements, and how it serves high-IOPS, small-file and random read access patterns characteristic of AI/ML workloads.

6.2.3 Data Sharing and Protection

[TR-1] The O:eror shall describe capabilities for snapshots, project level permissions, and isolation mechanisms preventing unauthorized access or data leakage across project boundaries.

6.2.4 Policy, Lifecycle and Monitoring Tools

[TR-1] The O:eror shall describe usage reporting, quota enforcement options, and any policy engine for automated data lifecycle operations, as well as monitoring and diagnostic capabilities.

6.2.5 Expandability

[TR-1] The O:eror shall describe how capacity and metadata services may be expanded over time, along with expected impacts on availability and performance.

6.2.6 Security Controls

[TR-1] The O:eror shall describe security capabilities relevant to collaborative and AI storage environments, including encryption, access controls, and audit/logging mechanisms.

7.0 Management Zone

7.1 System Management Architecture

The Management Zone includes the tools, networks, services, and security mechanism required to configure, operate, monitor, and maintain the ESIF-HPC-4 system. Access to the Management Zone shall be restricted to authorized administrative personnel.

7.1.1 Management Infrastructure

[TR-1] The O:eror shall describe the overall management architecture, including service nodes, management servers, management storage, and all software components supporting management operations.

7.1.2 Configuration and Cluster Management

[TR-1] The O:eror shall describe provided configuration management capability, either built-in or integrable with existing tools, and how system configuration consistency is maintained across upgrades.

7.1.3 Management Network

[TR-1] The O:eror shall describe the management network topology, how management tra:ic is isolated from user tra:ic, and whether a dedicated out-of-band network is included.

7.1.4 Remote Administration

[TR-2] The O:eror shall describe remote console and power control capability (e.g. IPMI, Redfish)

7.1.5 High-Availability and Upgrade Strategy

[TR-2] The O:eror shall describe how management services remain operational during upgrades or maintenance, and which operations require planned downtime.

7.2 Operating System and Workload Management

7.2.1 Base Operating System

[TR-1] The O:eror shall describe the Linux distribution used throughout the system, including licensing model, update cadence and kernel customization (if any).

7.2.2 OS Upgrade and Patching

[TR-1] The O:eror shall describe how OS upgrades, including security patches, can be performed while minimizing downtime and disruption to running workloads.

7.2.3 Job Scheduler and Resource Manager

7.2.3.1 [TR-1] Job Scheduler and Resource Manager The O:eror shall describe the provided job scheduler and resource manager. If Slurm is not o:ered, the O:eror shall describe functional equivalence and architecture-level advantages for ESIF-HPC- 4.

7.2.3.2 [TR-2] Shared Scheduling The O:eror shall describe how the scheduler supports multiple simultaneous jobs and users on a single node where relevant.

7.3 System Monitoring

NLR will be active in the design, implementation and usage of cluster monitoring infrastructure.

Data collected may be exported from the system for use as research data and may be linked with power and energy usage and jobs data.

7.3.1 Monitoring Capabilities

[TR-1] The O:eror shall describe provided monitoring tools capable of collecting system health, hardware status, environmental metrics, and performance information from compute, management, storage, network components, as well as available interfaces for integrating facility-level telemetry (e.g. CDU temperatures, flow rates, or power data) when exposed through standard protocols.

7.3.2 Sensors and Telemetry

[TR-1] The O:eror shall describe available sensors, supported telemetry protocols (e.g. Redfish, IPMI, SNMP, Modbus, OS-level monitoring), and maximum supported polling frequencies.

7.3.3 Data Export and APIs

[TR-2] The O:eror shall describe APIs or other mechanisms allowing NLR to export telemetry data to external systems (e.g. Prometheus, Grafana, research workflows).

7.3.4 Logging Infrastructure

[TR-2] The O:eror shall describe how system logs are aggregated and retained, with capacity to store at least seven days of cluster-wide logs without impacting system operation.

7.4 Power and Energy

7.4.1 Power and Energy Measurement Interfaces

[TR-2] The O:eror shall describe system- or rack-level interfaces capable of reporting node, CPU, GPU, memory, and facility-level power and energy metrics.

7.4.2 Power and Energy Control Features

[TR-3] If available, the O:eror shall describe optional control interfaces such as system-level power caps or cooling-related telemetry*(e.g. water temperature, flow).

7.5 Security

ESIF-HPC-4 will be a Federally owned system and NLR will be required to operate it in a secure manner. NIST publication 800 53 o:ers guidance on the operating of Federal systems and is one of the security frameworks NLR basis its security policy. Header designations below are taken from the NIST 800 53 documentation.

7.5.1 Preventing Program Execution

[TR-1] CM-7(2): Prevent Program Execution: Describe how o:eror provided software and kernel drivers can be operated with standard Linux security architecture tools like selinux.

7.5.2 Inventory

[TR-1] CM-8(1): Updates During Installations / Removals: Upon delivery, updates and upgrades inventory will be provided or updated.

7.5.3 Backup and Restoration

[TR-1] CP-9 System Backup; CP-9(1): Testing for Reliability and Integrity; CP-9(2): Test Restoration Using Sampling; CP-10: System Recovery and Reconstitution; CP-10(2): Transaction Recovery; CP- 10(4): Restore Within Time Period: Identify critical system level software, license, applications, middleware and configurations. Describe the process for backing up and restoring.

7.5.4 Replaceable Media

[TR-1] MP-6 Media Sanitization: O:eror shall describe the process of replacing media. NLR may return used media if properly sanitized, otherwise it will be destroyed.

7.5.5 Security Notification and Remediation

[TR-1] SI-2 Flaw Remediation; SI-7 Security Alerts, Advisories, and Directives: The O:eror shall describe its vulnerability handling and patch management process for vendor-supplied software and firmware, including expected timelines from confirmed availability for:

• Non-critical vulnerabilities

• Vulnerabilities listed in the CISA Known Exploited Vulnerabilities (KEV) catalog

• Common Vulnerabilities and Exposures (CVEs) with a score of Critical or High (per the current CVSS version of CVSS by NVD).

7.5.6 Cryptographic Protection and Remote access

[TR-1] SC-12: Cryptographic Key Establishment and Management; SC-13: Cryptographic Protection;

AC-17(2): Protection of Confidentiality and Integrity Using Encryption: The system will use encrypted protocols for remote and web logins, including switches and BMC.

7.5.7 Monitoring and Endpoint Protection

[TR-1] SI-4(1): System-wide Intrusion Detection System; SI-4(2): Automated Tools and Mechanisms for Real-time Analysis; SI-4(3): Automated Tool and Mechanism Integration; NLR will install EDR (Endpoint Detection and Response) agents. Describe any known interaction issues with O:eror software from EDR tools.

7.5.8 Baselines

[TR-2] CM-2: Baseline Configuration: Describe the recommended secure baseline configuration for the system and how it is validated and documented. The O:eror shall describe methods to assess configuration drift and to support auditability of the system’s current security-relevant configuration state.

7.5.9 Supply Chain Risk Management

[TR-1] SR-08, SR-10; SR-11; SR-12; Packaging of materials should be done in a way to easily detect tampering. Anti-counterfeit measures should be implemented. Safe disposal procedures should be followed to prevent the leaking of data and cryptographic keys. NLR should have early notification of compromise or potential compromises to the supply chain that could adversely a:ect the system or components of the system.

8.0 Benchmarks

Benchmark performance is a critical component of evaluating the proposed ESIF-HPC-4 system architecture. NLR will assess how well real scientific applications, data-intensive workflows, and key microbenchmarks perform on the o:ered solution. The O:eror is required to provide benchmark results for the application, microbenchmark, and I/O test defined in Tables 2-4, using the instructions and input configurations provided in the ESIF-HPC-4 benchmark repository ( https://github.com/NatLabRockies/ESIFHPC4/).

NLR provides three categories for benchmark runs: baseline, ported, and optimized; to capture both architectural suitability and potential performance improvements achievable with reasonable development e:ort. These categories are defined to ensure comparability across proposals and to reflect realistic expectation for system readiness and application support at delivery.

The O:eror shall return benchmark results using the ESIF-HPC-4 Benchmark Results spreadsheet and shall include a full benchmark report in the Technical Volume Appendix. The O:eror is responsible for providing temporary licenses for any compilers or tools used to generate benchmark results that are not part of the delivered system.

An application throughput improvement metric derived from a subset of application benchmarks is used to evaluate system-level capability relative to the current Kestrel system. A detailed description of the application throughput improvement metric is provided in Appendix A. The application throughput improvement benchmark configurations represent realistic NLR workflow sizes and must demonstrate improvement over the provided reference performance results.

NLR encourages the O:eror to use best practice methods to obtain reliable benchmark results within the allowed modification categories. The O:eror shall adhere to benchmark-specific instructions and constraints documented in each benchmark’s README.md file in the ESIF-HPC-4 repository.

8.1 Benchmark Categories

We define the following general rules for baseline, ported, and optimized benchmark results in the benchmark response:

• Baseline (as-is): no source-code modifications permitted. Library substitutions permitted if those libraries will be available at system delivery. Reasonable changes to compilation options permitted unless otherwise prohibited by the benchmark README.

• Ported: minimal source code changes required solely to enable execution on the proposed architecture are permitted. Examples include addition or modification of directives or pragmas, and/or replacement of existing architecture-specific language constructs with another…