NIH-CIT-HPC RFI vF.docx

SOURCES SOUGHT NOTICE/REQUEST FOR INFORMATION

NATIONAL INSTITUTES OF HEALTH (NIH)

NATIONAL LIBRARY OF MEDICINE (NLM), CENTER FOR INFORMATION TECHNOLOGY (CIT)

TITLE: NIH Biowulf Cluster Technology Refresh This announcement constitutes a Sources Sought Notice (SSN)/Request for Information (RFI) (market survey). This is NOT a Request for Quote (RFQ) or a Request for Proposal (RFP). This announcement is for information and planning purposes only and is not to be construed as a commitment by the Government, implied or otherwise, to issue a solicitation or award a contract.

Introduction and Background

The National Library of Medicine (NLM), Office of Acquisitions on behalf of the National Institutes of Health (NIH) National Library of Medicine (NLM) and the NIH Center for Information Technology (CIT) is requesting information from qualified vendors regarding their capability to supply an incremental update to the Biowulf High Performance Computing (HPC) cluster (https://hpc.nih.gov). This SSN/RFI is aimed towards system integrators who can provide HPC compute and networking hardware and integrate it with the existing Biowulf hardware and network infrastructure. Storage is beyond the scope of this RFI, however, we seek information about how refreshed compute and networking hardware will integrate with existing high performance storage systems.

The information supplied in response to this RFI will guide the NIH in developing detailed specifications for a refresh of approximately 20% of the existing Biowulf compute nodes and associated high-performance, low-latency networking hardware. We do not seek information about software (either at the system or application level) since all hardware will be provisioned with the existing Biowulf software stack. Integration with the portion of the system that will remain in place will be critical to the overall success of the project. Therefore, responses should clearly state what information NIH would need to supply for a successful integration effort. Additionally, we request information about specific technologies that NIH should consider for the refresh effort. This information may be used to develop an RFP or RFQ for the system refresh.

In addition, we request that potential offerors provide their business size and what NAICS code equipment and integration and support services that could be proposed would fall under. If any equipment or services would potentially be sourced from a subcontractor, please provide that information as well, if known. The additional detail will further assist in the potential development of an RFP or RFQ.

Current Biowulf Infrastructure and Capabilities

Biowulf is the largest HPC system in the world completely dedicated to supporting biomedical sciences and has the distinction of being the first such supercomputer listed among the Top 500 most powerful computers in the world. Biowulf operates primarily as a capacity system, meaning that it specializes in executing many simultaneous jobs, each of which uses a small percentage of the overall computing power of the system. This contrasts with a capability system, which runs relatively few jobs, but with each job using a substantial portion (potentially all) of the system’s resources.

Biowulf supports more than 3,000 users across all 27 of NIH’s Institutes and Centers (ICs). The Biowulf staff makes over 900 distinct scientific applications available to the user base, and many users also install or develop their own applications. These applications support a broad range of sub-disciplines within the life sciences such as molecular modelling and structural biology, genomics, cryo-electron microscopy, statistical modelling, and advanced artificial intelligence and machine learning. For a complete list of supported applications, please see https://hpc.nih.gov/apps/.

One of the key features of Biowulf is that it is designed to support the broad range of applications described above. Some of these applications are very compute intensive and require fast central processing units (CPUs) with advanced integer and floating-point mathematical support. Other applications, particularly those that rely on databases, are memory bound and require fast access to system memory to perform rapid queries. Still others are input/output bound and must read and write a large amount of data from fast storage systems. Many applications and workloads combine some or all of these requirements.

In the last several years, accelerators, especially GPUs, have become an important part of the life sciences computing ecosystem. Numerous simulation and data processing applications now support GPU acceleration, typically either via Nvidia’s CUDA toolkit or the more open standard OpenCL. Furthermore, the Biowulf user community is increasingly adopting deep learning and artificial intelligence techniques for tasks such as statistical analysis and structure prediction. As a result, the Biowulf staff has deployed over 800 GPUs in the cluster.

Biowulf in total consists of approximately 4,100 compute nodes with over 100,000 cores, including the roughly 20% of the system that will be refreshed. To ensure that both latency sensitive MPI jobs and storage intensive workloads run with the best possible performance, the Biowulf staff has constructed a large-scale InfiniBand network to interconnect these nodes and connect them with over 40 PiB of primary storage. We request information on future networking directions that are compatible with the existing infrastructure and that will integrate seamlessly with it. More details about specific requirements and the existing configuration are given below.

Compute Requirements

We seek information on the range of different compute node configurations that potential offerors can supply. Specifically, we are interested in understanding which offerors can supply a wide variety of standard CPU nodes and GPU nodes. Given the rapid pace of technological change, we seek to understand how rapidly product lines are refreshed. Some directed questions in that regard are: Are recent processors such as AMD “Genoa” and Intel “Sapphire Rapids” available in quantity? Which current GPUs are currently being shipped to customers?

Beyond basic details surrounding product availability, we seek recommendations on technological directions to investigate for a potential RFP or RFQ. The current Biowulf node configuration is highly heterogenous, including both AMD and Intel GPUs as well as several generations of Nvidia GPUs. For full details, please see https://hpc.nih.gov/systems/hardware.html. We request information on potential configurations or a range of configurations for standard CPU-based compute nodes, GPU-accelerated nodes, and some large memory systems. Specific guidelines for each type of node follow:

Standard compute nodes

· Due to the memory-intensive nature of many Biowulf jobs, nodes should be configured with 8 GiB of memory per CPU core. However, there should be flexibility to allow the memory to core ratio to be adjusted in either direction when developing an RFP or RFQ.

· To allow sufficient density, nodes should have at least 64 CPU cores. Higher densities are possible, and we are interested in understanding the advantages and disadvantages of increasing the number of cores per node.

· Nodes must be able to support remote direct memory address (RDMA) for MPI communication and for interaction with storage systems.

· 1 Gbps Ethernet connections for management and lights out management (e.g. via IPMI or dedicated baseboard management controller [BMC] software) should be included. The BMC software must support remote power and remote console capability and be accessible via both HTTPS and SSH.

· The presence of large-capacity, fast local scratch space is becoming increasingly important for many workloads. We seek information on cost effective options for provisioning this. Three (3) TiB of local scratch space is the minimum requirement. Local scratch space need not be redundant.

GPU compute nodes

· The Biowulf staff has been configuring GPU nodes with four (4) GPU devices each, and this should be considered the minimum number per node. Given the Biowulf application profile, we are interested in the potential advantages and cost considerations of deploying systems with 8 GPUs.

· We seek information about potential offerings that include composability, that is, the ability to assign GPU resources to compute nodes in a dynamic fashion.

· Historically, Biowulf has deployed Nvidia GPUs. We seek information about GPUs both from Nvidia and from other suppliers. Specific data about relative performance and software compatibility would be very helpful.

· Requirements regarding the memory to core ratio, RDMA and management connectivity, and local scratch space are identical to those for standard compute nodes.

Large memory nodes

· Large memory nodes should support at least 4 TiB of system memory.

· At least 64 compute cores should be present on large memory nodes.

· Requirements regarding the memory to core ratio, RDMA and management connectivity, and local scratch space are identical to those for standard compute nodes.

Beyond the compute node hardware configuration itself, power and cooling are also concerns. A rack of CPU based nodes typically draws less than 50 KW of peak power, while a rack of GPU nodes typically draws under 30 KW. More recent Biowulf nodes use rear door heat exchangers to dissipate most of the heat load. These heat exchangers are supplied by cooling distribution units (CDUs) that interconnect with them via under-floor manifolds.

We request information on the heat load and power requirement for racks of standard CPU nodes and GPU nodes. Large memory nodes would likely be intermixed with one of the other node types and not have a dedicated rack. We further request information on the cooling options for each type of rack, especially whether direct liquid cooling would be an option, and if so, compatibility with our existing cooling infrastructure. Since we have an overall facility power cap, details about how the power and cooling requirements would change with different potential system configuration would be extremely helpful, if available.

Network Requirements

In past acquisitions, HPC has procured both the data and management networking equipment along with the compute nodes. We seek to understand potential options that maintain compatibility with existing infrastructure.

The current Biowulf data network is primarily InfiniBand based. It consists of a mix of EDR and HDR switches organized in a fat-tree topology. Most of the switches are unmanaged; an external subnet manager running version 6.2 of Mellanox’s UFM software in a redundant configuration is used to manage the fabric. A cluster of four (4) InfiniBand to Ethernet gateways (model SX6036G) provides connectivity to a relatively small Ethernet data network that hosts the HPC service nodes and contains the 100 Gbps uplink to the NIH backbone. See the bottom of https://hpc.nih.gov/systems/hardware.html for a high-level diagram of the network architecture.

The top layer of the fat-tree topology consists of six (6) Nvidia Quantum HDR switches that connect directly to 12 second layer spine switches (also Nvidia Quantum) that ultimately provide connectivity to newer compute nodes, three (3) Mellanox CS7500 director switches that interconnect older-generation compute nodes, and 14 storage switches for native-InfiniBand access to GPFS over RDMA and NFS over RDMA based disk systems.

New equipment would need to maintain compatibility with this existing infrastructure. We would strongly prefer that InfiniBand be retained as the primary network technology. In that case, it is likely that we would design a new spine layer for the refreshed hardware and connect it to the existing aggregation layer. However, we are interested in information on alternative approaches if there is a noticeable price or technological advantage present. We would expect, at minimum, 200 Gbps connectivity to each node along with the ability to perform inter-node MPI communication via RDMA as well as to be able to use native RDMA transport to storage systems.

Required Installation, Integration, and Support Services

We seek information about integration services available from potential offerors. Typically, Biowulf nodes are delivered pre-installed in their compute racks (with PDUs included) and with all rack-internal connections pre-made. Additionally, BMC addresses are pre-configured with values supplied in advance by NIH. The supplier must provide a list of all data and management network hardware addresses in advance so that our internal provisioning system may be made ready to provision the nodes. It is acceptable and desired for the nodes to arrive on-site with blank hard disks and no operating system loaded. Once on-site in the NIH Data Center, the supplier is responsible for making the required network connections to existing infrastructure, as specified in the order document for the system. Typically, as part of this service, the supplier will provide documentation on all internal and external cable connections in the form of a spreadsheet. We wish to know what offerors are willing to offer similar service and what documentation would be required from NIH to allow successful execution.

We also request information on support offerings. Our preference is for full support of all compute and network hardware and system firmware for a period of five years. This support would include both remote and on-site break-fix and other technical support available on an 8-5 Monday through Friday schedule. Respondents should clearly state their standard policies and practices regarding which system components are considered field replaceable by Biowulf systems personnel and which require a technician visit.