A-2 Requirements Document.pdf

Statement of Requirements

A100 GPU Cluster

I. BACKGROUND

The National Institute of Standard and Technology (NIST) Software and Systems Divisions (SSD) and Computational Mathematics Divisions (CMD) are examining software abstractions aimed at taking advantage of multi-core CPUs and multiple GPUs, available in the same compute node, and coupling these abstractions to several important scientific applications. The abstractions that have been identified, multithreaded asynchronous dataflow graphs, simplify the task of developing performance-oriented software. These abstractions treat parallelism as a first-class citizen in the software development effort. They allow developers to gain insight into how to orchestrate the various tasks in their applications on CPUs and GPUs to obtain performance where overlapping data motion and compute tasks is essential.

On the applications side, it is important to show that these software abstractions can lead to concrete increases in performance in real-world applications. NIST is currently developing a suite of programs designed to compute the properties of molecules exposed to intense, attosecond (10-18 s) light pulses.

These programs are compute-intensive but are not yet highly parallel.

It is necessary for developers of these abstractions and applications to have a platform that delivers performance. This platform must also provide sufficient data throughput from hardware to maintain utilization by efficiently sending data between the central processing units, graphics processing units, and high-speed network interconnect.

In support of this mission, the acquisition of a cluster of 4 compute nodes and high-speed low latency networking between the machines is required. Two of the compute nodes must feature four SXM4 A100 GPUs, and the other two compute nodes must feature two PCIe 4.0 A100 GPUs. Additionally, both systems must feature high-throughput NVMe SSDs that utilize PCIe gen 4, one or two high-core count CPUs, and high throughput 200 Gbit InfiniBand Networking.

II. SCOPE

The scope of this procurement is the provision of 4 (four) compute nodes and high-speed low latency networking with a warranty.

III. SPECIFICATIONS

CLIN 0001: The contractor shall provide two (2) NVIDIA 2U SXM4 GPU Servers meeting the following minimum specifications:

• 2U Rackmount Server with a minimum 2200W power supply and at least four 2.5” drive bays

• Motherboard for dual socket EPYC ROME/Milan Processors ONLY, which contains:

o At least sixteen slots of RAM for a minimum of DDR4-3200 o At least four PCIe Gen 4 x16 slots o At least one PCIe Gen with 4 x4, x8, or x16 slots

• Two AMD EPYC 7742 ONLY CPUs with at least 64-cores at a minimum of 2.25 GHz each

• At least sixteen 64GB DDR4 3200 MHz or more of RAM (for a total of 1 TB)

• At least four NVIDIA A100 SXM4 GPUs ONLY with 40 GB GPU Memory

• At least two SATA 6 Gb/s 2.5” SSDs with a minimum of 960 GB each

• At least two 2.5” PCIe 4.0 NVMe SSDs with a minimum of 1.92 TB each

• At least One MCX653106A-HDAT ConnectX-6 VPI Adapter Card HDR NVIDIA InfiniBand ONLY and

200 GbE Dual-Port QSFP56 PCIe4.0 x16

• Ubuntu 18.04 operating system (or equivalent Linux operating system) installed on SATA SSD

• System must be fully assembled and tested

CIN 0002: The contractor shall provide two (2) 2U PCIe 4.0 GPU Servers meeting the following minimum specifications:

• 2U Rackmount Server with a minimum 1600W power supply and at least six 3.5” (or 2.5”) drive bays

• Motherboard for single socket EPYC ROME/Milan Processors ONLY, which contains:

o At least eight slots of RAM for a minimum of DDR4-3200 o At least four double-width PCIe Gen 4 x16 slots o At least two PCIe Gen 4 x16 slots

• One EPYC 7742 CPU ONLY with at least 64-cores at a minimum of 2.25 GHz

• At least 8ight DDR4 3200 MHz with a minimum of 64 GB RAM each (total 512 GB)

• At least two NVIDIA A100 PCIe 4.0 GPUs ONLY with 40 GB GPU Memory

• At least two SATA 6 Gb/s 2.5” SSDs with a minimum of 960 GB each

• At least two 2.5” PCIe 4.0 NVMe SSDs with a minimum of 1.92 TB each

• At least one MCX653106A-HDAT ConnectX-6 VPI Adapter Card HDR NVIDIA InfiniBand ONLY and

200 GbE Dual-Port QSFP56 PCIe4.0 x16

• Ubuntu 18.04 operating system (or equivalent Linux operating system) installed on SATA SSD

• System must be fully assembled and tested

CLIN 0003: The contractor shall provide one (1) Networking Switch and a minimum of ten (10) Cables meeting the following minimum specifications:

• One MQM8790-HS2F Quantum HDR NVIDIA InfiniBand Switch with 40 QSFP56 Ports

• At least ten (10) 2-meter HDR NVIDIA InfiniBand 200 gb/s QSFP56 cables

IV. WARRANTY

The Contractor shall provide a one-year manufacturer’s warranty for parts and labor to commence after government acceptance.

V. DELIVERY

Delivery of the hardware and software shall be made F.O.B Destination to the address below. Delivery shall be scheduled at a time mutually agreed upon by the Government and Contractor but not later than 90 days after receipt of order (ARO).

NIST

100 Bureau Drive

Building 301 Shipping & Receiving Gaithersburg, MD 20899

Due to ongoing limited site access of the NIST facilities, the Contractor SHALL coordinate and confirm the delivery date and procedures in writing with the Designated Government Official prior to ANY shipment to NIST. Failure to confirm this ability to receive products shall release the Government of any liability.

VI. PLACE OF PERFORMANCE

All work shall be completed at the Contractor’s facility. Delivery of the equipment shall be to NIST at the delivery address stated above. Normal duty hours are Monday through Friday, 7:00 a.m. to 3:00 p.m.

with the exception of Federal holidays.

VII. INSPECTION/ACCEPTANCE

The NIST TPOC will verify that all original manufacturer performance requirements are met. Inspection and acceptance shall take place not later than fourteen (14) days after delivery.

VIII. PAYMENT SCHEDULE

The Contractor shall be paid, in accordance with Net 30-day payment terms, upon receipt and acceptance of a proper invoice, in accordance with the following schedule:

1. 100% after receipt and acceptance by the TPOC of the fully delivered systems.

2. The Government anticipates inspection will occur upon:

a. After the testing procedures set forth in this document have been complete

NOTE: Partial shipments and partial invoices will not be accepted.