CF-204100_QuestionsAndAnswers_2020-02-28.pdf
PDF 129 KB Posted
- Attached to
- High Performance Supercomputer Cluster
- Solicitation number
- CF-204100
- Issued by
- Wisconsin
About this file
This is a Questions and Answers document related to a High Performance Supercomputer Cluster procurement by the University of Wisconsin Eau Claire's Purchasing Services Department on behalf of the Chemistry and Materials Science and Engineering department. The bid solicitation (CF-204100) requests a high-performance computing system comprised of 38 two-socket compute nodes equipped with 12 V100 GPUs, storage file server nodes, and head/login nodes. The cluster must utilize Intel CPUs with a minimum of 20 cores at 2.4GHz, ConnectX6 InfiniBand cards supporting HDR 100 connectivity, and CentOS 8.x operating system. Critical specifications address power consumption, physical footprint, and memory configuration requirements. The document clarifies that the cluster will be housed in two university-supplied racks, each capable of handling 11kVA for a combined total power limit of 22kVA, and that the university will provide Power Distribution Units (PDUs) and InfiniBand switches.
The vendor is required to install and support the operating system, SLURM job scheduler, and computational software packages including VASP and MedeA, which will be provided by the university. Memory configuration must follow optimal server sizing standards, with vendors instructed to round up to the next optimal memory size per node. All InfiniBand cards and related firmware must be uniform in model and version across the cluster. Storage and head node CPUs should utilize twenty-core processors. The university does not require interoperability with existing InfiniBand infrastructure, and vendor-provided InfiniBand equipment must be homogeneous throughout the system to ensure consistency and ease of maintenance.
View the file
Other files
On GovTribe
Work with this file
- Download the original file
- Contacts named in this file
- Similar government files
- Ask GovTribe AI about this file
Text version
High Performance Supercomputer Cluster
CF‐204100
Bid Questions
Q1: The requested equipment will use quite a bit more than the 15kVA power limit. Can this limit be raised? Or can the single‐rack requirement be relaxed to two 15kVA racks?
A1: Equipment can go in two racks that we will supply. Each rack can handle 11kVA so the total power limit can go to 22kVA
Q2: Does the vendor need to provide PDUs or are those included with the University provided racks?
If the vendor needs to provide PDUs, which receptacles are available at the data center?
A2: UW‐Eau Claire will provide the PDUs
Q3: Can you confirm the Infiniband requirement is for ConnectX‐6 HDR100 and not ConnectX‐6
HDR? HDR would require PCIe 4.0, which is not available on Intel platforms.
A3: We are looking for ConnectX‐6 cards that support HDR 100.
Q4: In Section 6 – Specifications
It stated cluster cannot consume over 15kVA of power:
The bid calls for 38 total two socket nodes with 12 V100 GPUs – this = at least 18kVA and most likely closer to 21kVA
(Each two socket server without GPUs is 400‐500w and each GPU is 250w (300w if SXM2)
The desired configuration will be over 15kVA in the one rack.
Shall we go to two racks? Or less nodes?
A4: Each rack can handle 11kVA so the total power limit can go to 22kVA
(CONTINUED…)
Q5: Memory per core:
SkyLake and now CascadeLake CPUs have six memory channels per CPU (12 ‐ per server) and all must be populated with = size memory dimms to achieve full memory bandwidth and to get desired benchmark results.
The stated Intel CPU desired is 20core and 2.4GHz min.
With 40cores per server then:
Using 8GB dimms = 96 GB per server = 2.4GBs per core
Using 16GB dimms = 192GB per server = 4.8GBs per core
Using 32GB dimms = 384GB per server = 9.6GBs per core
Using 64GBdimms = 768GB per server = 19.2GBs per core
These are the memory per node options and memory per core options.
Which of these are desired for the different node options requested since 6GB, 8GB, 16GB per core don’t quite match up.
Should we assume you want to round up to the next optimal server memory size?
A5: Yes, please round up to the next optimal server memory size
Q6: Do you already have the software requested? We want to account for this element and would like to know if the software will need to be considered as supported by the RFB award winner for implementation and support moving forward?
Are you planning to migrate the software from the current system(s)?
A6: We will be installing software on the new cluster. Vendor will be responsible for installing
CentOS8.x, SLURM and dependencies and computational packages VASP and MedeA, which we will provide.
Q7: What are the core count requirements for the CPUs on the Storage File Server Node and the
Head and Login Node?
We saw this was called out in the compute nodes; we just want to make sure that your needs are realized in the Storage nodes as well.
A7: Please use twenty‐core CPUs.
(CONTINUED…)
Q8: What are the quantities and cable lengths needed for the InfiniBand switches?
Do we need to account for interoperability with existing InfiniBand switches?
A8: No
Q9: Firmware, and model numbers are my main concern here.
Are the existing Mellanox cards all the same or are the disparate models/versions?
A9: We would expect what you are providing to all be the same
Q10: The bid document states on Page 26, Section 8, Attachment B, Specifications, item #1 “Cluster cannot consume over 15kVA of power”, and item #3 “Cluster must fit in one rack”. Our
Engineering team has run a calculation and based on the specifications that the overall cluster will exceed 15kVA of power. Nor‐Tech does have a solution, and we would be willing to submit that solution in our bid, but technically it would require an amendment to allow for us to provide one additional APC Rack which matches identically your current APC Rack. Otherwise, the overall node count would need to be reduced.
Please advise how we should proceed.
Q10: Each rack can handle 11kVA so the total power limit can go to 22kVA
File details come from the government source that posted it. Updated .