v3_SOW_FY25_LQ3_Attachment_A_Specification.docx

ATTACHMENT A

FERMI LQ3 InfiniBAND CLUSTER

TECHNICAL SPECIFICATION

FY2026

Table of Contents

Acquisition Summary

Fermi Research Alliance (FRA) intends to purchase a HPC farm (LQ3) of consisting of Infiniband 52 servers, Luster, and administrative nodes with specifications as detailed in this technical specification document.

Available at Fermilab are existing racks that vendors are required to use without cost for their proposed compute farm. Specifications for these existing racks are given in section B below.

All integration work must be performed at the Fermilab site. Vendors must either:

1. provide the labor for this integration work, or

2. subcontract to a third party;

Such vendor proposals must include a plan giving the schedule for the integration work and must name the subcontractor. The vendor is solely responsible for working with the chosen subcontractor and for meeting the stated integration schedule.

Vendors may submit multiple proposals, if several options can satisfy the requirements. Every proposed configuration should have its own set of documents as listed below:

1. Technical Proposal

a. Completed Attachment C “Technical Proposal” worksheet.

b. Single rack configuration which is either text or a diagram describing the layout of various components in the rack.

2. Power efficiency curves.

3. Completed Attachment D “Cost Proposal” worksheet including

a. Total cost for a quantity of 36 servers within the configuration

i. 1-12 optional servers (pricing match)

b. Total cost for 4 OSS (Luster) servers within the configuration

c. Total cost for 2 AMD MDS (Luster) servers

d. Total cost for 4 Admin nodes

e. Total cost NVIDIA 920-9B210-00FN-0M0- MQM9700-NS2F- NDR -64 Ports

f. Total cost for ALL IB cables, GbE switch(s), C13-C14 Power Cords and CAT6 cablesCompleted Attachment E “Vendor Questionnaire” worksheet.

4. A delivery and server integration plan including (if any) subcontractor information

5. A Statement of Warranty per the requirements of Attachment A, Section E.

A. Compute Server Specifications (QTY 40 base with 1-12 options)

A.1 Single Compute Server Configuration Summary:

1) 2 AMD EPYC Turin 9355 32Core/64Threads 3.55GHz 256MB

2) Motherboard compatible with #1 above

3) 24 Micron 64GB DDR5-6400 ECC RDIMM (1,536GB)

4) 2 Micron 5400 PRO 960GB 2.5" Internal SATA/600 SSD - Mirror

5) 2 Micron 7450 PRO 1.92TB 2.5" Internal SSD -Mirror

6) BROADCOM MEGARAID 9540-8i: 8 internal ports; one x8 SFF-8654; x8 PCI-e Gen 4.0; 12Gb/s SAS and 16GT/s PCIe Gen4 (NVMe) compliant; SAS3808 Controller; RAID 0, 00, 1, 10; SAS/SATA

7) CacheVault Power Module

8) NVIDIA ConnectX-7 MCX75310AAS-NEAT -1PORT 400Gb/s NDR IB - PCIe 5.0 x16, Secure boot, No Crypto, Tall

9) Onboard 10Ge or Intel® X550-AT2 10Gb/s 2-port LAN Card RJ45

10) Rackmount case with rails

11) 8 Compute servers installed on rails within rack.

12) Dual tool-less replaceable power supplies

13) Internal cooling conforming to processor manufacturer specifications

The following requirements apply to all configurations:

14) All systems must support stock RHEL/Alma/Rocky 9, including installation over network (PXE) kickstart. All devices (e.g., drives, raid, and network) should be available during installation

15) Tested Operating system (RHEL/Alma/Rocky 9) installed when delivered, with network DISABLED.

(ONBOOT=”no”)

16) Hostname, Ethernet, IPMI BMC addresses and IPMI password configured as provided by FRA.

A.2 Detailed Hardware Specifications The following hardware specifications apply to all configurations above unless noted:

A.2.1 MOTHERBOARD:

The proposed motherboard must have the following characteristics:

a) BIOS must allow system to boot without a keyboard and monitor

b) Hardware support for remote temperature measurements (processor and motherboard), voltage sensors, reset, power control, power cycling, and serial over LAN using IPMI 2.0 (or higher version) and Redfish

c) On-board video required

d) Dedicated on-board IPMI networking required

e) BIOS and network interface must support PXE network booting features

f) An ability to record BIOS settings from one motherboard and replicate the same settings on a different motherboard using a utility in Linux is highly desirable.

All motherboards supplied by the vendor must have the same version of all embedded firmware (BIOS, BMC, Ethernet, disk drives, etc.). All compute servers must have their BIOS parameters configured as specified by FRA. All compute servers must have IPMI parameters, including IP addresses, configured as specified by FRA; all IPMI sensor data repositories (SDR’s) must be preloaded so that sensor data (temperatures, fan speeds, voltages) can be read via IPMI from each compute server. All compute servers must have BIOS serial redirect configured to allow access to the BIOS screens using serial over LAN; serial redirect should be configured to “always” (i.e., not just during POST).

A list of BIOS and IPMI settings will be provided to the vendor when the purchase order is awarded.

A.2.3 MEMORY:

Must meet the following requirements:

a) To achieve highest memory bandwidth performance vendors must follow memory DIMM population rules as recommended by the CPU manufacturer.

Memory brand and part number must be on the approved list for the motherboard as provided by the motherboard manufacturer

A.2.4 SSD:

Minimum Requirements:

a) SSDs with SATA interface, “enterprise” grade

b) The system drive must have a warranty consistent with the Warranty Specifications in section E.1.

A.2.5 RACKMOUNT CASE:

A case meeting these requirements:

a) Cases that can be opened tool-free are preferred.

b) No multiple server chassis.

c) All ports must be labeled accurately.

d) Cable routing inside the case must not block the airflow of any fans.

e) The serial number must be easily readable from a label on either the front or the back of the case.

f) Maximum depth, measured from the front edge, should fit a 39-inch-deep FRA-provided rack.

g) Vendor must supply full specifications for the case in their bid

h) Dual tool-less replaceable power supplies are preferred.

A.2.6 POWER SUPPLY:

a) Must be U.L. Approved

b) Must be approved by the motherboard manufacturer.

c) Must be installed per the motherboard manufacturer’s specifications.

d) Must operate within the power supply vendor’s specifications.

e) Must have PFC circuitry

f) Dual tool-less replaceable power supplies are preferred.

g) Must have sufficient power output for the system configuration.

h) Must have 80% or higher efficiency at half-load. Higher efficiency solutions will be considered favorably during evaluation.

i) Vendor must provide power supply efficiency curve(s) that includes at least no-load (idle) and highest-load data points as measured during the execution of the “mprime” stress test and power-up; (https://www.mersenne.org/download/#stresstest).

j) Must support 120/208 VAC @ 60Hz.

k) The AC power cables supplied by the vendor should be short enough to minimize slack in the system racks. The power cables must be arranged within the rack in a manner that does not interfere with the air flow from the compute servers.

A.2.7 INTERNAL COOLING:

a) Units must have adequate airflow to cool all components to the component vendor’s specifications, assuming a computer room with nominal ASHRAE recommended 72 F ambient (cold aisle) air.

A.2.8 BIOS SPECIFICATIONS:

a) Vendor shall configure BIOS, Ethernet, and IPMI parameters according to FRA specifications. All compute servers delivered by the vendor must have the same revisions of all internal firmware (BIOS, Ethernet, BMC, disk drives etc.).

B. Luster OSS Server Specifications (QTY 4) Single OSS Luster Configuration Summary:

a) 4U Rack chassis,Front -24 x 3.5" Hot-swap SATA/SAS Bays, 2 x 2.5" hot-swap SATAIII Rear drive bays. 1+1 Redundant 1200W 80Plus Platinum Power Supply.

b) AMD 1 x Socket SP5 (LGA 6096), supports AMD EPYC 9005/ 9004 series Processors. 24 (12-channel, 1-DIMM per Channel) Memory slots.

c) AMD EPYC Turin 9135 16Core/32Threads 3.65GHz 64MB L3 Cache 200W Processor, PN:100-000001150

d) 8 Micron 16GB DDR5-6400 ECC RDIMM (128GB)

e) 2 Micron 5400 PRO 960GB SATA 2.5" SSD, OS Drives - Mirror

f) BROADCOM MegaRAID 9560-8I - 8-Port TRI-MODE RAID CONTROLLER.

g) CacheVault Power Module

h) 24 HGST 0F38652 DC HC560 20TB 512MB 512e SAS12Gb/s 7200rpm 3.5" HDD (No RAID, THIS IS FOR LUSTRE)

i) NVIDIA ConnectX-7 MCX75310AAS-NEAT -1PORT 400Gb/s NDR

C. Luster Admin Server Specifications (QTY 4)

Single Admin Sever Configuration Summary:

a) 1U Rack chassis, 1+1 Redundant 1200W 80Plus Platinum Power Supply. 4 x 2.5" hot-swap drive bays, AMD 1 xSocket SP5 (LGA 6096)

b) AMD EPYC Turin 9135 16Core/32Threads 3.65GHz 64MB L3 Cache 200W

c) 8 Micron 16GB DDR5-6400 ECC RDIMM (128GB)

d) 2 Micron 5400 PRO 960GB SATA 2.5" SSD, OS Drives - Mirror

e) BROADCOM MegaRAID 9560-8I - 8-Port TRI-MODE RAID CONTROLLER, 1 x8 SFF-8654 PCIe Gen 4.0

f) CacheVault Power Module

g) 1M x8 SFF-8654 to 2 x4 SFF-8643 Mini-SAS HD Cable NVIDIA ConnectX-7 MCX75310AAS-NEAT -1PORT 400Gb/s NDR IB - PCIe 5.0 x16, Secure boot, No Crypto, TallBracket

D. Luster AMD MDS Server Specifications (QTY 2) Single AMD MDS Luster Configuration Summary:

a) 1U Rack chassis, 1+1 Redundant 1200W 80Plus Platinum Power Supply. 12 x 2.5" hot-swap drive bays, AMD 1 xSocket SP5 (LGA 6096)

b) AMD EPYC Turin 9135 16Core/32Threads 3.65GHz 64MB L3 Cache 200W

c) 12 Micron 16GB DDR5-6400 ECC RDIMM (192GB)

d) 2 Micron 5400 PRO 960GB SATA 2.5" SSD, OS Drives - Mirror

e) 4 Micron 5400 MAX 960DB SSD 2.5" SATA/600 - 5 DWPD SSD 2.5" SATA/600 - 5 DWPD

f) BROADCOM MegaRAID 9560-8I - 8-Port TRI-MODE RAID CONTROLLER, 1 x8 SFF-8654 PCIe Gen 4.0

g) CacheVault Power Module

h) 1M x8 SFF-8654 to 2 x4 SFF-8643 Mini-SAS HD Cable NVIDIA ConnectX-7 MCX75310AAS-NEAT -1PORT 400Gb/s NDR IB - PCIe 5.0 x16, Secure boot, No Crypto, TallBracket

E. Infiniband Rack Component Specifications

1) 4 OSS Luster Servers

2) 2 MDS Luster Servers

3) 4 Admin Servers

4) Nvidia 920-9B210-00FN-0M0- MQM9700-NS2F (64 port)

5) 3 Netgear Ethernet 48 Port Managable Switch

6) InfiniBand Cables connectiong all the Compute Servers, Luster OSS

7) Servers, Luster Admin Server and Luster AMD MDS to the

8) MQM9700-NS2F

9) 1 NVIDIA passive Optical splitter cable, IB twin port NDR 800Gb/s to

10) 2x400Gb/s, OSFP to 2xOSFP, 10m fiber (Needed to connected to LQ2)1 NVIDIA passive Optical splitter cable, IB twin port NDR 800Gb/s to 2x400Gb/s, OSFP to 2xOSFP, 3m (Needed to conned to LQ2)

11) Nvidia Twin port transceiver, 800 (2x400) Gbps, OSFP, 2x MPO12 APC, 850NM mmf, UP TO 50M, FINNED

12) Nvidia Passive Fiber cable, MMF, MPO12 APC to MPC12 APC, 30m

F. System Rack Component Specifications B.1 System Racks Configuration Summary

a) Standard 47U tall, 24-inch wide industrial equipment rack, with rails to fit 19-inch wide systems.

b) Entire cluster (rack shared) one or more Gigabit (or faster) Ethernet switches with sufficient ports to support one service network and one management network connection per motherboard and connections to each rack power distribution unit, with one uplink per switch.

c) Two 30A, horizonal 208V power distribution units to connect to power receptacles (L6-30R) above the top of each rack. Power distribution units must have integrated metering with precision to at least 0.1 A, with a visual display and remote current readouts. Note that PDUs with remote individual outlet control are not required.

d) Air flow through the rack shall be front-to-back; no side air input or output is allowed.

B.2 Detailed System Rack Component Specifications

B.2.1 SERVICE AND MANAGEMENT ETHERNET SWITCH(ES)

a) Must have sufficient ports for one service network and one management network connection per motherboard, and one connection to each rack power distribution unit

b) Each switch must have at least one Gigabit or faster Ethernet uplink, twisted pair media.

B.2.2 RACK POWER DISTRIBUTION UNIT

a) Two 30A, Horizonal 208 V power distribution units

b) Must be U.L. Approved

c) The power cord length should be sized to minimize slack. The power receptacles in the computer room will be located approximately 97 inches above the floor, oriented towards the back of the computer racks. Vendors should adjust power cord lengths accordingly, taking into account suitable cable bend radius.

d) Power cables should exit through the top of the rack.

e) Each power cord must end in a connector to plug into the existing 30A, 208V, 2 Pole, 3 Wire, Grounded Receptacle, NEMA #6, L6-30R IG Twist Lock power receptacles.

f) Eaton or APC PDU – Eaton EMAU07-10 example. (16 total)

g) Must have integrated safety circuit breakers to protect against over current.

h) Must have integrated metering with precision to at least 0.1 A, with a visual display and remote current readouts. Note that PDUs with remote individual outlet control are not required.

i) Horizontal power distribution units are required.

B.2.3 SYSTEM RACK, ETHERNET AND POWER CABLING

a) There will be one management Ethernet cable connection from each compute server management Ethernet interface to the management Ethernet network. The compute server management Ethernet interface that connects to the management Ethernet switch must be the one that is connected to the on-board Baseboard Management Controller (BMC).

b) There will be one service Ethernet cable connection from each compute server service Ethernet interface to the service Ethernet network.

c) There will be one power cable connection from each power supply of a compute server to one of the power distribution units. The total electrical load of each system rack should be evenly distributed across the power distribution units; further, if the individual power distribution units have multiple legs, the load on each unit should be evenly distributed across legs.

d) All Ethernet and power cables should be routed along the edge of the system rack using cable management strips.

e) To minimize slack, custom cable lengths for Ethernet and power are preferred.

f) Cables must not block any ventilation holes and must be of minimum length needed to easily connect and disconnect from the compute servers. Cable ties must be cut with cable tie cutter to avoid rough edges.

g) All Ethernet cables inside a system rack must be labeled at each end indicating where the cable came from and where it goes. The label text must be approved by FRA. Any labels that fall off due to poor adhesive quality must be relabeled by the vendor.

h) All cables must be dressed properly so that the weight of the cable does not produce strain on the connectors on either end of the cable.

G. Acceptance Testing

FRA will perform quality testing at several points in the manufacturing, delivery, and installation process:

Within a month of issuance of the PO, vendor shall deliver a single compute server and the proposed rack, cable layout, delivery plan, and integration plan to FRA. FRA will perform a final review of the vendor-proposed rack and cable layout plan, and the vendor-proposed integration and delivery plans, and will request modifications as necessary. Any changes to the BIOS configurations supplied must be noted.

Within 12 weeks of issuance of the PO, vendor will complete the delivery, installation, and integration of all equipment to Fermilab.

Two weeks prior to delivery the vendor shall provide a spreadsheet containing compute server name, all mac addresses, the assigned IPMI address with a SINGLE (same) password for all compute servers, Once the integration of all the systems is complete, FRA will subject all components to a 15-day acceptance test. The entire group of systems cannot have more than 5% downtime for the duration of this acceptance test, where downtime is calculated as the percentage of compute servers that are not available to run application code for any reason, including compute server or network failure. If there is greater than 5% downtime during the 15-day acceptance test, FRA reserves the right to return the entire group of systems. Note: If the systems fail the above acceptance tests, the vendor shall pay for the return packing and shipping of all systems. The vendor is responsible for parts and labor cost of repair during the 15-day burn-in period.

C.1 Requirements

C.1.1 START AND STOP OF DOWNTIME:

During the acceptance phase, each workday at noon (Central Time) FRA will report the status of any failed systems to the vendor’s representatives. The vendor has until noon on the second following business day to supply the necessary parts for repair. The minimum unit of downtime is one day. If any systems go down over the weekend, they will be reported and counted as down on Monday.

C.1.2 LEMON LAW:

A system is considered a lemon during the 15-day acceptance test period if it:

Is down for five working days in a row

OR

Has more than five separate repair incidents.

If a system is designated as a lemon according to this definition, it will be marked as down for the remainder of the testing period.

C.2 Tests

A “Dead on Arrival” test will be performed immediately after the physical installation of the cluster at Fermilab. “Dead on Arrival” is defined as failure to boot an operating system when first powered on after installation. If a compute server is “Dead on Arrival” it will be counted as down until the vendor either replaces it or supplies parts for repair.

After installation FRA will run tests continuously on the systems for the entire 15-day period. These tests will include the mprime stress test as well as memory and disk tests. In addition, during the acceptance tests FRA will power cycle the machines a number of times to determine whether there are any issues. FRA will test remote operation management via IPMI (system reset, power control, sensor readout, serial over LAN). During application tests, FRA personnel will measure power consumption and system temperatures. Each compute server under full load must not crash, power-down, or reset. Each compute server under full load must not exceed vendor recommendations for system or processor temperature.

H. Warranty and Delivery Requirements D.1 Warranty Specifications

The following is a summary of the warranty requirements of FRA that will be required for all purchases. An explicit Statement of Warranty must be included as part of the response to the bid, accepting all these conditions.

D.1.1 LENGTH AND COVERAGE OF WARRANTY:

a) A five-year warranty for all compute server parts and materials shall be required, including labor for onsite or third-party service.

b) The warranty period will not begin until the conclusion of the 15-day acceptance test.

c) Vendors must provide the on-site labor or contract with a 3rd party provider. Use of 3rd party service providers must be pre-approved by FRA.

d) Vendors using a 3rd party service provider must furnish evidence of a contractual agreement with the service provider before the delivery of the systems to Fermilab.

e) The FRA Service Desk will place service calls. Our expert FRA technical staff will perform a preliminary diagnosis when possible to help provide the Service Desk staff with specific problem identification. The Service Desk staff will relay this information to the vendor when they place the call. Please note, however, the Service Desk does not have access to the equipment nor the technical expertise for further interactive diagnosis of hardware when placing the call. The vendor is ultimately responsible for resolution of any problem.

f) The vendor is our single point of contact for problems found in any component of the delivered system and accepts complete responsibility for its resolution.

g) Vendor is fully responsible for providing any necessary travel, tools, shipping costs, or firmware upgrades necessary to maintain the high performance of the systems over the full warranty period. Defective systems must be brought back into service within five business days of the report of the problem.

h) Batch Maintenance Option – The vendor may supply at their option fully functional identical spare equipment at time of delivery. Specifically, the vendor must supply a number of servers equal to 2% of the full cluster count, rounded up to the nearest integer. These spares on hand (SOH) should be delivered with the cluster but will be stored in FRA-provided storage cabinets. This option allows the vendor to delay, at their discretion, repairs of failed machines and components until (SOH +1) machines need repair. It is assumed that the vendor will then repair all machines, including the spares, during their repair visit or will ship replacement servers and/or components, with the failed servers and/or components returned to the vendor by FRA.

i) FRA can provide cabinet space for vendor or on-site service provider use. Such cabinets will be provided with FRA supplied locks to which a FRA building manager will retain a master key, and for which the combination can be selected by the user. These cabinets will be located in secure data center areas. The maximum cabinet space allowed will be two cabinets, each of approximate size 4 foot by 2 foot by 7-foot height.

D.1.2 RESPONSE TIME:

a) Four-hour response during business hours is required. This response shall be an acknowledgement of the reported problem and the plan for remediation of the problem.

b) Two-hour callback during normal business hours is required if human contact is not possible on the initial call.

D.1.3 SERVICE LEVEL AGREEMENT:

a) FRA reserves the right to require replacement of the entire server if it has experienced five hardware service calls within a six-month time period.

b) If the vendor is unable to deliver service that satisfies these requirements, FRA reserves the right to procure such maintenance service and parts from another source. The vendor shall reimburse FRA for the cost of such services and parts. The vendor shall also reimburse FRA for any parts used at the current manufacturer list price less any discounts available to FRA. The vendor grants FRA the right at FRA’s discretion to deduct such expenses from pending invoices or use it as a credit or to request payment to FRA.

D.1.4 COVERAGE:

The Principal Period of Maintenance is defined as Monday through Friday from 8AM to 5PM US Central time. Flexibility of schedules can be made with prior coordination and approval.

D.1.5 COMPONENT WARRANTIES:

All compute server components must come with at least five years warranty. The statement of warranty must explicitly state whether the warranty on the CPU, motherboard, disk drives is from the manufacturer or from the vendor. If any components have less than five years manufacturer’s warranty, the vendor is responsible for repair or replacement after the expiration of the manufacturer’s warranty.

D.1.6 DELIVERY:

Vendor or the sub-contractor is responsible for the physical installation and integration of the compute servers, system racks, and network racks at the Fermilab computer facility. If the vendor choses to use the FRA-provided racks and/or power distribution units, at the time of delivery FRA will ensure that the racks are empty and ready for installation and integration of servers and network equipment. Vendor or the sub-contractor is required to deliver and unload all required materials at the designated Fermilab Data Center. Vendor or the sub-contractor is required to unpack all equipment in a designated work area, and then install them in the specified computer room. Vendor or the sub-contractor will be responsible for removing all unused components and packing materials offsite. All delivery and installation logistics and schedule will be coordinated through FRA.

D.1.7 SAFETY:

Any vendor or the sub-contractor service personnel that will perform work on-site must attend safety training required by FRA Employee Safety and Health policies. Prior to the delivery and installation, the vendor must prepare a job hazard analysis (JHA) and obtain approval from FRA before performing any work. During the delivery and installation, appropriate personal protective equipment (PPE) must be employed.

The primary service personnel who will provide warranty service must obtain an FRA Contractor ID, which requires completing the safety training required by FRA Employee Safety and Health policies. Depending on the work activities being performed, personnel providing warranty service must employ appropriate PPE prior to starting any work activities.

D.1.8 CONTRACTOR IDENTIFICATION:

All Vendor or sub-contractor Customer Service Engineers that will perform work on-site and who have been issued a permanent FRA ID are expected to keep their ID’s current. Another employee may not use the ID. If an ID is allowed to expire, entry into certain areas that have card readers will be prohibited. Contact a FRA Service Coordinator to arrange to have an ID renewed.

D.1.9 FRA POLICY ON COMPUTING:

Vendor or the sub-contractor staff members that will perform work on-site will be requested to read and comply with the ‘FRA Policy on Computing’ available at http://security.fnal.gov/policies/cpolicy.html

D.1.10 CONTACTS:

Vendor must provide technical contacts including name, address, telephone number, and email address for warranty service on the attached configuration sheet.

Appendix A: Computer Room

The computing system will be installed at Fermi National Accelerator Laboratory, Grid Computing Center in room CRA adjecent to LQ2 installation, Batavia, IL.

GCC Computer Rooms have a standard raised floor, providing a 3-foot tall plenum for forced cold-air cooling. Overhead electrical conduits provide two L6-30R (30A, 208V, 2 pole, 3 wire, grounding receptacles) per rack position.

FNAL IC TECHNICAL SPECIFICATION