N0017423R0028.A1-Modification.pdf

AMENDMENT OF SOLICITATION/MODIFICATION OF CONTRACT

1. CONTRACT ID CODE PAGE OF PAGES

1 26

2. AMENDMENT/MODIFICATION NO.

3. EFFECTIVE DATE

10-May-2023

4. REQUISITION/PURCHASE REQ. NO. 5. PROJECT NO.(If applicable)

6. ISSUED BY CODE N00174 7. ADMINISTERED BY (If other than item 6) CODE

NSWC IHD C02

4081 N. JACKSON RD

BLDG 841

INDIAN HEAD MD 20640-5116

See Item 6

8. NAME AND ADDRESS OF CONTRACTOR (No., Street, County, State and Zip Code) X

9A. AMENDMENT OF SOLICITATION NO.

N0017423R0028

X 9B. DATED (SEE ITEM 11)

25-Apr-2023

10A. MOD. OF CONTRACT/ORDER NO.

10B. DATED (SEE ITEM 13)

CODE FACILITY CODE

11. THIS ITEM ONLY APPLIES TO AMENDMENTS OF SOLICITATIONS

X The above numbered solicitation is amended as set forth in Item 14. The hour and date specified for receipt of Offer X is extended, is not extended.

Offer must acknowledge receipt of this amendment prior to the hour and date specified in the solicitation or as amended by one ofthe following methods:

(a) By completing Items 8 and 15, and returning 1 copies of the amendment; (b) By acknowledging receipt of this amendment on each copy of the offer submitted;

or (c) By separate letter or telegram which includes a reference to the solicitation and amendment numbers. FAILURE OF YOUR ACKNOWLEDGMENT TO BE

RECEIVED AT THE PLACE DESIGNATED FOR THE RECEIPT OF OFFERS PRIOR TO THE HOUR AND DATE SPECIFIED MAY RESULT IN

REJECTION OF YOUR OFFER. If by virtue of this amendment you desire to change an offer already submitted, such change may be made by telegramor letter, provided each telegramor letter makes reference to the solicitation and this amendment, and is received prior to the opening hour and date specified.

12. ACCOUNTING AND APPROPRIATION DATA (If required)

13. THIS ITEM APPLIES ONLY TO MODIFICATIONS OF CONTRACT S/ORDERS.

IT MODIFIES THE CONTRACT/ORDER NO. AS DESCRIBED IN ITEM 14.

A. THIS CHANGE ORDER IS ISSUED PURSUANT TO: (Specify authority) THE CHANGES SET FORTH IN ITEM 14 ARE MADE IN THE

CONTRACT ORDER NO. IN ITEM 10A.

B. THE ABOVE NUMBERED CONTRACT/ORDER IS MODIFIED TO REFLECT THE ADMINISTRATIVE CHANGES (such as changes in paying office, appropriation date, etc.) SET FORTH IN ITEM 14, PURSUANT TO THE AUTHORITY OF FAR 43.103(B).

C. THIS SUPPLEMENTAL AGREEMENT IS ENTERED INTO PURSUANT TO AUTHORITY OF:

D. OTHER (Specify type of modification and authority)

E. IMPORTANT: Contractor is not, is required to sign this document and return copies to the issuing office.

14. DESCRIPTION OF AMENDMENT/MODIFICATION (Organized by UCF section headings, including solicitation/contract subject matter where feasible.)

This amendment is to address Requests for Information, as w ell as to provide updates to the line items and Statement of Work. An extension of 6 days w ill be added for solicitation offer due dates. The due date has been changed from 12 May 2023 to 18 May 2023.

Except as provided herein, all terms and conditions of the document referenced in Item 9A or 10A, as heretofore changed, remains unchanged and in full force and effect.

15A. NAME AND TITLE OF SIGNER (Type or print) 16A. NAME AND TITLE OF CONTRACTING OFFICER (Type or print)

TEL: EMAIL:

15B. CONTRACTOR/OFFEROR

(Signature of person authorized to sign)

15C. DATE SIGNED 16B. UNITED STATES OF AMERICA

BY

(Signature of Contracting Officer)

16C. DATE SIGNED

10-May-2023

EXCEPTION TO SF 30

APPROVED BY OIRM 11-84

30-105-04 STANDARD FORM 30 (Rev. 10-83)

Prescribed by GSA

FAR (48 CFR) 53.243

N0017423R0028

SECTION SF 30 BLOCK 14 CONTINUATION PAGE

SUMMARY OF CHANGES

SECTION SF 30 - BLOCK 14 CONTINUATION PAGE

The following have been modified:

STATEMENT OF WORK (SOW)

STATEMENT OF WORK

NSWC INDIAN HEAD DIVISION SUPERCOMPUTING FACILITY

SUMMARY: Described in this SOW are the minimum requirements for the hardware and software, shipping, installation and software configurations, training, and five-year warranty and support for the cluster high performance supercomputer (HPC) to be procured to establish the

Naval Surface Warfare Center, Indian Head Division (NSWC IHD), MD supercomputing facility.

HPC at the end of installation, software configuration, and training will be on the NSWC IHD network for end users to remotely connect and use for their day-to-day modeling and simulations

(M&S) needs. Vendor may be versed with the hardware configuration listed herein. However, there are differences in how the HPC needs to be configured and made operational. Throughout this SOW, there are options for the selected vendor to suggest modifications in consultations with the NSWC IHD Technical POC and IT department in that order. Due to the unique requirements for hardware and software configurations, compatibility with the cyber security and STIGS requirements, and five years warranty and maintenance with next business day service at a secured facility, NSWC IHD is compelled to receive proposal / bid from vendors who satisfy the following requirements:

Vendor implies the original manufacturer or a manufacturer’s designated reseller of cluster

HPC. In the latter case, the reseller r and manufacturer agree that the requirements in totality of this SOW applies equally to the reseller and original manufacturer.

Vendor must have a proven track record of manufacturing cluster HPC, optimizing its performance for scientific and engineering computing in a multi-user environment, and selecting components that enhances the performance. The quote must include brochure / documentation and years of experience of supplying HPC as the supplemental documents with the quote.

Vendor must have a proven track record of having supplied and maintained / maintaining similar multi-node HPC to Department of Defense (DoD) laboratory/ies. This is to ensure that vendor has experience and is versed with the cyber security and STIGS to be applied by the

NSWC IHD IT team after installation. This requirement will mitigate any incompatibility of the hardware and software with cyber security and STIGS requirements before being delivered, which will otherwise render the HPC unusable on the NSWC IHD network. Please note that the cyber security and STIGS requirement being practiced by Department of Energy

(DOE) laboratories may be different from the DoD laboratories. Hence, the focus is on the

DoD laboratories.

Vendor must be familiar with and practicing TAA and GSA compliance of the hardware and software. The compliance applies to individual components as well as to the assembled racks.

NSWC IHD, at its own discretion, reserves the right to waive this requirement for a small percentage of components. For this waiver, vendor must summarize the list of non-compliant components at the end of their quote and agree to replace such components free-of-cost in the case of failure during the maintenance period of 5 years

The five years warranty and maintenance by vendor of all the components including UPS batteries and the assembled HPC includes 24x7 telephone support, dedicated port-to-port secured remote connection to the HPC for trouble shooting, and next business day replacement of the failed/failing components. It is inevitable that the HPC will contain components manufactured by a third party. NSWC IHD will not be responsible to contact any third party for this warranty and maintenance support needs. The vendor will be solely responsible to fulfill the warranty and maintenance support of all components manufactured by the vendor

(manufacturer or reseller as the case may be) or any third party. The vendor will setup a telephone number to directly reach the support instead of being routed through number of options.

The next business day support service will include overnight shipping of failed/failing components as indicated by the cluster management software and/or technical support and visit by vendor’s technician to NSWC IHD for the replacement.

Vendor must have security cleared U.S. National technicians for the secured port-to-port remote connections and the next business day part replacement services. HPC, after installation and commissioning, will not be accessible by a non-cleared technician even if he/she is a U.S. National.

1. Quote Structure: HPC will consist of several number of nodes and other components that make the HPC as given in Table 1 and described further. NSWC IHD must tag these as assets and enter them into the inventory. However, it will not be possible to individually tag and inventory individual components of the HPC. For this reason, vendors must provide the quote for individual racks in the format given below so that only racks are tagged as individual assets containing the components listed in the quote. The automated quote system of the vendor may not produce a rack wise quote format. In such case, the vendor will attach a supplemental quote rack wise referring to components in each racks to the line number or group of line numbers in the automated system quote. Additionally, vendors will attach the elevation of racks, electrical, and cooling requirements. The format for the main automated quote or the rack wise supplemental quote will be:

Hardware

Item No 1: Rack 1 containing which nodes and/or how many UPS with Price

…. And so on.

Item No L: Rack L containing …….

Software

Item No L+1: Description and price

… And so on.

Item No. M: ………

Uninterrupted Power Supply (UPS), if UPS is to be provided in a separate rack

Item No M+1: Description and price

…And so on

Item No N:……

Shipping

Item No N+1 – Shipping (FOB) with price. All shipments (if more than one) should be combined as one item for the purpose of the quote.

Installation and Software Configuration

Item No N+2: Installation and Final Configuration of Software

Training

Item No N+3: Training of NSWC IHD IT Staffs

Warranty and Support

Item No N+4: 5 Years warranty/Support with 24x7 telephone support and next business day

(NBD) replacement.

Vendor may absorb cost of some of the items above in the hardware cost, for example, shipping, installation and final configuration of software, etc. In that case, the cost of such item must be listed as $0.00 in the summary line items. At the least, the cost for warranty and 5 years support must be absorbed in the hardware cost to enable simplified procurement as envisaged in this announcement.

2. HARDWARE: HPC hardware and software configuration and capacity (C&C) are given in

Table 1 specifying the type and number of nodes, processors’ base clock frequency, RAM memory per core, total RAM memory, type, and capacity of hard drive(s) on the nodes, and NVIDIA GPU if applicable. Also given are some of the key components required to make the cluster an HPC.

Vendor will provide other needed components, but not given in Table 1, of the specifications to maintain or exceed the overall performance and maintainability of the HPC. Hence is the emphasis on the experience of the vendor (reseller / manufacturer) described above. The salient features are explained below to highlight differences mentioned above and remove ambiguity to help vendors to account for all the listed hardware and services.

2.1. Processor Cores: All processors are AMD EPYC series or better. All nodes are dual sockets

(processors) nodes. The numbers of cores in Table 1 are the base number of cores (sum of cores for two processors) before hyperthreading. Many engineering software in use at NSWC IHD either do not utilize or do not perform optimally after enabling hyperthreading. Hence, the increase in number of threads (generally twice) enabled by hyperthreading will not be considered.

2.2. Definition of Minimum Requirements: Table 1 gives the minimum requirements of critical components. Vendor may provide the minimum or better than minimum configuration at its own discretion which may translate to the total cost.

AMD processors or any other processor with better base clock frequency and/or larger number of cores per processor are allowed with restrictions. The better clock frequency is permitted if the required clock frequency of Table 1 has met the end of line (EOL). A larger number of cores per processor will not educe the number of nodes in individual category. For that matter, the minimum number of master, login, disc, and compute nodes must be maintained independent of the higher clock frequency of the processors. The vendor may choose to provide single socket (processor) nodes. The single socket nodes must be uniform throughout the system. In that case, the number of node in each category must double.

On the contrary, the number of nodes has to increase uniformly in all categories if the processor meets the base clock frequency requirement but has a smaller number of cores. This will increase the total hard drive and number of GPUs if applicable and may become cost prohibitive.

RAM memory per core will remain as given in Table 1 or larger. If the total number of cores on each node is more than the specified value, then the total RAM memory on the node will also increase to meet the requirement.

All processor to memory channels must be populated with RAM DIMMs on all nodes for optimal performance. Total RAM memory on nodes is allowed to exceed the minimum requirement if the DIMM capacity being offered is such that populating all channels increases the total RAM memory.

The number, type, and capacity of hard drive(s) per node will be as specified. The total hard drive on one or more categories of nodes will increase if the number of nodes increases.

2.3. Master Nodes: The required configuration has two master nodes for traditional functions of administering, managing, controlling nodes, and providing the system and user applications software. All software will be installed on the master nodes and made available to users on the login nodes. Similarly, all the audit files will reside on master nodes. For this reason, master nodes have larger disk space that cannot be reduced. The two master nodes can be configured in three different ways as given later or in a better, alternate way in consultation with the NSWC IHD

Technical POC and IT department in that order.

Vendor may prefer to use software installed on some other class of nodes (login or disk) which will provide all the functionalities of master nodes and decide to remove the master nodes from the configuration. Removal of master nodes from the configuration is not permitted to maintain uniformity across all vendors bidding for the contract. Such removal will be detrimental to the

HPC performance by not having other functions as envisaged for the HPC.

2.4. Login Nodes: Login nodes are for users’ interactive work that will include editing and developing source code, compilation, single / multiple cores execution for not more than 15 minutes to test the software, and submitting finals runs on compute nodes. Vendor in consultation with NSWC IHD IT team will decide the final configuration as given later. In addition to their work and data on login nodes, users may also need additional space to install software in their home folder. For this reason, login nodes are configured with large disk space which can be increased, but not reduced.

2.5. Compute Nodes: These are traditional compute nodes for parallel computing. In contrast to diskless compute nodes, the disk on compute nodes serves two purposes by having two partitions

– (a) the operating system (OS) partition with enough space for future upgrades will contain SWAP space equal in size to the RAM memory on the node to act as the virtual memory, and (b) the second partition will be the “local scratch”. If software running on a node needs to write scratch data that does not need to be preserved, then such data will be written locally on the local scratch for optimal performance. Vendor has the option to eliminate two partitions and provide space for both function in one sole partition.

2.6. Compute Nodes with GPU: These are same as compute nodes described above with the exception that they have one NVIDIA A100 GPU per node. The number of GPUs cannot be reduced below 8 in total. increasing the number of these nodes due to the lower number of cores per processor will increase the number of GPUs which will be cost prohibitive and not advised.

2.7. Large Memory Compute & Visualization Nodes with GPU: Their function and configuration will be same as the compute nodes with GPU. Their RAM size per core is twice

(8GB) compared to compute nodes (4GB). Accordingly, the SWAP space size on their disk will also be twice the size of those on the compute nodes.

2.8. Disk Node: With users’ home folders mounted on login nodes, they will have folders for the long-term data storage on the disk node. The output files during software execution needing to be stored will be written on the disk node. For this reason, the disk space on the disk node is the largest and can be increased, but not reduced.

2.9. Infiniband and Switches: These are the backbone of the cluster HPC. Vendor must provide the minimum required given in Table 1 or better. Vendor may choose a better option to boost the inter-node data transfer rate and data transfer bandwidth. Vendor may also choose a better option if the specified capacity has reached end of line (EOL). Vendor will connect all the racks with and configure the Infiniband for the entire cluster to work as one HPC. Furthermore, they need to be configured for easy additions of future rack(s). The capacity and numbers of switches for users’ connectivity must allow, at a minimum, 50 concurrent users without network latency.

2.10. TPM 2.0 or Higher Enabled: As per DON’s critical requirements, all nodes must be TPM

2.0 or higher enabled for encryption. This must be provided and stated in the quote for each node, each class of node, or as a note at the end stating: “All nodes have TPM 2.0 or Higher Enabled”.

Vendor will provide necessary hardware on the motherboard of every node to enable TPM 2.0 or higher.

2.11. No Bluetooth / No Wireless: As the second critical requirement, there must not be any

Bluetooth or Wireless hardware on any node. It must be mentioned on each node or as a note at the end stating: “No Bluetooth / No Wireless”.

2.12. TAA and GSA Compliance: As the third critical requirement, all hardware must be TAA and GSA compliant. NSWC IHD may allow a very small percentage of non-TAA and GSA compliant hardware at its discretion. Vendors must list the non-compliant components at the end of the quote. Vendors must minimize the non-compliant components. The non-compliant components provided in the list may disqualify the vendor without further discussion.

2.13 No Hardware or Software RAID: As the fourth critical requirement, there must not be hardware enabled or software enabled RAID on any node and HPC as a whole.

2.14. Humidity and Temperature Sensors: Vendor must provide enough humidity and temperature sensors in every rack. These sensors must be setup to trigger the HPC shutdown if the humidity and/or temperature reaches the upper safe limit.

2.15. Built-in console: Vendor must include one, preferably two, built-in console(s) for system administration, exploration, and maintenance of the entire HPC as a whole.

3. SOFTWARE

The needed software is to be provided in part by the vendor and rest by NSWC IHD as follows:

Vendor will quote and provide the cluster management software with 5 years support. All the provision for the support listed above apply equally to the software.

NSWC IHD will provide the Red Hat Enterprise Linux (RHEL) Operating System (OS), Intel oneAPI developer suites even for the AMD processors based system, AMD Zen Software

Studio developer suites for the AMD processor based system, and SLURM resource management software. Vendor will provide the list in the quote for number of RHEL OS licenses needed for the master nodes, login nodes, compute nodes, compute nodes with GPU, large memory compute and visualization nodes with GPU, and disc node. Vendors will confirm the number of licenses required with the Department of Navy (DON) RHEL POC before listing them in the quote.

Vendor will install and configure all software (provided by the vendor or NSWC IHD) as given in the installation section.

4. UNINTERRUPTED POWER SUPPLY (UPS)

In the event of a power outage, the purpose of the UPS is to delete all waiting jobs in the SLURM queues, terminate all the users’ jobs running at the time, and shutdown the entire HPC. Vendor will determine the time required “H Minutes”. The UPS number and total capacity needs to enable safe completion of termination and shutdown sequences. HPC will be started manually after the power is restored. Vendor has the choice of (a) the preferred method of installing UPS in the lower half of racks while the upper racks have nodes, or (b) the entire UPS installed in a separate rack and connected with the HPC.

5. SHIPPING

Vendor has the following options for the shipping.

Vendor can ship the individual components of HPC and assemble them in the rack adhering to the rack configuration as was provided in the quote or as supplemental quote.

Vendor can ship the fully assembled racks per the quote.

Vendor can also choose to mix the above two modes for shipping.

Vendors must provide all shipping tracking numbers to NSWC IHD Technical POC for tracking and receiving.

6. INSTALLATION AND SOFTWARE CONFIGURATION

The chosen vendor will install the hardware, install and configure software, and make the HPC ready for use at the end as follows:

6.1. The vendor will assemble all the components in the racks, so that each rack has the components as provided in the quote / supplemental quote. The racks will be installed in the server room and connected to the wall electrical outlet as pre-determined by the vendor during the visit mentioned later.

6.2. Vendor will install RHEL on all nodes in the presence of NSWC IHD IT staffs. The purpose of IT staffs’ during this installation is not to help or suggest installation. Vendor can avail the technical support from RHEL as required on the licenses provided by NSWC IHD. Vendor will install:

Full version of RHEL, not the minimal or customized version, “devel” and “group packages”, which are not installed by default.

NSWC IHD will provide the list of the required “devel” and “group packages”. The list will also contain a few select configurations independent from those of the vendor. The vendor will configure the OS for the best performance of the HPC.

6.3. In consultation with the NSWC IHD Technical POC / IT team, the vendor will choose one of the three options given below or an alternate better option to configure the two master nodes:

(a) One master node performing all the functions and the other acting as standby and the two switching their function at regular intervals, or

(b) Both master nodes operating together and splitting the workload nearly equally.

6.4. In consultation with the NSWC IHD IT team, vendor will choose one of the following options to configure the login nodes. Both options require the users’ home folders to be mounted on the login nodes.

(a) Users’ login to the master node and work on the login nodes with their home folder mounted on the login nodes, or

(b) Users directly login to login nodes and work.

6.5 Vendor, in consultation with NSWC IHD IT staffs, will install InfiniBand and setup all network switches including those to be used by users for the remote connection. As mentioned earlier, the number of switches and their setup for users must be to provide concurrent remote connections to

50 users without affecting network bandwidth available on the HPC.

6.6 Vendor will install cluster management software, interface it with the built-in console for the display of the system and system status, and to query, administer, explore, and manage individual nodes, as well as HPC as a whole.

6.7 Vendor will help the NSWC IHD IT team while the latter implements STIGS security measures.

6.8 Vendor will install NSWC IHD provided Intel oneAPI development software. Vendor can avail the technical support from Intel on the license provided by NSWC IHD. The software will have default compilation flags and documentation for setup of the environmental variables. These must be provided to users through CRON at the time of their account creation.

6.9 Vendor will install NSWC IHD provided AMD Zen Software Studio development package for the AMD processor based system only. This is a free software available from AMD. However, NSWC IHD will facilitate and vendor can avail the technical support from AMD, if required. Like oneAPI, the default compilation flag and environment variable setup must be provided to individual users by default.

6.10 Vendor will also install NSWC IHD provided resource management software SLURM by

SchedMD. SLURM is a free software, but NSWC IHD will procure one-year technical support from SchedMD which the vendor can choose to utilize during the installation.

6.11 It is often found that the default path after OS installation does not include the paths for key packages and utilities or the default path is altered during STIGS. Vendor will make all the needed efforts in consultation with the NSWC IHD IT team to provide them in the default path. It is preferred that /etc/bashrc is copied to users’ /.bashrc file for further customization by the users.

6.12 Vendor must provide support to NSWC IHD IT staffs during STIGS that will occur before training after all the software has been installed and stress test has been successfully completed.

7. TRAINING

Vendor will include in the quote and provide training to NSWC IHD IT staffs. The training will start after the STIGS customization has been completed by the IT staffs in consultation with the vendor technical staffs. Vendor will determine the number of days needed for this first training and quote it as a separate line item. Vendor will also provide 2 days follow-up refresher training after one year and account for this refresher training as the training cost in the quote. NSWC IHD

IT may require yearly training in subsequent years for which vendor will submit a new quote when the need arises. The initial and refresher trainings will include, but not limited to:

7.1. Shutting down and booting the entire HPC.

7.2. Shutting down and isolating individual node(s), if required.

7.3. Use of the cluster management software through the built-in console.

7.4. Checking the general health of the HPC on the console.

7.5. Troubleshooting any error(s) or message(s) displayed by the cluster management software.

7.6. Setting and changing the soft and hard limit of disk quota for users.

7.7. Upgrading OS and the sequence, if any.

7.8. Upgrading other software as required. Etc.

8. WARRANTY AND SUPPORT

Vendor will provide 5 years warranty and support for all the hardware including UPS and software.

The software in this section refers to the cluster management software provided by the vendor, as well as RHEL OS provided by NSWC IHD. 5 years warranty and support will include the following:

8.1. It may be the case that the vendor procures hardware and/or software from a third-party and provides the HPC to NSWC IHD as a package. The 5 years warranty and support apply solely to the vendor and not to such third-party vendors. It implies that NSWC IHD will contact the vendor for any warranty and/or support issues and it will be the sole responsibility of the vendor to resolve the issue on its own or in consultation with the third-party vendor. NSWC IHD will not be responsible to directly contact the third-party vendor.

8.2. 24x7 telephone support for any hardware / software issues. Vendor will assign a U.S. Citizen

(preferably with a security clearance) support personnel at their end for this telephone support.

Vendor will also provide the single dial phone number to reach the support instead of being routed through several options after dialing.

8.3. Next business day (NBD) replacement of any hardware that fails during the 5 years. A U.S.

Citizen technician with a security clearance is required for visiting NSWC IHD for the NBD service at a short notice.

8.4. In select circumstances, NSWC IHD may provide a dedicated port-to-port connection to the vendor’s support personnel to remotely access the HPC for troubleshooting or help. Vendor must designate a U.S. Citizen personnel with a security clearance at their end for this purpose.

8.5. HPC performance depends not only on the state-of-the-art hardware and software, but also on how well the OS is installed, configured, upgraded, and maintained on the nodes. For this reason, even though NSWC IHD will procure and provide RHEL OS licenses, the maintenance of RHEL

OS will rest on the vendor in association with the NSWC IHD IT staffs and RHEL team for troubleshooting, any dependencies during upgrades, etc. The vendor will have free access to RHEL via the licenses procured by NSWC IHD.

8.6. NSWC IHD does not expect any support needed for Intel oneAPI and AMD Zen Software

Studio. However, a few rare occasions may be foreseen when the vendor’s support may be required, for example, conflict of the future upgrades of these two software programs with the OS or other software, future upgrades being installed in a new folder that changes the environmental variables, etc. NSWC IHD will make earnest attempt to solve such issues by contacting Intel and

AMD respectively. But the vendor agrees to troubleshoot or provide support for any conflict with the RHEL OS.

8.7. Similar to above, the vendor’s support may be needed, and vendor will provide the support for any future issues with SLURM with or without the help of SchedMD. The issue foreseen, but not limited to, is if future RHEL OS upgrades conflict with the installed SLURM or vice versa that disables the queues.

8.8. UPS battery life is in general three years. The 5 years support applies to all the UPS and UPS batteries for uninterrupted operation of the HPC. This implies that the vendor will replace the batteries free of cost at the end of their life as a part of the maintenance. Vendor should include the batteries replacement cost in the quote if necessary for one time replacement in 5 years.

However, batteries are susceptible to fail before the end of their life. Hence, the vendor will provide the best quality batteries and replace them free-of-cost as a part of the 5 years warranty/ support service.

9. OTHERS

9.1. Vendor’s Visit to NSWC IHD: The technical personnel of the selected vendor will visit

NSWC IHD at the earliest after securing the contract to:

Determine the optimal position of the HPC racks in the server room.

Advise on the dedicated electrical connection to be drawn to the server room and type / number of wall / under floor outlets for the HPC.

Advise on the present capacity of the air-conditioning and on the standby unit to be installed by NSWC IHD. The room has under floor air cooling and the present capacity is considered sufficient for the HPC.

NSWC IHD IT team is designing a secure procedure to connect the HPC to the internal network for end users to connect to the HPC machine from their desk without network interruptions and loss of bandwidth while meeting the security and STIGS measures. The vendor’s visiting team will consult and advise the NSWC IHD IT team on this subject matter.

9.2. CRON Development-1: It is being increasingly found that the scratch files written on local scratch during program executions are not deleted automatically after execution completes. They need to be manually deleted more frequently than desired. Vendor in collaboration with NSWC

IHD IT team will develop and install CRON script which will automatically detect the program termination and delete all files created temporarily by the program on local scratch.

9.3. CRON DEVELOPMENT-2: HPC has NVIDIA GPUs which will require NVIDIA drivers.

It is the case that RHEL OS upgrades do not upgrade the NVIDIA drivers and these drivers need to be re-built after the RHEL kernel upgrades. This may not be the case for the multi-node HPC or for the RHEL version being installed. Vendor will confirm this with RHEL during OS installation. If that is the case, vendor in collaboration with NSWC IHD IT team will develop and install a CRON script which will detect kernel upgrade(s) and automatically build NVIDIA drivers

(if required) on individual nodes.

9.4. Stress Test: After completion of the software installation and configuration, vendor will carry out stress testing by running the HPC at 100% load for a specified time. Any defective component or the components that fails during stress testing will be replaced and stress testing will be repeated till the test shows all components working without failure.

9.5. File System: Vendor will determine the optimal file system for HPC in consultation with the

NSWC IHD Technical POC and IT team in that order. The file system will play a major role in the optimal performance of the HPC. There may be a few file systems that are considered superior, but they may not be allowed due to the security and STIGS requirements. Hence, determining the optimal file system within the security measures is critical. However, the selected file system must be capable to scale up in the future for accommodating additional racks.

Table 1: Final Configuration and Capacity (C&C) of HPC and Other

Requirements for NSWC IHD Supercomputing Facility (refer to section 2 of

SOW). All Processors are AMD

EPYC Series or Better.

ITEM

NUMBE DESCRIPTION

R

HARDWARE

NODES COMPONENTS NUMBERS

Login

Number of Nodes 4

Number of Processor per Node 2

Base Clock Frequency of Processor 2.2 GHz

Total Number of Cores per Node 64

Total Number of Cores 256

RAM Memory Per Core 8 GB DDR4 3200 ECC

Total RAM Memory per Node

512 GB DDR4 3200

ECC

Total RAM Memory

2.05 TB DDR4 3200

ECC

Hard Drive Per Node 6.4 TB NVMe

Total Hard Drives 25.6 TB NVMe

Master

Number of Nodes 2

Number of Processors per Node 2

Base Clock Frequency of Processor 3.2 GHz

Total Number of Cores per Node 32

Total Number of Cores 64

RAM Memory Per Core 8 GB DDR4 3200 ECC

Total RAM Memory per Node

256 GB DDR4 3200

ECC

Total RAM Memory

512 GB DDR4 3200

ECC

Hard Drive Per Node 6.4 TB NVMe

Total Hard Drives 12.8 TB NVMe

Compute without

GPU

Number of Nodes 24

Number of Processors per Node 2

Base Clock Frequency of Processor 2.2 GHz

Total Number of Cores per Node 64

Total Number of Cores 1536

RAM Memory Per Core 4 GB DDR4 3200 ECC

Total RAM Memory per Node 256 GB DDR4 3200

ECC

Total RAM Memory 6.14 TB DDR4 3200

ECC

Hard Drive Per Node 1.6 TB NVMe

Total Hard Drives 38.4 TB NVMe

Compute with GPU

Number of Nodes 8

Number of Processors per Node 2

Base Clock Frequency of Processor 2.2 GHz

Total Number of Core per Node 64

Total Number of Cores 512

RAM Memory Per Core 4 GB DDR4 3200 ECC

Total RAM Memory per Node 256 GB DDR4 3200

ECC

Total RAM Memory 2.05 TB DDR4 3200

ECC

Hard Drive Per Node 1.6 TB NVMe

Total Hard Drives 12.8 TB NVMe

NVIDIA A100 40 GB PCIe GPU per Node 1

Total NVIDIA A100 40 GB PCIe GPUs 8

Large

Memory

Compute and

Visualization with GPU

Number of Nodes 8

Number of Processors per Node 2

Base Clock Frequency of Processor 2.2 GHz

Total Number of Cores per Node 64

Total Number of Cores 512

RAM Memory Per Core 8 GB DDR4 3200 ECC

Total RAM Memory per Node 512 GB DDR4 3200

ECC

Total RAM Memory 4.1 TB DDR4 3200

ECC

Hard Drive Per Node 1.6 TB NVMe

Total Hard Drives 12.8 TB NVMe

NVIDIA A100 40 GB PCIe GPU per Node 1

Total NVIDIA A100 40 GB PCIe GPUs 8

Disk

Number of Nodes 1

Number of Processors per Node 2

Base Clock Frequency of Processor 3.0 GHz

Total Number of Cores per Node 32

Total Number of Cores 32

RAM Memory Per Core 8 GB DDR4 3200 ECC

Total RAM Memory per Node 256 GB DDR4 3200

ECC

Total RAM Memory 256 GB DDR4 3200

ECC

480 GB SATA SSD Hard Drive Per Node 2

18TB SAS 12G HDD Hard Drives 12

OTHER HARDWARE

Intra/Inter

Connectivity

InfiniBand HDR 100 with 100/200 GB Switches

As needed

10 GB Switches for Management/Remote Access

As needed for 50 Concurrent Users

Built-in Console

Management Console with KVM Switches

9 UPS To Support Entire System (see notes) As Needed

10 Racks Enough racks to house the above hardware and UPS and scalable for future additions

As Needed

Temperature and Humidity sensors that are interfaced with the cluster management software to notify their values reaching upper safe limits

SOFTWARE

To be provided, installed / configured by the vendor

Cluster Management Software

5 Years Support

To be provided by

NSWC IHD

and installed/con figured by the vendor

Red Hat Enterprise Linux (RHEL) operating system (OS) for various nodes depending on their functions to be procured on DON ELA contract with DLT

Intel oneAPI for Cluster with MKL &

VTUNE

5 years Premium support recurring cost

5 Years Support, 20

AMD Zen Software Studio containing (a)

AOCC 3.2, (b) AOCL LibM 3.9, © uProf

3.6, (d) ZenDNN 3.3, and (e) Spack

5 years Support, 20

Resource Management Software SLURM

1 Year Support with

Option to Renew the

Support, 50 Concurrent

Users

MobaXterm software for users to connect to the HPC using Windows machine with

Linux environment ported to Windows

1 Year License recurring cost

OTHERS

NSWC IHD

Visit

Vendor will visit NSWC IHD at the earliest possible after the contract/PO award and before shipping to survey the installation site and suggest any improvements needed for the racks’ placement, electrical and cooling.

Shipping

Shipping to Site (FOB)

Installation

Racks ready hardware and software installations and their configurations for ready-to-be-deployed on RDTE network

As per the SOW

STIGS

Help to the NSWC

IHD IT Staffs during

STIGS

Stress Tests

Entire System Loaded

100% For Specified

Time After installation at NSWC IHD

Training Training on site NSWC IHD IT Staffs as specified in this SOW after installation and at the end of one year as described above

WARRANTY / SUPPORT

Hardware /

Software

5 Yrs. Full Hardware / Software Maintenance Contract with 24x7

Telephone Support and Next Business Day (NBD) Hardware

Replacement Support by a U.S. Citizen and Security Cleared Support

Personnel. Vendor / manufacturer (solely or combined) will be solely responsible for this 5 years warranty / support although some other vendor may be the manufacturer of the part/s to be replaced. NSWC

IHD will hold vendor / manufacturer responsible and the sole point of contact for any warranty/support needs. UPS batteries are known to fail before their stipulated lifetime, or their lifetime may be less than

5 years. Vendor may include the replacement cost in the total cost of UPS in the quote.

NOTES

A

In the event of power failure, vendor will determine the time needed to terminate jobs waiting in the SLURM queues, terminate all the ongoing users' runs, and safely shutdown the machine. UPS capacity must provide the power needed to safely execute this shutdown sequence without any stress to the machine. The machine will be powered up manually after power restoration. Users will restart the terminated runs from initial state or backup as needed.

B

All hardware must be TAA and GSA compliant. This must be noted in the quote either in the beginning or at the end. Vendor must provide the list of any non-compliant components at the end of the quote. NSWC IHD reserves the right at its discretion to approve or disapprove such non-compliant components. The disapproval will lead to rejection without any discussion with the vendor.

C

TPM 2.0 or Higher must be enabled on all motherboards. This must be noted in the beginning or at the end of the quote.

D

NO WIRELESS / BLUETOOTH – It must be noted at the beginning or at the end of the quote and the HPC must not have any Wireless or Bluetooth hardware.

E

NO HARDWARE / SOFTWARE RAID – It must be noted at the beginning or at the end of the quote and the HPC must not have Hardware or Software enabled RAID.

SECTION SF 1449 - CONTINUATION SHEET

SOLICITATION/CONTRACT FORM

The required response date/time has changed from 12-May-2023 02:00 PM to 18-May-2023 02:00 PM.

SUPPLIES OR SERVICES AND PRICES

SUBCLIN 0001AA

The CLIN extended description has changed from:

Login Nodes Key Component are:(a) Number of Nodes - 4(b) Base Clock Frequency of Processors - 2.2 GHz(c)

Total Number of Cores - 256(d) RAM Memory Per Core - 8 GB DDR4 3200 ECC(e) Total RAM Memory per Node

- 2.05 TB DDR4 3200 ECC(f) Hard Drive Per Node - 6.4 TB NVMe (g)

Total Hard Drives - 25.6 TB NVMe

To:

Login Nodes Key Component are:(a) Number of Nodes - 4(b) Number of Processor per Node - 2(c) Base Clock

Frequency of Processor - 2.2 GHz(d) Total Number of Cores per Node -64(e) Total Number of Cores - 256(f) RAM

Memory Per Core - 8 GB DDR4 3200 ECC(g) Total RAM Memory per Node - 512 GB DDR4 3200 ECC(h) Total

RAM Memory - 2.05 TB DDR4 3200 ECC (i) Hard Drive Per Node - 6.4 TB NVMe(j) Total Hard Drives - 25.6 TB

NVMe

SUBCLIN 0001AB

Master Nodes Key Components are:(a) Number of Nodes - 2(b) Base Clock Frequency of Processors - 3.2 GHz(c)

Total Number of Cores per Node - 64(d) RAM Memory Per Core - 8 GB DDR4 3200 ECC(e) Total RAM Memory per Node - 512 GB DDR4 3200 ECC(f) Hard Drive Per Node - 6.4 TB NVMe

(g) Total Hard Drives 12.8 TB NVMe

Master Nodes Key Components are:(a) Number of Nodes - 2(b) Number of Processors per Node - 2(c) Base Clock

Frequency of Processor - 3.2 GHz(d) Total Number of Cores per Node - 32(e) Total Number of Cores - 64(f) RAM

Memory Per Core - 8 GB DDR4 3200 ECC(g) Total RAM Memory per Node - 256 GB DDR4 3200 ECC(h) Total

RAM Memory - 512 GB DDR4 3200 ECC(i) Hard Drive Per Node - 6.4 TB NVMe(j) Total Hard Drives - 12.8 TB

SUBCLIN 0001AC

Compute Nodes without GPU Key Component are:(a) Number of Nodes - 24(b) Base Clock Frequency of

Processors - 2.2 GHz(c) Total Number of Cores - 1536(d) RAM Memory Per Core - 4 GB DDR4 3200 ECC(e)

Total RAM Memory per Node - 6.14 TB DDR4 3200 ECC(f) Hard Drive Per Node - 1.6 TB NVMe

(g) Total Hard Drives 38.4 TB NVMe

Compute Nodes without GPU Key Component are:(a) Number of Nodes - 24(b) Number of Processors per Node -

2(c) Base Clock Frequency of Processor - 2.2 GHz(d) Total Number of Cores per Node - 64(e) Total Number of

Cores - 1536(f) RAM Memory Per Core - 4 GB DDR4 3200 ECC(g) Total RAM Memory per Node - 256 GB

DDR4 3200 ECC(h) Total RAM Memory - 6.14 TB DDR4 3200 ECC(i) Hard Drive Per Node - 1.6 TB NVMe(j)

Total Hard Drives - 38.4 TB NVMe

SUBCLIN 0001AD

Compute Nodes with GPU Key Component are:(a) Number of Nodes - 8(b) Base Clock Frequency of Processors -

2.2 GHz(c) Total Number of Cores- 1536(d) RAM Memory Per Core - 4 GB DDR4 3200 ECC(e) Total RAM

Memory - 6.14 TB DDR4 3200 ECC(f) Hard Drive Per Node - 1.6 TB NVMe(g) Total Hard Drives - 38.4 TB

Compute Nodes with GPU Key Component are:(a) Number of Nodes - 8(b) Number of Processors per Node - 2(c)

Base Clock Frequency of Processor - 2.2 GHz(d) Total Number of Core per Node - 64(e) Total Number of Cores -

512(f) RAM Memory Per Core - 4 GB DDR4 3200 ECC(g) Total RAM Memory per Node - 256 GB DDR4 3200

ECC(h) Total RAM Memory - 2.05 TB DDR4 3200 ECC(i) Hard Drive Per Node - 1.6 TB NVMe(j) Total Hard

Drives - 12.8 TB NVMe(k) NVIDIA A100 40 GB PCIe GPU per Node- 1(l) Total NVIDIA A100 40 GB PCIe

GPUs - 8

SUBCLIN 0001AE

Large Memory Compute Nodes with GPU Key Component are:(a) Number of Nodes - 8(b) Base Clock Frequency of Processors - 2.2 GHz(c) Total Number of Cores - 512(d) RAM Memory Per Core - 8 GB DDR4 3200 ECC(e)

Total RAM Memory - 4.1 TB DDR4 3200 ECC(f) Hard Drive Per Node - 1.6 TB NVMe

(g) Total Hard Drives - 12.8 TB NVMe(h) NVIDIA A100 40 GB PCIe GPU - 1

(i) Total NVDIA a100 40 GB P Cle GPUs - 8

Large Memory Compute Nodes with GPU Key Component are:(a) Number of Nodes - 8(b) Number of Processors per Node - 2(c) Base Clock Frequency of Processor - 2.2 GHz(d) Total Number of Cores per Node - 64(e) Total

Number of Cores - 512(f) RAM Memory Per Core - 8 GB DDR4 3200 ECC(g) Total RAM Memory per Node - 512

GB DDR4 3200 ECC(h) Total RAM Memory - 4.1 TB DDR4 3200 ECC(i) Hard Drive Per Node - 1.6 TB NVMe(j)

Total Hard Drives - 12.8 TB NVMe(k) NVIDIA A100 40 GB PCIe GPU per Node - 1(l) Total NVIDIA A100 40

GB PCIe GPUs - 8

SUBCLIN 0001AF

Disk Node KeyComponent are:(a) Number of Nodes - 1(b) Base Clock Frequency of Processors - 3.0 GHz(c) Total

Number of Cores - 32(d) RAM Memory Per Core - 8 GB DDR4 3200 ECC(e) Total RAM Memory - 256 GB DDR4

3200 ECC(f) Hard Drive 480 GB SATA SSD Per Node - 2(g) Hard Drive 18TB SAS 12G HDD - 12

Disk Node Key Component are:(a) Number of Nodes - 1(b) Number of Processors per Node - 2(c) Base Clock

Frequency of Processor - 3.0 GHz(d) Total Number of Cores per Node - 32(e) Total Number of Cores - 32(f) RAM

Memory Per Core - 8 GB DDR4 3200 ECC(g) Total RAM Memory per Node - 256 GB DDR4 3200 ECC(h) Total

RAM Memory - 256 GB DDR4 3200 ECC(i) 480 GB SATA SSD Hard Drive Per Node - 2(j) 18TB SAS 12G HDD

Hard Drives - 12

The following have been added by full text:

REQUEST FOR INFORMATION (RFI)

RFI Supercomputer - N0017423R0028

1.) Q: Are you looking or prefer any specific brand for the nodes?

A: No. Must meet requirements provided.

2.) Q: How come there is no specification for the CPUs in the list of items?

A: Refer to sections 2.1, 2.2, 3, 6.9, 8.6, and Table 1

3.) Q: Will NAVSEA NSWC ID consider a cloud-based HPC offering as part of this RFQ?

A: No

4.) Q: What is the facilities’ electrical power limit for power/cooling?

A: There are no limits. Refer to section 9.1 for the requirement of the final vendor's site visit for power/cooling.

5.) Q: Are there any rack size and weight limitations?

A: No

6.) Q: There are number of places in the RFQ document that mentions “See

Schedule” but it’s not clear where in the document the “Schedule” is defined.

A: Schedule refers to the entire document as given in the field 20 on page 1.

Specific Delivery Schedule Information can be found under Delivery

Information on Page 31 and 32.

7.) Q: What is the expected delivery time frame after award?

A: The government requests delivery 4 months after date of contract. Please include your delivery date in the proposal that meets that requirement. Should you not be able to meet that date please provide an alternate date with justification for not being able to meet the date.

8.) Q: For DFARS 252.211-7003, will the customer provide barcoding the equipment?

A: Yes. NSWC IHD has strict regulation to barcode the equipment and assign the Asset Number. However, vendors can choose to barcode equipment per their convention for their easy reference in the future.

9.) Q: We noticed that Section 2.12 is missing. Please advise.

A: Acknowledged and Scope has been corrected in Amendment. The present

2.6 will become 2.7, 2.7 will become 2.8, 2.8 will become 2.9, 2.9 will become 2.10, 2.10 will become 2.11, and 2.11 will become 2.12.

10.) Q: In Section 8.3 “.... with U.S Citizenship and a security clearance …" What level of

Security Clearance is required?

A: Secret

11.) Q: Compute and all other node types than management – Is the notion of a provisioning system with ‘stateless’ OS – that is, nodes netboot from a centralized ‘golden image’ every time, with no permanent OS installed on individual systems’ drives – acceptable for consideration? Or is an operating system to be installed and individually managed on every node a hard requirement? Disks for swap/scratch would still be included and accessible, as per other requirements.

A: Stateless OS will not be accepted. OS installed and managed on individual nodes is a hard requirement.

12.) Q: Section 2.8 specifies the Infiniband network to be able to support ’50 concurrent users without network latency’. Section 6.5 specifies that it must be able to provide ’connections to 50 users without affecting network bandwidth available on the HPC’. These are two different metrics – looking for clarity on the exact type of metrics expected regarding fabric performance at the 50 concurrent user level.

A: High bandwidth and low latency applies to the Infiniband and Switches.

13.) Q: Section 9.4 – Stress test. What is the anticipated time for this test? Is there a specific benchmark or set of benchmarks desired for this Stress Test?

A: Stress test is a standard procedure for HPC. This is the last operation after installing all hardware, software, and configuring them to the needs. Vendors can choose their own software for the stress test to fully load and run the machine for a minimum of 24 hours to detect any anomalies or failing / failed parts.

14.) Q: The disk node is to contain, at a minimum, 12 18TB SAS 12G HDD Hard Drives.

Section 2.13 Specifies No Hardware or Software RAID to be utilized anywhere in the system. There are alternative ways of presenting this storage space in a unified fashion to other clients (if that is the intent), but depending on the technology used they would potentially forego any sort of active data parity/protection at a device level. This would mean that the loss of one device would immediately result in the loss of all data. Would just like to confirm that the strict prohibition against HW/SW RAID of any kind in the system takes this into account, or that the intended configuration/use of these disks has taken this into consideration.

A: There are no intentions to provision the disk node to any other client.

Hence, the disk node MUST NOT BE CONSIDERED A STORAGE

SERVER as implied by this question. Purpose of the disk node is to provide larger disk space to users for writing output files during code execution and long term storage of their data. It has to be configured in a way that code executing on any node should be able to write the output on the users' space assigned on the disk node.

15.) Q: There are no specific requirements for node metrics or monitoring collection other than being able to “display of the system, and system status, and to query, administer, explore, and manage individual nodes, as well as the HPC as a whole.”

Are there any further specifics on expectations for what items are able to be ‘queried/explored’ on a node and HPC-wide basis?

A: Monitoring collection will be through Auditing which is a part of STIGing.

This question pertains to the Cluster Management Software to be provided by the vendor, which will be accessed through the display terminal and also remotely by the system administrator. The vendor will explain all the features of the software and configure as needed during the installation and training.

16.) Q: In reference to item No 0001AD: Failed to mention GPU per node plus Total Hard

Drives - 38.4 TB NVMe is not correct. We believe it should be 12.8 TB (=1.6 TB x 8).

A: Compute Nodes with GPU after correction will show: (c) Total Number of

Co…