ReleaseIDIQ_Appendix_A.1-Boulder-2017.pdf

NOAA R&D HPCS RFP

Appendix A.1

Detailed HPCS Architecture Boulder

16 January 2017

Boulder Facility Organization and Subsystems Overview

The Boulder HPC systems are divided amongst two rooms in the David Skaggs Research Center (DSRC) in Boulder, CO. Room 2B407 is primarily for low power density air cooled equipment, such as parallel file systems, Front Ends/Login nodes, Batch nodes, Infrastructure servers, and Home File System. There is also a 288 node compute cluster located in 2B407. However, this cluster requires a special air curtain for cold air isolation.

Room GA405 is for high density air cooled equipment such as compute racks. Each compute rack is comprised of 60-70 nodes, IB switches and Ethernet switches.

Four of the five compute clusters are located on the GA405 room. Both rooms are connected via 18 active, 150 meter, QDR IB links. So all compute systems reside on the same IB fabric.

Jet is comprised of 5 separate clusters with a total core count of 45,808 cores. A single job cannot span multiple clusters. All clusters can be accessed through the Batch system from the common Front-ends/Login nodes.

System Intel Processor

Install Date

Clock (GHz)

Nodes Total Cores

Memory Per Node

(GB)

Memory Per Core

(GB)

tJet Westmere 8/2010 2.66 758 9096 24 2 uJet Westmere 11/2011 2.66 590 7080 24 2 sJet Sandy Bridge 8/2012 2.6 340 5440 32 2 vJet Ivy Bridge 8/2014 2.6 288 4608 64 4 xJet Haswell 9/2015 8/2016*

2.3 816 19584 64 2.67

* xJet expansion

Jet has approximately 4.82PB of scratch storage. This is across 3 parallel file-systems (2 DDN Lustre and 1 Panasas). Combined peak performance of all three scratch file systems is ~46GB/s. All clusters reside on the same High Speed Interconnect/Infiniband fabric, making the parallel file systems globally accessible from all nodes. There is also a Home File System (HFS) which is globally accessible from all nodes over the Ethernet network.

The Home File System (HFS) and other NFS file systems are being served by a NetApp FAS8120. The HFS is globally accessible from all nodes over the Ethernet network.

The HFS and HPC infrastructure servers are backed up on a daily and weekly basis to a combination of spinning disk and tape. The Open Source software Amanda is used for all backups.

There is a tape storage archive located at the NESCC site in Fairmont, WV. Data can be transferred to and from this archive from Jet’s Front Ends. The retention period for the archive is 1-5 years or permanent. All projects with a compute allocation are allowed to store data on the archive.

Data transfer services are provided by a combination of 10GbE connected Data Transfer Nodes (DTN’s) and the 10GbE connected Front Ends. The DTN’s are used when a data transfer is initiated from outside of Jet. The Jet FE’s are used when a data transfer is initiated from Jet.

Jet’s Batch system is comprised of three separate pieces of software. Torque is the resource manager and is responsible for job submission, node health status and job launching. Moab is the scheduler and is responsible for scheduling jobs based on priority and managing dedicated system resources with reservations.

MAM in the allocation manager and is responsible for managing the monthly allocation of core hours for each project. MAM and Moab are also used for the daily and monthly reporting of system utilization.

Diagrams