Site Visit Presentation 2017 GFDL

NOAA R&D Site Visit

High Performance Computing and Communications (HPCC)

General Information

● Restrooms - See the Site Visit Lead or presenters for details

● Cell Phones - Please turn off. No calls or pictures allowed during Site Visit

● Site Visit & Procurement - Part of the Acquisition Planning process:

○ A Request for Proposal (RFP) has not yet been issued; a Draft RFP was released on www.fbo.gov for industry review on 14 July 2017

○ All technical and acquisition-related information discussed today is subject to change

○ Any conflicts between information provided today and future

Government correspondence shall be resolved in favor of the Final RFP

● Asking Questions - Use the index cards provided for submitting questions

● Access to Questions/Answers - All questions and answers provided today will be posted to www.fbo.gov following today’s event

● Other Items - An Attendees List and an electronic copy of the presentation will be posted to www.fbo,gov shortly after today’s event http://www.fbo.gov http://www.fbo.gov

Agenda 10:00 AM Introduction

10:10 AM NOAA Research and Development High Performance Computing Presentation

10:30 AM Questions and Answers (First Round)

11:15 AM Site Tours (4 Group Tours Planned)

12:45 PM Questions and Answers (Second Round)

02:00 PM Closing Remarks

02:15 PM Adjourn

Introduction - Acquisition Management Personnel

Site Visit Lead Personnel:

● Fairmont, WV - Mr. Frank Indiviglio

● Princeton, NJ - Dr. Bernie Siebers

● Boulder, CO - Mr. Forrest Hobbs

Acquisition Management Personnel:

● Program Manager - Dr. Brian Gross

● Branch Chief/Contracting Officer - Mr. McKenzy Hunter

● Contracting Officer/Contract Specialist - Ms. Claudia S. Womble

● Contract Specialist - Mr. Berchan Torrejon

● Contracting Officer’s Representative (COR) - Dr. Bernie Siebers

● Alternate Contracting Officer’s Representative (COR) - Mr. Frank

Indiviglio

NOAA’s Mission

To understand and predict changes in Earth’s environment and conserve and manage coastal and marine resources to meet our nation’s economic, social, and environmental needs

NOAA Is Vital to American Economy

One-third of GDP ($4 trillion) is reliant on accurate weather and climate information.

Example of NOAA’s role:

NOAA provides weather, water, and climate forecasts and warnings for the private and public sectors.

Annually, NOAA provides 76 billion observations, 1.5 million forecasts, and 50,000 warnings.

-NOAA provides economic benefits of $240 million per year in mitigating flood losses -NOAA’s aviation forecasts reduce aviation delays and save the industry $580 million per year.

NOAA’s improved hurricane forecasts save taxpayers $640,000 per non-evacuated mile

NOAA’s role:

NOAA has improved wind speed estimates by 15 percent since 2004 and reduced track forecast error by 50 percent since 1990

NOAA Saves Lives

On average, hurricanes account for 20 deaths per year and $5.1 billion in damage.

Minutes

Days/Hours

Weeks

Seasons

Years

T ra ns po rt at io n

Fo re ca st

L ea d T im e

Warnings & Alert

Coordination

Watches

Forecasts

Threats Assessments

Guidance

Outlook

Pr ot ec tio n of

L ife

Pr op er ty

Sp ac e O pe ra tio n

R ec re at io n

E co sy st em

St at e/ L oc al

Pl an ni ng

E nv ir on m en t

Fl oo d M iti ga tio n

N av ig at io n

A gr ic ul tu re

R es er vo ir

C on tr ol

E ne rg y

C om m er ce

Benefits

H yd ro po w e r

Fi re

W ea th er

H ea lth

Continuum of Products

Decades Centuries

Reports

High Performance Computing Locations

Research and Development (R&D) HPC Systems

Operational HPC Systems

Boulder, CO Development HPC

• nJet (421 Tflops)

• xJet (720 Tflops)

Fairmont, WV Development HPC

• Theia (3,000 Tflops)

• Zeus (383 Tflops)

Oak Ridge, TN Research HPC

• Gaea (4,000 Tflops)

Princeton, NJ Post-Processing & Analysis

Reston, VA

• Tide (830 Tflops)

• Luna (2,060 Tflops)

Orlando, FL

• Gyre (830 Tflops)

• Surge (2,060 Tflops)

Cross Site IT Infrastructure

● NOAA’s R&D HPCS Infrastructure includes the following:

○ Networking

■ Internet and N-wave connection points for logins and data transfer

■ Perimeter security via firewalls located at each of the major sites;

ESRL, GFDL, NCEP, and NESCC

■ Layered DMZ strategy

○ IT Services

■ Core services which allow NOAA’s users and administrators to securely access and use R&D HPCS resources

■ Virtual Machines (VM) which allow multiple services to be isolated from one another but utilize shared hardware

■ A common set of hardware deployed across three sites; ESRL, GFDL, and NESCC

■ A common set of Enterprise tools for deploying, configuring and managing servers and services

○ Account Management and AIM System

R&D Supercomputing N-Wave Network

What is N-Wave?

• NOAA high bandwidth, low latency network connecting scientists to HPC

• Built using multiplexed fiber-optic links supplied by national Research and

Education network community including Internet2 and university run regional network consortiums

Infrastructure Services

● Enterprise Firewalls, Routers, Switches, and Network Test Servers are deployed and maintained across four different sites.

● Ten Core services are supported

○ Login Bastions

○ CAC/PIV authentication

○ RSA authentication

○ Directory Services

○ X.509 User and Host Certificates

○ Domain Name Services

○ Data Transfer Nodes

○ Mail Relay

○ Network Time Protocol

○ Task tracking and Change Management sites

Infrastructure Services (Cont.)

● Six Supporting services are supported

○ Configuration Management and Task Tracking

○ Operating System Mirrors/Repositories

○ Backup Systems

○ Automated Monitoring and Alerting

○ Centralized System Logging

○ Log Parsing and Analysis

● Infrastructure Hosted Services

○ DocuWiki sites for user and administrator documentation

○ Software Version Control

○ Help Desk

○ HPC Dashboard and Status page

○ Software mirrors

● Approximately 60 VMs deployed across the program

Princeton- Climate PP&A Cluster

Facility Background

The Computer Building is approximately 18,000 square feet, of which the data center occupies 10,000 square feet; the remainder is occupied by mechanical/electrical support spaces, offices, loading dock and the operations center. Of the 10,000 sf data center, 7,000 sf is raised floor.

The raised floor is twenty four inches (24”) in depth and consists of a ConCore 2500 Raised Floor Panel vented using the DirectAire All Steel (AirFlow) Panel with Tate Drop-In Manual Opposed Blade Dampers and the SmartAire Automatic VAV (variable air volume) Control System.

The hard deck is concrete cladded with armstrong VCT (Vinyl CompositionTile).

Currently, there is approximately 2,600 square feet of available raised floor space within the facility and 1,800 sf non-raised space.

Additional space could be made available through the relocation of the SL8500 Silo Library II. to the hard deck or the consolidation of the existing equipment to a more optimized layout.

The site is owned by Princeton University and leased to NOAA.

Princeton- Climate PP&A Cluster

Princeton- PP&A Cluster

Fire Detection, Suppression and Alarm Systems

Detection by way of VESDA and standard detectors. Suppression by way of FM-200 and pre-action sprinkler system. Alarm system is Simplex 4100U.

Power Distribution System

The primary voltage of the electrical system is 13.2 KV utility service with provisions for a future service connection. The utility service feeds a 3.75MW 13.2kV/4,160V transformer (installed 2011) which then services the 4,160V switchgear. The 4,160volt switchgear has a spare cubicle available for future expansion.

The 4,160V switchgear feeds two (2), 4,160/480 volt, 3 phase, 4 wire free standing transformers. One transformer is specific to the data center and the second transformer is specific to the office building. Each transformer feeds a main 480/277 volt switchboard which subsequently serves 480/277 volt and 120/208 volt distribution panelboards. Current utilization hovers around 600KW which leaves 3,150KW in spare capacity.

The 480V distribution panels provide service to two (2) 800 KVA UPS units. The UPS units will each provide a total of 15 minutes of runtime after a power event. UPS-A is the primary UPS for the data center and UPS-B is connected to the bypass circuit of UPS-A. If UPS-A fails, the UPS-B is automatically connected to the load without interruption of service. However, due to facility protocol, both of the UPS systems are programmed to shut down at end of factory recommended maximum battery life. UPS-A provides power to an electrical panelboard which then provides power to three (3) Eaton Powerware power distribution unit (PDU) cabinets. Each PDU serves multiple PDI remote power panels located throughout the data center.

There is one spare PDU (No. 02) available as an on site hot swap configuration. Current UPS utilization hovers around 225KW which leaves 575KW redunant with an additional 800KW in reserve.

Emergency Standby Diesel Generator Power

The existing emergency power for the facility is provided by one (1) – 80 kilowatt (80 kW) natural gas fueled generator set. The generator is dedicated to one remote power distribution panel labeled RPP-Gen. It currently provides backup power to the HFS, Data Portal and GFDL IT cabinets. It also provides emergency power and lighting for the operations center as part of its configuration.

GFDL Generator Enclosure Project (Substantial Completion date 11/30/17)

NOAA is preparing to construct a masonry enclosure, at the GFDL facility, to house three (3) one (1) mega-watt standby diesel generators. NOAA has purchased two (2) of the three (3) standby diesel generators and has designed a solution with integrated capabilities to add a third one (1) mega-watt generator along with the associated mechanical and electrical equipment including the automated transfer switch, paralleling gear, unit heaters, electrical panel and a fire alarm panel to support this configuration.

The units will provide emergency standby power to completely backup both the main building and the computer building with 400KW of spare capacity in an N=1 configuration.

Air Handling Equipment and Systems

Cooling is provided via down-flow chilled water Computer Room Air Handling units (CRAHs) which are installed within the NOAA High Performance Data Center Area. There are also additional chilled water air handlers serving other areas such as the UPS room, Office area, and Printer Room. The current data center load is 1350 gpm (approximately 563 tons). The additional air handlers require approximately 471 gpm (approximately 200 tons) of cooling capacity as well.

Hydronic Piping and Pumping Systems

The chilled water system equipment consists of water-cooled chillers, cooling towers, condenser water pumps, primary chilled water pumps, and secondary chilled water pumps. There are three chillers, Chiller #1, Chiller #3, and Chiller #4, which are served by three cooling towers.

Chiller #1 is a 400 ton (960 gpm) centrifugal chiller, served by a dedicated primary chilled water pump (960 gpm). Chiller #1 is supported by a two speed cooling tower #1 and dedicated condenser water pump (1543 gpm) COWP-1.

Chiller #3 is a 350 ton (840 gpm) centrifugal chiller, served by a dedicated primary chilled water pump (840 gpm). Chiller #3 is supported by Cooling Tower #3, which is equipped with a variable speed drive, and dedicated condenser water pump (1050 gpm) COWP-3.

Chiller #4 is a 250 ton (600 gpm) rotary screw chiller, served by two (lead-lag) dedicated primary chilled water pumps (600 gpm each). Chiller #4 is supported a single speed Cooling Tower #4, which is equipped with a variable speed drive, and two (lead-lag) dedicated condenser water pumps (750 gpm each) COWP-4A and COWP-4B.

\] 18

Mechanical System Recommendations (5 Year Plan)

Currently the central plant equipment serving the NOAA high performance data center area possesses sufficient cooling capacity to provide approximately 10 tons of cooling to support the five year outlook plan to install an additional 0.1 MW of computing capacity throughout the life of the plan.

Electrical System Recommendations (5 Year Plan)

The existing electrical system expansion capabilities were evaluated as part of the five year outlook plan for the facility with a projected load of approximately 100KW. The new projected load for the facility of 700KW could easily be handled by our existing infrastructure with over 3.0MW to spare.

Additional improvements

In September, GFDL will be installing five (5) sets of Cold Isle Containment for the PPA, DMF and GFDL IT Cabinets. Five sets of doors along with a containment header over each row will be placed.

Suggestions

It has been suggested by the R&D HPC Program that Library II. SL8500 Silo should be relocated from the raised floor to the hard deck. This relocation will allow for an additional 600 square foot of raised floor be available for future planning and use.

Thank You

Please Provide Question Cards to Government Representatives

Site Tour Schedules

Fairmont, WV Princeton, NJ* Boulder, CO

Tour 1 11:15am – 12:15pm 11:15am – 12:00pm 11:15am – 11:45am

Tour 2 11:15am – 12:15pm 11:15am – 12:00pm 11:15am – 11:45am

Tour 3 11:50am – 12:50pm 12:05pm – 12:35pm 11:50am – 12:20pm

Tour 4 11:50am – 12:50pm 12:05pm – 12:35pm 11:50am – 12:20pm

*Row in bold indicates which schedule to follow.