Site Visit Presentation 2017 ESRL

NOAA R&D Site Visit

High Performance Computing and Communications (HPCC)

General Information

● Restrooms - See the Site Visit Lead or presenters for details

● Cell Phones - Please turn off. No calls or pictures allowed during Site Visit

● Site Visit & Procurement - Part of the Acquisition Planning process:

○ A Request for Proposal (RFP) has not yet been issued; a Draft RFP was released on www.fbo.gov for industry review on 14 July 2017

○ All technical and acquisition-related information discussed today is subject to change

○ Any conflicts between information provided today and future

Government correspondence shall be resolved in favor of the Final RFP

● Asking Questions - Use the index cards provided for submitting questions

● Access to Questions/Answers - All questions and answers provided today will be posted to www.fbo.gov following today’s event

● Other Items - An Attendees List and an electronic copy of the presentation will be posted to www.fbo,gov shortly after today’s event http://www.fbo.gov http://www.fbo.gov

Agenda 10:00 AM Introduction

10:10 AM NOAA Research and Development High Performance Computing Presentation

10:30 AM Questions and Answers (First Round)

11:15 AM Site Tours (4 Group Tours Planned)

12:30 PM Questions and Answers (Second Round)

02:00 PM Closing Remarks

02:15 PM Adjourn

Introduction - Acquisition Management Personnel

Site Visit Lead Personnel:

● Fairmont, WV - Mr. Frank Indiviglio

● Princeton, NJ - Dr. Bernie Siebers

● Boulder, CO - Mr. Forrest Hobbs

Acquisition Management Personnel:

● Program Manager - Dr. Brian Gross

● Branch Chief/Contracting Officer - Mr. McKenzy Hunter

● Contracting Officer/Contract Specialist - Ms. Claudia S. Womble

● Contract Specialist - Mr. Berchan Torrejon

● Contracting Officer’s Representative (COR) - Dr. Bernie Siebers

● Alternate Contracting Officer’s Representative (COR) - Mr. Frank

Indiviglio

NOAA’s Mission

To understand and predict changes in Earth’s environment and conserve and manage coastal and marine resources to meet our nation’s economic, social, and environmental needs

NOAA Is Vital to American Economy

A third of the GDP ($4 trillion) is reliant on accurate weather and climate information.

Example of NOAA’s role:

NOAA provides weather, water, and climate forecasts and warnings for the private and public sectors.

Annually, NOAA provides 76 billion observations, 1.5 million forecasts, and 50,000 warnings.

-NOAA provides economic benefits of $240 million per year in mitigating flood losses -NOAA’s aviation forecasts reduce aviation delays and save the industry $580 million per year.

NOAA’s improved hurricane forecasts save taxpayers $640,000 per non-evacuated mile

NOAA’s role:

NOAA has improved wind speed estimates by 15 percent since 2004 and reduced track forecast error by 50 percent since 1990

NOAA Saves Lives

On average, hurricanes account for 20 deaths per year and $5.1 billion in damage.

Minutes

Days/Hours

Weeks

Seasons

Years

T ra ns po rt at io n

Fo re ca st

L ea d T im e

Warnings & Alert

Coordination

Watches

Forecasts

Threats Assessments

Guidance

Outlook

Pr ot ec tio n of

L ife

Pr op er ty

Sp ac e O pe ra tio n

R ec re at io n

E co sy st em

St at e/ L oc al

Pl an ni ng

E nv ir on m en t

Fl oo d M iti ga tio n

N av ig at io n

A gr ic ul tu re

R es er vo ir

C on tr ol

E ne rg y

C om m er ce

Benefits

H yd ro po w e r

Fi re

W ea th er

H ea lth

Continuum of Products

Decades Centuries

Reports

High Performance Computing Locations

Research and Development (R&D) HPC Systems

Operational HPC Systems

Boulder, CO Development HPC

• nJet (421 Tflops)

• xJet (720 Tflops)

Fairmont, WV Development HPC

• Theia (3,000 Tflops)

• Zeus (383 Tflops)

Oak Ridge, TN Research HPC

• Gaea (4,000 Tflops)

Princeton, NJ Post-Processing & Analysis

Reston, VA

• Tide (830 Tflops)

• Luna (2,060 Tflops)

Orlando, FL

• Gyre (830 Tflops)

• Surge (2,060 Tflops)

Cross Site IT Infrastructure

● NOAA’s R&D HPCS Infrastructure includes the following:

○ Networking

■ Internet and N-wave connection points for logins and data transfer

■ Perimeter security via firewalls located at each of the major sites;

ESRL, GFDL, NCEP, and NESCC

■ Layered DMZ strategy

○ IT Services

■ Core services which allow NOAA’s users and administrators to securely access and use R&D HPCS resources

■ Virtual Machines (VM) which allow multiple services to be isolated from one another but utilize shared hardware

■ A common set of hardware deployed across three sites; ESRL, GFDL, and NESCC

■ A common set of Enterprise tools for deploying, configuring and managing servers and services

○ Account Management and AIM System

R&D Supercomputing N-Wave Network

What is N-Wave?

• NOAA high bandwidth, low latency network connecting scientists to HPC

• Built using multiplexed fiber-optic links supplied by national Research and

Education network community including Internet2 and university run regional network consortiums

Jet Network

Infrastructure Services

● Enterprise Firewalls, Routers, Switches, and Network Test Servers are deployed and maintained across four different sites.

● Ten Core services are supported

○ Login Bastions

○ CAC/PIV authentication

○ RSA authentication

○ Directory Services

○ X.509 User and Host Certificates

○ Domain Name Services

○ Data Transfer Nodes

○ Mail Relay

○ Network Time Protocol

○ Task tracking and Change Management sites

Infrastructure Services (Cont.)

● Six Supporting services are supported

○ Configuration Management and Task Tracking

○ Operating System Mirrors/Repositories

○ Backup Systems

○ Automated Monitoring and Alerting

○ Centralized System Logging

○ Log Parsing and Analysis

● Infrastructure Hosted Services

○ DocuWiki sites for user and administrator documentation

○ Software Version Control

○ Help Desk

○ HPC Dashboard and Status page

○ Software mirrors

● Approximately 60 VMs deployed across the program

Boulder DOC Campus

DSRC

Shipping/Receiving

Security

DSRC Building

GA405

2B201

A B

C

D

A

A

2A305 Meeting Room

Rest Rooms

Lobby

Garden Level (basement)

N

B

2nd Level Facility

Monitoring

HPC Facilities

• DSRC Infrastructure

• Power:

• 2x 1250KW diesel generators

• TVSS Throughout the building

• EPO Protection

• Cooling:

• Towers (4 - Sized to match the capacity of each Chiller)

• Chillers (4x470Ton, 3 are generator backed)

• Flatplate Heat Exchanger (weather permitting)

• 4-Primary Pumps; 4-Secondary Pumps; 4-Condenser Water Pumps

• Fire Detection and Suppression

• Wet pipe sprinkler throughout the building

• Cerberus smoke detection throughout

• Fire Extinguishers:

• CO2 within the Data Centers

• FE-36 Throughout the hallways

• 2B201

• ~ 2400 Sqft for HPC (1998)

• ~200KVA for HPC

• 3 Generator backed-up UPS’s. 1x 225 KVA, 2x 100KVA (install: 2016/2017)

• Underfloor 120/208V single phase whips

• Cooling: 12” Underfloor air via 4 x 30 ton CRAC’s

• Fire Suppression

• Wet Pipe Sprinkler system (155 deg.F) GSA Requirement

• Cerberus Smoke Detection - GSA System

• VESDA Detection

• Clean Agent Suppression (FM-200)

• GA405

• ~2100 sqft. ,~46 rack slots (2x4’), (const: 2006/2010)

• ~800 KVA. 2x 500KVA UPS’s(install: 2016)

• UPS-1: Underfloor 208V single and 3-phase whips

• UPS-2: Overhead 48x 3 pole 20A/240V IEC309 whips

• 2x APC 266KVA Modular PDU’s, 480V:415V, (12x42”)

• No generator backup. Ride through ~10 min outages.

• Air Cooling: 24” underfloor (6x 30T CRAC’s) and overhead (2x 40T XDP)

• Fire Suppression

• Wet Pipe Sprinkler system (200 deg.F) GSA Requirement

• VESDA Detection

• Clean Agent Suppression (Ecaro)

HPC Facilities Monitoring

• Business hours monitoring

• 6am to 5pm, M-F (Excludes weekends and holidays)

• Monitor:

• DSRC and GSD Networks; N-Wave

• Data

• Physical Space

• SCADA Environmental Monitoring

• Jet System Status

HPC Facilities Monitoring

• After hours Monitoring

• Text and Email Notifications (24/7/365)

GA405

2B201

A B

C

D

A

A

Garden Level (basement)

N

B

2nd Level

2B201

~2400 sqft for HPC ~200KVA for HPC

- 1x225, 2x100 KVA UPS’s

- Generator Backup

- Underfloor 120/208V 1-ph whips ~4x 30 ton CRAC’s

GA405

~2100 sqft, ~46 (2x4’) rack slots 2x 500 KVA UPS’s,

- UPS-1: Underfloor 208V single and 3-phase whips

- UPS-2: Overhead 48x 3 pole 20A/240V IEC309 whips

- 2x APC 266KVA Modular PDU’s, 480V:415V, 12x42” 6x 30 ton CRAC’s, XDV/XDO

Inter-room IB cables

IB Aggregation

~2100 sqft, ~46 (2x4’) rack slots 2x 500 KVA UPS’s,

- UPS-1: Underfloor 208V single and 3-phase whips

- UPS-2: Overhead 48x 3 pole 20A/240V IEC309 whips

- 2x APC 266KVA Modular PDU’s, 480V:415V, (12x42”) 6x 30 ton CRAC’s, XDV/XDO

~2100 sqft, ~46 (2x4’) rack slots 2x 500 KVA UPS’s,

- UPS-1: Underfloor 208V single and 3-phase whips

- UPS-2: Overhead 48x 3 pole 20A/240V IEC309 whips

- 2x APC 266KVA Modular PDU’s, 480V:415V, (12x42”) 6x 30 ton CRAC’s, XDV/XDO

Jet Compute Systems

Jet Compute Rack Examples tJet 70 node rack Example xJet 60 node rack Example

Jet User File Systems

/lfs3 /lfs2 /pan2 /home

Type Scratch Scratch Scratch NAS

Tech Lustre/DDN Lustre/DDN Panfs NFS/ Netapp

Capacity 3.10 PB 1.07 PB 0.65 PB 26 TB

Performance ~23 GB/s ~16 GB/s ~12 GB/s +350 MB/s

Installed 2014-2016 2012 2011 2014

Jet IB HSN

Thank You

Please Provide Question Cards to Government Representatives

10:00 AM Introduction

10:10 AM NOAA Research and Development High Performance Computing Presentation

10:30 AM Questions and Answers (First Round)

11:15 AM Site Tours (4 Group Tours Planned)

12:30 PM Questions and Answers (Second Round)

RDHPCS Tour

GA405

2B201

A B

C

D

A

A

2A305 Meeting Room

Rest Rooms

Lobby

Garden Level (basement)

N

B

2nd Level Facility

Monitoring

Site Tour Schedules

Fairmont, WV Princeton, NJ Boulder, CO*

Tour 1 11:15am – 12:15pm 11:15am – 12:00pm 11:15am – 11:40am

Tour 2 11:15am – 12:15pm 11:15am – 12:00pm 11:25am – 11:50am

Tour 3 11:50am – 12:50pm 12:05pm – 12:35pm 11:45am – 12:10pm

Tour 4 11:50am – 12:50pm 12:05pm – 12:35pm 11:55am – 12:20pm

*Row in bold indicates which schedule to follow.

10:00 AM Introduction

10:10 AM NOAA Research and Development High Performance Computing Presentation

10:30 AM Questions and Answers (First Round)

11:15 AM Site Tours (4 Group Tours Planned)

12:30 PM Questions and Answers (Second Round)

Backup

GA405

2B201

A

BC

D

A A

2A305 Meeting Room

Rest Rooms

Lobby

Garden Level

N