NACS_Virtual_Pre-Solicitation_Conference_Presentation112816_final.pdf

National Aeronautics and Space Administration www.nasa.gov

NASA Advanced Computing Services (NACS)

November 28, 2016

Virtual Pre-Solicitation Conference

NNA17554082R

Agenda

• Procurement Process Overview Veronica L. Gutierrez, Contracting Officer

• Statement of Work/Technical Overview William Thigpen, Branch Chief, Advanced Computing

Daniel Q. Duffy, High Performance Computing Lead

• Open Questions and Answers

This Pre-Solicitation Conference is intended to:

• Provide the current status of the NASA Advanced Computing Services (NACS) acquisition;

• Familiarize participants with the Agency Overview and NACS Statement of Work (SOW)/technical requirements;

• Allow potential offerors an opportunity to submit questions regarding the recently posted Synopsis and NACS requirements.

General Guidance

• These slides shall not be interpreted as a comprehensive description of the Government’s requirements. Please refer to the Draft Statement of Work and Draft Request for Proposal (RFP).

• At the end of the Pre-Solicitation Conference we will allow for questions.

Questions related to this Pre-Solicitation Conference may also be submitted in writing to the Contracting Officer at Veronica.L.Gutierrez@nasa.gov. Presentation and all questions and answers will be made available on the Federal Business Opportunities (FBO) websites.

Procurement Process Overview

Freedom of Information Act (FOIA) FOIA Requests may be submitted electronically to:

Contract NNA07CA29C/ARC hq-foia@nasa.gov

Contract NNG13HQ01C/GSFC gsfc-foia@mail.nasa.gov

No proprietary information can be disclosed.

NACS Requirements Overview

• The NACS procurement is a consolidation of the following current contracts: NASA ARC, NASA

Supercomputing Support Services (NS3), NNA07CA29C and NASA GSFC, Computational and Information Science and Technology Office and Technical Services (CISTO-SCTS), NNG13HQ01C.

• A seven-month extension to the current NS3 contract is anticipated.

• NACS will provide world-class High Performance Computing (HPC) to all NASA missions in support of mission-driven science and engineering. In addition to supporting the NAS at ARC and the NCCS at GSFC, the NACS may be used to provide advanced computing services to other NASA Centers and Jet Propulsion Laboratory (JPL).

• North American Industry Classification System (NAICS) Code 541513 for Computer Facilities Management Services, with a size standard of $27.5 Million.

Acquisition Description

• Single Award Hybrid Cost Plus Award Fee (CPAF) and Cost Plus Fixed Fee (CPFF) contract consisting of a core contract management services and core technical requirements and Indefinite Delivery/Indefinite Quantity (IDIQ) requirements for additional technical requirements.

• The maximum contract value of the proposed contract is $1.1 Billion. The IDIQ contract line items will have a minimum ordering value of $100,000 and a maximum ordering value of $915 Million.

• Ten-year performance period consisting of a one-year Base Period, and nine one-year Option Periods.

A 30-day Phase-in Period will be included in the Base.

• Work is to be performed at NASA Ames Research Center and Goddard Space Flight Center.

General Procurement Information

• This procurement will be conducted through a Full and Open Competition pursuant to FAR and NFS Part 15- Contracting by Negotiations.

• FAR 52.215-1 and NFS 1815.209 allow for an award to be made without discussions. The Government may award a contract based solely on the initial proposals received. The Government reserves the right to hold discussions if award on the basis of initial offers is determined not to be in the best interest of the Government. If discussions are necessary, then a competitive range will be set and negotiations will commence. The initial proposals to the Government should contain the most favorable terms from a price and technical standpoint.

Source Evaluation Process

• ARC/GSFC intends to utilize the following three evaluation factors for the NACS

Procurement.

• Mission Suitability

• Past Performance

• Cost

• The Mission Suitability Factor is evaluated at the subfactor level and is the only factor scored. The Past Performance Factor is evaluated and given a Confidence Rating. The Cost/Price Factor is evaluated, but not scored.

• Use oral presentations for portions of Mission Suitability and Past Performance.

Source Evaluation Process (continued) Mission Suitability Factor The following sub-factors and points have been determined for the Mission Suitability Factor.

Mission Suitability Sub-factors Points Management Approach 500 Technical Approach 400 Small Business Utilization 100 Total 1000

(Small Business Utilization Elements: Small Business Subcontracting Plan and Commitment to the Small Business Program )

Source Evaluation Process (continued)

• This procurement will require submission of a Safety and Health Plan no later than 10 days after contract award.

Small Business Utilization

• The Government assessed the appropriate subcontracting goals for this acquisition. The small business goals for this procurement, expressed as a percent of total contract value are as follows:

Total Small Business (SB): 12%

Small Disadvantaged Business (SDB) Concerns 5.0%

Women Owned Small Business (WOSB) Concerns 4.0%

Historically Underutilized Business Zone (HUBZone) Zone 1.0%

Veteran Owned Small Business (VOSB) Concerns 2.0%

Service-Disabled Veteran-Owned Small Business (SDVOSB) Concerns 1.0%

• Large Businesses must submit Small Business Subcontracting Plan. Small Businesses are not required to submit Small

Business Subcontracting Plan; however, Small Businesses are required to indicate the amount of effort proposed to be done by a Small Business either at the prime level or at the first tier subcontract level. All Offerors are required to respond to the Commitment to the Small Business Program.

Acquisition Schedule (Tentative)

Synopsis Issued November 8, 2016 Virtual Pre-Solicitation (ARC) November 28, 2016 Questions/Comments Due December 9, 2016 Issuance of Draft RFP January 2017 Pre-Proposal/Industry Day January/February 2017 Issuance of RFP (45 days) March 2017 Receipt of Proposals April 2017 Contract Award (Phase-In) January 2018

NASA Advanced Computing Services (NACS)

Site Overviews

High-End Computing Capability (HECC) Project NASA Advanced Supercomputing (NAS)

NASA Ames Dr. Piyush Mehrotra

William Thigpen https://www.nas.nasa.gov/hecc/ https://www.nas.nasa.gov/

NASA Center for Climate Simulation (NCCS) Goddard Space Flight Center (GSFC)

Dr. Phil Webster Dr. Daniel Duffy http://www.nccs.nasa.gov/

HEC Program Office NASA Headquarters

Dr. Tsengdar Lee Scientific Computing Portfolio Manager http://www.hec.nasa.gov/

NASA High-End Computing Program

STATEMENT OF WORK (SOW)/

TECHNICAL OVERVIEW

NACS Technical Areas

1. Program Management

2. Technical Integration

3. High Performance Computing Environment

4. Networking and Communications

5. User Services

6. Facility Operations

7. Information Technology Security

8. Application Services and Tools

9. Visualization and Data Analysis

10. Modeling and Simulation

11. Data Publication and Distribution

12. Big Data Analysis and Analytics Support

13. Enhanced Collaborations and Visiting Scholars Program

14. Advanced Computing Research, Development and Enhancement

15. Other Advanced Computing System Support

Summary of Technical Areas (1)

1. Program Management: managing the contractor’s support for all administrative and technical activities, providing an interface between contractor management and the Government, and drawing on corporate knowledge and capabilities to support development of innovative solutions for the Government.

2. Technical Integration: conducts or manages any high-level and crosscutting technical activities such as the development and implementation of advanced computing best practices that are designed and optimized for NASA and the individual center environments.

3. High Performance Computing Environment: Provide acquisition support and state-of-the-art management of robust HPC and data environments to support NASA mission requirements for high-fidelity computational modeling, simulation, and data analysis.

Summary of Technical Areas (2)

4. Networking and Communications: network development, engineering, integration, testing, deployment, operations, and performance studies in support of high-bandwidth and specialized HPC and other advanced computing requirements for the NACS facilities.

5. User Services: maintain highly effective and efficient information exchange between users, NACS facilities, and Mission Directorate sponsors about all aspects of the systems and services through a variety of outlets.

• NAS: Includes two 24x7 functions: 1) control room operation, to provide operations and monitoring of the NAS supercomputers, their support systems, and the facility; 2) to provide the initial and primary contact with users of the NAS facility.

• NCCS: Requirement for help desk to be the initial and primary contact with the users of the NCCS facility and to be staffed from 8:00 a.m. to 6:00 p.m. Eastern Time (U.S.)

Monday through Friday.

Summary of Technical Areas (3)

6. Facility Operations: support for physical plant operations and maintenance (O&M), property control, computer system administration for facility staff, website maintenance, and facility multimedia and communications.

7. Information Technology Security: provides comprehensive IT security management, operation, monitoring and safeguards to protect All NACS information systems and IT assets.

8. Application Services and Tools: focuses on enhancing the performance and productivity of NASA’s advanced modeling, simulation, and data analysis applications.

9. Visualization and Data Analysis: develops and employs advanced visualization and data analysis in both interactive and batch environments to enable users to derive increased value from their computations.

Summary of Technical Areas (4)

10. Modeling and Simulation: conducts advanced modeling and simulation analyses requested by NASA programs and projects.

11. Data Publication and Distribution: develops and maintains data publication and distribution services to enable the NACS user community to quickly be able to publish and distribute key science and engineering research products.

12. Big Data Analysis and Analytics Support: advance NACS’ capabilities to support big data analysis and analytics development of methods and tools, continue and enhance collaboration with other NASA organizations involved with big data initiatives, and seek opportunities for outreach to discover new users and create awareness of NACS’ capabilities to support big data analysis and analytics on NASA data.

NASA Ames Research Center NASA Advanced Supercomputing

An Overview

NAS Division Organization Division Chief /

Deputy Division Chief Piyush Mehrotra /

John Parks (Acting)

Computational Aerosciences Branch

Chief Cetin Kiris

Advanced Computing Branch Chief

William Thigpen

Computational Physics Branch Chief

Seokkwan Yoon (Acting)

Assistant Division Chief of Operations Ana Grady-Hiser

NAS Project Portfolio

• Engineering Risk Assessment (ERA) - NAS experts in risk modeling and simulation support the ERA project, which provides advanced, physics-based risk analyses to support the development of safe, reliable space launch vehicles and exploration systems.

• Heliophysics Modeling & Simulation (HMS) - The Heliophysics Modeling and Simulation (HMS) team develops high-fidelity modeling and simulations tools for research on, and modeling of, the solar interior and atmosphere.

• NASA Earth Exchange - NAS is a partner in the NASA Earth Exchange project, a collaborative environment that utilizes the Pleiades supercomputer along with a NAS-operated, custom collaboration environment for Earth scientists to gain and share new insights into how the Earth's ecosystems interact and respond to climate variability and change. Current projects include an important study to understand the carbon balance of the U.S. and the North American continent.

• Quantum Artificial Intelligence Lab (QuAIL) - The NAS facility hosts the Quantum Artificial Intelligence Lab, a collaboration among NASA, University Science Research Associates, and Google to explore the potential for quantum computers to tackle computational challenges that are too difficult or impossible for traditional supercomputers to handle.

• Big Data at NAS – NASA’s data from in place assets and simulation results provide challenges that were not seen in the past. The NAS division is working at providing viable solutions to these problems.

• Computational Aerosciences - To develop, enhance, and apply predictive capabilities in computational aerosciences, and perform large-scale simulations that advance the goals of the NASA Human Exploration and Operations and Aeronautics Research and Directorates

• Computational Physics - To develop multi-disciplinary physical models, numerical methods and algorithms, and technologies in support of NASA Science and Space Technology Mission Directorates.

• High-End Computing Capability (HECC) Project – HECC provides the high-end computing resources and expert services to NASA scientists and engineers across all of the Agency's mission organizations.

Engineering Risk Assessment (ERA)

• Provides advanced risk analyses to support the development of complex space launch and exploration systems that are typically difficult to represent effectively using traditional risk assessment methods.

• Quantifies system risks through a combination of probabilistic analyses, physics-based simulations of key risk factors, and failure timing and propagation models

• Develops dynamic, integrated risk models to not only quantify the probabilities of individual failures, but also to learn about the specific systems, identify the driving risk factors, and guide designers toward the most effective strategies for reducing risk.

Simulation of blast wave passing over the Orion crew module during an abort from a vehicle explosion.

ERA Work and Benefits

• First developed and applied through NASA's

Simulation Assisted Risk Assessment (SARA) Project, which generated detailed crew-risk probability estimates for the Ares I Crew Launch Vehicle.

• Performed risk assessments for other potential launch vehicle designs, exploration systems, and missions.

• Contributes to NASA's broader safety policies and procedures by supporting key safety studies, helping to develop effective safety/risk guidelines for future space programs.

• ERA has exceptional potential to benefit the design and evaluation of a wide range of systems and technologies. Its dynamic, multi-fidelity approach provides more realistic representation of failures and their propagation, identifies sensitivities and dependencies within highly complex systems, and most importantly, facilitates risk-informed design throughout every stage of the development process.

Simulation of debris fragmentation resulting from a vehicle explosion to determine probability of debris striking the aborting crew module.

Heliophysics Modeling & Simulation Project

• Develops high-fidelity modeling and simulations tools for research on, and modeling of, the solar interior and atmosphere. The project supports NASA's Living With a Star (LWS) program, which has a goal to provide a predictive understanding of the Sun's system, specifically of the space weather conditions near Earth and in the interplanetary medium.

• Addresses the LWS program's primary goal to develop first-principles-based models for the coupled Sun- Earth system by providing tools to U.S. space weather forecasting organizations, such as the National Oceanic and Atmospheric Administration.

• Take a unique, multidisciplinary approach to deliver real-time models to the scientific and operational communities for forecasting both magnetic fields and velocity fields on the surface of the Sun.

Coronal mass ejection observed by NASA's Solar Dynamics Observatory (SDO), in extreme ultraviolet radiation emitted by ionized helium atoms heated to 80,000 Kelvin. The eruption is caused by a magnetic field that was generated by a dynamo process beneath the visible surface of the Sun. (NASA/SDO)

NASA Earth Exchange (NEX) A collaborative environment that brings scientists and researchers together in a knowledge-based social network along with tools, computing power and data to accelerate research, innovation and provide transparency.

VISION

To provide “science as a service” to the Earth Science community addressing global environmental challenges.

GOAL

To improve efficiency and expand the scope of NASA Earth science technology, research and applications programs.

NEX

Project Manager: Piyush Mehrotra Principal Scientist: Ramakrishna Nemani

NEX Overview

NEX Infrastructure

NEX Software Platform

NEX Science and Applications

Data repository – 1.3PB+ Compute Infrastructure (HPC, GPUs, Shared Memory, Quantum)

Collaboration Portal and Knowledge Base

Science Data Management

Workflow and Process

Management

Outreach and Engagement

Knowledge Management

Cloud Infrastructure (OpenNEX)

Global Science Products Carbon Monitoring Water Management Climate Landcover (1m)

Quantum Computing: D-Wave Two™ System

• Collaboration between NASA / Google /

USRA

• D-Wave 2 Installed at NAS

• Washington processor – 1,097 qubits

(quantum bits – niobium superconducting loops encoding 2 magnetic states)

• Physical characteristic » 10 kg of metal in vacuum at 15 mK » Magnetic shielding to 1 nanoTesla

(50,000x less than Earth’s magnetic field) » Uses 12 kW electrical power

• Focused on solving discrete optimization problems via quantum annealing

Big Data Challenges for NASA Users

• Data Discovery – finding what data is available and where

• Indexing, federated metadata service and semantic reasoning

• Increased WAN bandwidth

• Fault tolerant and resilient hardware/software infrastructure

– Tools/models/algorithms - developing analytics/analysis software at scale

• Mechanisms for sharing software to reduce duplication

• Software for workflow description and management to tie all components together and facilitate re-use

• Analysis/Analy:cs infrastructure – inadequacy of available resources

• I/O infrastructure

• Large memory spaces for in-core analysis

• Support for the heterogeneous resources in an integrated environment:

distributed memory & shared memory systems, hadoop cluster, accelerators, FPGAs etc.

• Data Dissemina:on– difficult to share knowledge across a wider community

• Support for dissemination and sharing of code, data products, results, etc…..

Based on a HECC survey: NAS Technical Report: NAS-2014-02.pdf

Big Data Related Projects @ NAS

Applica'ons

Analy'cs

Hardware Infrastructure

! Xeon PHI, GPU, FPGA, Hadoop

Computational Aerosciences

• HEOMD (SLS, MPCV & Commercial Crew): To develop and apply aeroscience modeling capabilities for space vehicle design and analysis, including databases and induced environments for ascent and abort, staging, debris, plume impingement, retro-propulsion, and launch environment. Develop and apply physics based Engineering Risk Assessment (ERA) models for space vehicles.

• ARMD (Fundamental Aeronautics): To advance fundamental aeroscience theory, models, methodologies, codes, and understanding; develop and apply aeroscience modeling capabilities for air vehicle design and performance, efficiency, emissions, noise, and safety analysis.

• The computational aerosciences teams support many NASA and commercial aeronautics and aerospace missions and projects, including:

– Space Launch System, Aquila-II, SWORDS, and BioSentinel

– Commercial Crew Program, Sierra Nevada Corporation, and

SpaceX

– Planetary Defense Project

– Office of Safety and Mission Assurance

– Fundamental Aeronautics Program (AATT, CST, Rotary Wing, ERA)

Gulfstream Landing Gear Noise Prediction

Ascent Aero-database for Space Launch System

Prediction Noise Generation from High-Lift Devices

Shock-Plume Interaction Wind Tunnel Support for CST

Computational Physics

• Solar Stellar Modeling & Analysis: To understand the variability of the

Sun and its influence on space weather and the solar planetary system, and to understand the variability of stars and their effects on the habitability of accompanying planets.

- Customers Heliophysics and Astrophysics (SMD)

• Planetary Atmosphere Simulation: To drive advances in thermophysical chemistry of planetary atmospheres and Earth’s ionosphere, relevant to entry systems and space-weather effects.

- Customers: Game Changing Program - Entry Systems Modeling (STMD)

- Material Response to High-Enthalpy Environments: To develop understanding of material response to high-enthalpy environments at the micro-scale, and develop physics-base material response models for designing thermal protection systems.

- Customers: Game Changing Program - Entry Systems Modeling (STMD)

• Planetary Defense: To develop advanced risk models and simulations to assess and characterize the potential threat due to impacts from various classes of asteroids and other near-Earth objects

- Customers: ARC Planetary Defense Project for the NASA Near-Earth

Object Officeon the surface of the Sun.

Satellite optical communication and CubeSat instruments

Preliminary Cart3D Simulations of Chelyabinsk Meteor

Chemistry products for ionosphere and planetary entry simulations

High End Computing Capability

Project

NASA’s HEC Requirements: Capacity

• HEOMD (engineering-related work) require HEC resources that can handle large numbers of relatively-low CPU-count jobs with quick turnaround times.

Over 1500 simulations utilized ~ 2 million processor hours to study launch abort systems on the next generation crew transport vehicle

Over 4 million hours were used over a 4 month project to evaluate future designed of the next generation launch complex at the Kennedy Space Center

The formation of vortex filaments and their roll-up into a single, prominent vortex at each tip on a Gulfstream aircraft

NASA’s HEC Requirements: Capability

• ARMD and SMD (aeronautics and science related work) require HEC resources that can handle high fidelity relatively-large CPU-count jobs with minimal time-to-solution. Capability enables work that wasn’t possible on previous architectures.

NASA is looking at the oceans, running 100’s of jobs on Pleiades using up to 10,000 processors. Looking at the role of the oceans in the global carbon cycle is enabled by access to large processing and storage assets

For the first time, the Figure-of-Merit has been predicted within experimental error for the V22 Osprey and Black Hawk helicopter rotors in hover, over a wide range of flow conditions

To complete the Bolshoi simulation, which traces how the largest galaxies and galaxy structures in the universe were formed billions of years ago, astrophysicists ran their code for 18 straight days, consuming millions of hours of computer time, and generating massive amounts of data

KEPLER

NASA’s HEC Requirements: Time Critical

• NASA also has need for HEC resources that can handle time-sensitive mission-critical applications on demand (maintain readiness)

ReEntry

Storm Prediction

KEPLER

UAVSAR produces polarimetric (PolSAR) and interferometric (repeat-pass InSAR) data that highlight different features and show changes in the Earth over time

HECC enables the enormous planetary transit searches to be completed in less than a day, as opposed to more than a month on the Kepler SOC systems, with significantly improved accuracy and effectiveness of the software pipeline

HECC Assets

HECC Traditional Computer Floors

PDU

ISU 6

0 2 4 6 8 10 Scale in Feet

Visualization Lab Building 258

AMES RESEARCH CENTER

MOFFETT FIELD, CA 94035

N258 COMPUTER ROOM 125

John Parks Facilities

Chris Henze Visualization hyperwall (128-screen display) hyperwall Visualization

Systems

Updated

June 13, 2014 Updated By

Chris Buchanan

PDU

N258-Rm 125

Current Diagram

Sto rag e

DDN

RAID

SG

I

D R ack

SG

I

D R ack

SG

I

D R ack

SG

I

D R ack

SG

I D

R ack

SG

I

D R ack

SG

I

D R ack

SG

I

D R ack

PDU B

75 KVA

PDU D

75 KVA

AIR

Phone Sw.

AMES RESEARCH CENTER

MOFFETT FIELD, CA 94035

N233A COMPUTER ROOM 189/190

Updated On Updated By

September 26, 2013 Chris Buchanan

0 2 4 6 8 10 Scale in Feet

NAS HECC

Building 233A Computer Floor Diagram

B D FA C E G

1 8 1 8 1 8 1 8 1 8 1 82 3 4 6 75tile#

H I J 1 8 1 8 1 71 8

K

L

M

N

B D FA C E G

1 8 1 8 1 8 1 8 1 8 1 82 3 4 6 75

L

K

M

N

PDU D

125 KVA

C om m sgi

ALTIX

C

CF

E2

PB

S2

PDU 4

125 KVA

C

D

C

E

M D

S

189190

1 8 1 8 1 8 1 7 H I J

C

B stkesssec net

C

A net

C

F net

Networks Systems Proj. Mgr Facilities

D D

N

SpectraLogic

SpectraLogic

SpectraLogic

125 KVA

G

N233A-Rm 189/190

Current Diagram

CE

T

L

AB

(N

IC

S) console

D D

N1

(IS

K)

W es tm er e W at er

W es tm er e W at er

W es tm er e Ai r

PDU 8

125 KVA

PDU 7

125 KVA

PDU 3

125 KVA

PDU 9

125 KVA

Ne ha le m

A irNe ha le m A ir

PDU 5

125 KVA

D D

N5

(IS

K)

D D

N6

(IS

K)

Ne h

Ai r

N258/230 16,800

N233A/190 6,900 N233A/189

2,700 N258/125 1,275

N258/131 2,080

HECC Modular Computer Floors

R&D 088 16,800

AMES RESEARCH CENTER

MOFFETT FIELD, CA 94035

N258 COMPUTER ROOM 230

Updated Updated By

November 7, 2016 Chris Tanner

0 2 4 6 8 10 Scale in Feet

B D F HA C E G

B D F HA C E G

R

S

T

U

V

W

X

NAS HPC Facility Modular Supercomputing Facility Floor Diagram

R

S

V

W

X

T

U

I J K

MSF Diagram

Project Manager FacilitiesNetworksCSS

Notes:

1. xx.

101001002003004005006007008 Broadwell

Bl an k

016 015 014 013 012 011 010 009 102

Bl an k

Broadwell

W at er Co nt ro l W at er

Co nt ro l

El ec tr ic al P an el

Fan Wall Evaporative Media & Filter Wall

Fan Wall Evaporative Media & Filter Wall

User Interface

2800kVA Transformer 13.8KV/415V Switchgear & 5 kVA

Transformer 415V/120V

Pleiades Specifics

• 161 SGI Racks (7.58 PF; 936 TB; 32,308 SBUs/hr) 158 SGI Altix ICE X Racks (7.21 PF; 931 TB; 32,134 SBUs/hr)

• 26 racks of ICE-X with Intel Xeon processor E5-2670 (Sandy Bridge):

623 TF; 59.9 TB; 3,407 SBUs/hr

• 75 racks of ICE-X with Intel Xeon processor E5-2680v2 (Ivy Bridge):

2.419 PF; 345.6 TB; 13,608 SBUs/hr

• 29 racks of ICE-X with Intel Xeon processor E5-2680v3 (Haswell):

2.004 PF; 267.3 TB; 6,974 SBUs/hr

• 28 racks of ICE-X with Intel Xeon processor E5-2680v4 (Broadwell):

2.167 PF; 129.0 TB; 8,145 SBUs/hr

3 SGI Coyote Racks (371 TF; 5 TB; 175 SBUs) (note, accelerators are not counted in SBU numbers)

• 2 racks of Intel Xeon processor E5-2670 (Sandy Bridge) and Nvidia K40 graphic processors: 296 TF; 4 TB; 117 SBUs

• 1 rack of Intel Xeon processor E5-2670 (Sandy Bridge) and Intel Xeon Phi 5110P accelerator processor: 75 TF; 1 TB; 58 SBUs

Cores

• 22,944 Intel Xeon processors (246,048 cores)

• 64 Nvidia GPUs (184,320 cores)

• 64 Intel Xeon Phi processors (3,804 cores)

Nodes

• 11,376 nodes (dual-socket blades)

• 64 nodes (dual-socket + GPU)

• 32 nodes (dual-socket + dual-Phi)

• 14 Login nodes

Networks

• Internode: Dual-plane partial 11D hypercube (FDR)

• Gigabit Ethernet Management Network

Electra Specifics

• 16 SGI Racks (1.24 PF; 147 TB; 4,654 SBUs/hr)

• 16 racks of ICE-X with Intel Xeon processor E5-2680v4 (Broadwell): 1.24 PF; 147 TB; 4,654 SBUs/hr

Cores

• 2,304 Intel Xeon processors (32,256 cores)

Nodes

• 1,152 nodes (dual-socket blades)

Networks

• Internode: Dual-plane fully-populated 7D hypercube

(FDR)

• Gigabit Ethernet Management Network

• Metro-X IB extenders for shared storage access

Merope Specifics

• 36 SGI Altix ICE X ½ Racks (162 TF; 28 TB; 1,152 SBUs)

• 36 ½-racks of 8400EX with Intel Xeon processor E5670

(Westmere): 162 TF; 28 TB; 1,152 SBUs 2,304 Intel Xeon processors (13,824 cores)

• 2,304 six-core Westmere

• 2.93 GHz processors (13,824 cores)

Endeavour Specifics

• 32 TF constellation-class supercluster

• 2 SGI Ultra Violet 2 nodes with Intel Xeon E5-4650L

2.6 GHz processors

• One 512-core node with 2 TB globally addressable RAM (Endeavour1)

• One 1,024-core node with 4 TB globally addressable RAM (Endeavour2)

• Interconnect

• Intranode: NUMALink-6 (enable large SSI)

• Dual-Plane QDR InfiniBand connectivity into Pleiades infrastructure » 1 connection from each node into IB0 for TCP traffic (pbs, login, …) » IB1 is for I/O traffic to the Lustre file systems.

Endeavour1 has 3 connections and Endeavour2 has 4 connections.

• 10 Gb Ethernet can be used for WAN traffic

Advanced Visualization: hyperwall and CV

• Supercomputing-scale visualization system to handle massive size of simulation results and increasing complexity of data analysis

• 8x16 LCD tiled panel display (23 feet x 10 feet)

• 245 million pixels

• Debuted as #1 resolution system in the world

• In-depth data analysis and software

• Two primary modes

• Single large high definition image

• Sets of related images (e.g. parameter study)

• High-bandwidth to HEC resources

• Concurrent Visualization: Runtime data streaming allows visualization of every simulation time step - ultimate insight into simulation code without increase in traditional disk I/O

• Traditional Post-Processing: Direct read/write access to Pleiades filesystems eliminates nee for copying large datasets

• GPU-based computational acceleration R&D for appropriate NASA codes

Storage and Archive

• Lustre File Systems (39.6 PB in 7 file systems)

• DDN

» 14 DDN RAID Systems, 9.9 PB total, 3 facility-wide file systems

• NetApp » 62 RAID Systems, 29.7 PB total, 4 facility-wide file systems

• NFS File Systems

• 3 home file systems 3.7 TB total

• 2 facility-wide scratch file systems 59 TB & 1 PB

• .4 PB for NEX

• Archive System

• 490 PB Maximum Capacity

• 6 T950 Spectra Logic Libraries

HECC Growth

HECC Growth

Largest HECC LINPACK Result

HECC

HECC Conducts Work in Four Major Technical Areas

Supercomputing Systems

Data Analysis and Visualization

Application Performance and User Productivity

Networking

Provide computational power, mass storage, and user-friendly runtime environment through continuous development of management tools, IT security, systems engineering

Facilitate advances in science and engineering for NASA programs by enhancing user productivity and code performance of high-end computing applications of interest

Create functional data analysis and visualization software to enhance engineering decision support and scientific discovery by incorporating advanced visualization technologies

Provide end-to-end high-performance networking analysis and support to meet massive modeling and simulation distribution and access requirements of geographically dispersed users

Supporting Tasks Facility, Plant Engineering, and Operations: Necessary engineering and facility support to ensure the safety of HECC assets and staff Information Technology Security: Provide management, operation, monitoring, and safeguards to protect information and IT assets User Services: Account management and reporting, system monitoring and operations, first-tier 24x7 support Internal Operations: NASA Division activities hat support and enhance the HECC Project areas

Resource Utilization

User Location

CT – 6

DC – 6

MA – 56

MD -- 166

NH – 5

NJ – 16

RI – 2

VT – 3

Offshore – 71

United States of America

80°W 75°W50°N 70°W

25°N

30°N

35°N

40°N

45°N

85°W90°W95°W100°W105°W110°W115°W120°W125°W

Tropic of Cancer

Lake Superior

La ke

M ic hi ga n

Lake Huron

Lake Erie

Lake Ontario

ATLANTIC

OCEAN

Gulf of Mexico

PACIFIC

OCEAN

W

E

N

S

20°N

22°N

160°W 156°W 60°N

70°N

50°N 160°W 140°W

Bering Sea

Gulf of Alaska

250 500

250 500

0mi

0km

100 200

100 200

0mi

0km

National boundary State boundary

LEGEND

125 250

125 250

0mi

0km

1 162

Quarterly Utilization Over 10+ Years

5,000,000

10,000,000

15,000,000

20,000,000

25,000,000

30,000,000

35,000,000

40,000,000

45,000,000

50,000,000

55,000,000

60,000,000

65,000,000

Q3 F Y20

Q4 F

Q1 F

Q2 F

Q3 F

St an da rd

B ill in g

U ni ts

SOMD

ESMD

NAS

NLCS

NESC

SMD

HEOMD

ARMD

Alloc. to Orgs

75% of Peak Capacity

Aeronautics Support (55,958,567 SBUs) Advanced Air Vehicles # of projects: 142 # of SBUs used*: 16,966,350

✦ HECC is used to develop concepts and technologies for dramatic improvements in the noise, emissions, and performance of transport aircraft.

✦ HECC is used to develop concepts and technologies to increase rotorcraft speed, range and payload, and decrease noise, vibration and emissions.

✦ HECC is used to develop advanced computer-based prediction methods for supersonic aircraft shape and performance and to develop technologies that will help eliminate today's technical barriers (such as sonic booms) to practical, commercial supersonic flight.

✦ HECC is used to develop computer-based tools and models and scientific knowledge that will lead to significant advances in our ability to understand and predict flight performance for a wide variety of air vehicles.

Transformative Aeronautics Concepts # of projects: 45 # of SBUs used*: 26,195,249

✦ HECC is used to develop and utilize Reynolds-averaged Navier-Stokes (RANS) and Large Eddy Simulation (LES) methods, and hybrid RANS-LES techniques to improve calculation methods for propulsion flows dominated by turbulent boundary layers and mixing.

✦ HECC is used to assess natural laminar flow concepts, to elucidate the physics and control of boundary layer transition in swept wing flows and drag reduction concepts for compressible boundary layers.

✦ HECC is used to validate chemistry, chemistry-turbulence and spray models being developed under the National Jet Fuels Combustion program.

Airspace Operations and Safety & Other # of projects: 8 # of SBUs used*: 506,017

� HECC is used for developing reliable computational tools for predicting and analyzing stability & control characteristics of aircraft prior to or while encountering loss-of-control flight conditions characterized by abnormal flight (e.g., stall), abnormal vehicle conditions (e.g., damage, jammed control surfaces), external upsets (e.g., wake vortex, wind shear, gusts), and icing.

� HECC is used to develop methods for computing aerodynamic performance degradation associated with ice accretions on swept wing geometries.

� HECC is used to produce real-time icing impact fields for flight planning and post mission analysis.

Integrated Aviation Systems # of projects: 10 # of SBUs used*: 12,290,951

� HECC is used for accurate prediction of airframe noise from a full scale aircraft and evaluation of flap and landing gear noise reduction concepts in flight environments.

� HECC is used to develop technology for compact, high-power-density electric motors to power an all-electric general-aviation aircraft or helicopter, a hybrid turbine-electric regional airliner or a large transport with many small engines distributed around the aircraft.

� HECC used for parametric studies conducted to optimize size, shape and placement of an array of fluidic actuators for maximizing the lift for control surfaces on an aircraft, which would help reduce the size of control surface and the weight of an aircraft.

*October 1, 2015 to September 30, 2016

Aviation Safety Program The Aviation Safety Program (AvSP), part of NASA’s Aeronautics Research Mission Directorate, helps to develop new ways to achieve exceptional levels of safety for air travel despite increasingly crowded skies and congested airports.

Over the past decade, collaboration between in-dustry and government to proactively identify new risks has led to historically low rates of commercial accidents. But as air traffic volume increases, the vigilance of the aviation community must continue.

That’s why, working with partners from academia and in the public and private sectors, AvSP con-ducts foundational research and develops new technologies to overcome the emerging challeng-es created by the nation’s transition to the Next Generation Air Transportation System (NextGen).

To help provide solutions, the program inves-tigates improvements to increase the inherent safety of aircraft systems and structures, ways to avoid atmospheric hazards, and development of next-generation concepts for on-board and on-ground safety systems. AvSP studies:

systems;

eliminate any potential issues;

systems to improve overall performance;

issues from ever occurring; and related systems.

Images (Clockwise, from top-left) Data Mining: -ing safety issues are already proving useful to commercial airlines. Flight Safety: Subscale models are put through loss-of-control scenarios in NASA wind tunnels to test new recovery techniques. Engine Icing Prevention: Studies are being done into the types of atmospheric conditions that can form ice particles inside engines, leading to power loss. Human-Friendly Flight Decks: NASA uses

Human Exploration and Operations & Safety Support (51,658,239 SBUs)

Surface pressure coefficient with sonic iso-surfaces, from an 11,640-processor computation of the transonic flow about a Space Launch System configuration. The time-accurate computation is being performed to predict unsteady surface pressures and ultimately the development of buffet loads.

� HECC is used to simulate the effect of larger solid rocket boosters and new propulsion systems on the launch facility at Kennedy Space Center, such as investigating whether ignition overpressure waves generated during liftoff are fully suppressed by the existing water suppression system.

� HECC is used to evaluate visiting-vehicle induced loads on the International Space Station (ISS) during mated and rendezvous operations and to evaluate crew Extra-vehicular Activity/Intra-vehicular Activity and attitude control loads on ISS.

� HECC is used in developing a combustion response model to investigate combustion instability in hydrocarbon-fueled rocket engines.

� HECC is used for technology development for entry, descent and landing systems.

*October 1, 2015 to September 30, 2016

Multi-Purpose Crew Vehicle # of projects: 7 # of SBUs used*: 1,636,080

✦ HECC is used to support the creation of hundreds of computational solutions that model the flow field around the Crew Module and Launch Abort System for all flight regimes to be used as input for the aerodynamic databases.

� HECC is used to run computational fluid dynamics simulations to study the aerodynamic and aerothermal environments for the Multi-Purpose Crew Vehicle.

� HECC is used to develop and deploy a prototype system for rapid aerodynamic performance database generation and to use it on real-world problems faced by the Human Exploration and Operations mission directorate.

Space Launch Systems # of projects: 13 # of SBUs used*: 31,423,169

� HECC is used for computational fluid dynamics simulations of Space Launch Systems ascent to assess aerodynamic performance, protuberances, stage separation, and plume effects (such as plume-induced flow separation) for evolving vehicle designs.

� HECC is used for computational fluid dynamics analysis of Advanced Booster development efforts in the combustion stability areas.

� HECC is used for prediction of the launch induced environment for the Space Launch System including liftoff acoustics, ignition over-pressure, separation environments, debris, Launch Pad Abort Environments and hydrogen entrapment.

� HECC is used to simulate tanks and main propulsion system components (including feedlines, valves, manifolds, ducts, and pogo accumulators) for evaluation of criteria such as flow uniformity and component pressure drop.

HEOMD - Space Flight Operation & General # of projects: 32 # of SBUs used*: 11,042,783

NASA Engineering & Safety Center # of projects: 8 # of SBUs used*: 7,556,207

� HECC is used for simulations to provide guidance to the Space Launch System advanced booster designers by providing aerodynamic loading implications for various potential advanced booster geometric configurations.

� HECC is used to improve the capability to predict combustion stability in liquid rocket engines to increase NASA engineers’ capability to more confidently and efficiently identify and mitigate combustion stability issues in engine development programs.

� HECC is used to used for studies of large eddy simulations of oblique-shock / supersonic hot jet interaction, aimed at prediction of plume-induced vibroacoustics.

Science Support (94,488,707 SBUs) Astrophysics # of projects: 100 # of SBUs used*: 37,875,367

� HECC is used by the Kepler mission to find Earth-sized planets around other stars and to fully analyze the Kepler data to find any undiscovered planets still “hiding” in the data.

✦ HECC is used to understand the physics of high redshift galaxy formation and make detailed predictions that can be used to guide NASA observations of the first galaxies.

� HECC is used for quantifying the redistribution of matter in galaxies when supernova energy is deposited; exploring the growth of black holes and the impact of active galactic nuclei on galaxy evolution; and determining whether the ultraviolet light from stars in galaxies can "escape" to re-ionize the universe.

Earth Science # of projects: 136 # of SBUs used*: 18,460,872

✦ HECC is used to combine observational data with numerical simulations of the global ocean circulation to provide vital information for understanding climate change and its impact on land and sea ice, ocean ecology, and the global carbon cycle.

✦ HECC is used for high-resolution cloud resolving model simulations to provide unique and detailed insights into the processes that form tropical clouds and cloud systems, which account for approximately two-thirds of global rainfall.

✦ HECC is used explore the feedback mechanisms between polar ice sheet and atmosphere circulation in order to determine how global temperature changes translate into increased sea level rise.

✦ HECC is used to improve the understanding of the current balance of carbon in the Arctic and to provide a framework for early detection of future carbon destabilization.

Heliophysics # of projects: 100 # of SBUs used*: 19,458,069

✦ HECC is used for modeling solar magneto-convection in order to understand how magnetic fields emerge through the sun’s surface, heat the sun’s outer atmosphere, and produce sunspots, spicules, and flares.

✦ HECC is used for realistic multi-scale simulations to understand the complicated physics of the turbulent convection zone and atmosphere of the sun and for analyzing and interpreting observations from the NASA space missions.

✦ HECC is used to simulate small-scale magnetic fields generated by turbulent dynamo action just beneath the solar surface in order to accurately predict space weather events that impact the Earth environment.

Planetary Science # of projects: 89 # of SBUs used*: 18,694,399

� HECC is used to decipher the structure of the lunar interior to understand the origin and thermal evolution of the moon and to extend this knowledge to other bodies in the inner solar system.

� HECC is used to model the origin and evolution of Kuiper belt objects to determine how their properties constrain our current models of planet formation.

� HECC is used perform modeling and simulation of asteroid entry, breakup, airburst, blast propagation, and tsunamis to assess the risks that potentially hazardous asteroids could pose to populations and infrastructure in the event of an Earth strike.

*October 1, 2015 to September 30, 2016

NASA's High-End Computing Capability Project - It's Electra-fying 56

Return on Investment

Delivered Costs per

SBU

$21.50

$12.41

$8.25

$2.80

$1.41

$1.07

$0.73

$0.50

$0.37

$0.25 $0.26

D el iv er ed S ta nd ar d

B ill in g

U ni ts (S

B U

(30,000,000)

20,000,000

70,000,000

120,000,000

170,000,000

220,000,000

FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16

NASA Goddard Space Flight Center NASA Center for Climate Simulation

(NCCS)

An Overview

Sciences and Exploration Directorate Code 600

Computational and Information Sciences and Technology Office

(CISTO) Code 606 Chief: Dr. Phil Webster

Deputy Chiefs: Nancy A. Laubenthal and Bob Pierce

Networks and Information Technology Security Group

Code 606.1 Lead: Bill Fink

NASA Center for Climate Simulation (NCCS)

Code 606.2 Lead: Dr. Daniel Duffy

Information Science and Technology Research Group

Code 606.3 Lead: Dr. Patrick Coronado

Scientific Visualization Studio (SVS) Code 606.4

Lead: Dr. Horace Mitchell

Climate Model Data Services

Lead: Schnase/Duffy

Climate Informatics Lead: Dr. John Schnase

Advanced Software Technology Group Lead: Dr. Daniel

Duffy

GMAO

ASTG Advisor Dr. Tom Clune

Code 606 Organization Chart

NASA Center for Climate Simulation (NCCS)

Code 606.2 Lead: Dr. Daniel Duffy

Advanced Technology CS Lead: Dr. Daniel Duffy

Operations CS Lead: Tom Schardt

Facilities Lead(s): Dr. Daniel Duffy

User Support CS Lead: Ellen Salmon

Networks and Security CS Lead George Rumney

Discover Cluster CS Lead: Bruce Pfaff

Mass Storage CS Lead(s): Tom Schardt and

Adina Tarshish

NCCS Organizational Chart

• Provides an integrated high-end computing environment designed to support the specialized requirements of Climate and Weather modeling.

• High-performance computing, data storage, and networking technologies

• High-speed access to petabytes of Earth Science data

• Collaborative data sharing and publication services

• Advanced Data Analytics Platform (ADAPT)

• Primary Customers (NASA Climate Science)

• Global Modeling and Assimilation Office (GMAO)

• Land Information Systems (LIS)

• Goddard Institute for Space Studies (GISS)

• Variety of other Research and Development (R&D)

• High-Performance Science

• http://www.nccs.nasa.gov

• Code 606.2

• Located at NASA Goddard Space Flight Center in Greenbelt, MD.

NASA Center for Climate Simulation (NCCS)

Data Centric HPC, Big Data and IT Environment

DATA

Storage &

Management Global file system enables data access for full range of modeling and analysis activities

• Large scale HPC computing

• Comprehensive toolsets for job scheduling and system monitoring

• Large capacity storage

• Tools to manage and protect data

• Data migration support

• Help Desk

• Account/Allocation support

• Computational science support

• User teleconferences

• Training & tutorials

• Interactive analysis environment

• Software tools for image display

• Easy access to data archive

• Specialized visualization support

• Internal high speed interconnects for HPC components

• High-bandwidth to data center users

• Multi-gigabit network supports on-demand data transfers

HPC Computing Mass Storge and Stewardship

• Code repository for collaboration

• Environment for code development and test

• Code porting and optimization support

• Web based tools

Applications Support

Analysis & Visualization User Services

• Capability to share data & results

• Supports community-based development

• Data distribution and publishing

• Exposing analytics through services

Data and Analytics Services

Data Transfer

Security

Goddard Earth Observing System (GEOS) Model NASA Global Modeling and Assimilation Office (GMAO)

• FV3 Dynamical Core uses a Cubed-Sphere which maps the Earth onto faces of a cube

• There are 6 faces of the cube and multiple vertical layers

• Total number of grid points

• X * Y * Z * 6 Faces of the Cube

• Current GMAO Research

• Operational research forecasts are running at 27 KM resolution using about 27 million grid points

• Target operational research forecasts at a resolution of 12 KM in the very near future

• Reanalysis (including chemistry)

• Dynamic downscaling of reanalysis and forecasts down to 6 KM

• Highest resolution research runs are at 1.5 KM global resolution

Dynamic Downscaling Assessment

Midcontinent Summertime MCSs

• Warm / Dry Climate Model Biases

• Extreme weather events

West Coast Wintertime Atmospheric Rivers (ARs)

• Crucial for water resources/availability

• Associated with most flooding events

Northeast Wintertime Storms (NESs)

• Extreme precipitation/snowfall events

• Extreme wind events

Study in 2016 with a narrow scope – Focus only on 3 Impactful Phenomena

NASA Downscaling Models

• Regional Climate Model

• NASA Unified-WRF (NU-WRF) Based on WRF-ARW v3.5.1

• Initial/Boundary Conditions: MERRA-2 six-hourly re-analyses over CONUS

• Land Initial Conditions

• Land Information System (LIS) 10-yr spin-up of Noah

LSM

• Nudging (large scale forcing of certain variables to the synoptic scale)

• Simulations with and without spectral nudging of p, t, and horizontal winds above the PB

NASA Downscaling Models Continued

• MERRA-2 Replay at 12km – M2R12K

• Goddard Earth Observing System Model, Version 5 (GEOS-5) in replay mode at 12km resolution (global simulation)

• Boundary Conditions

• MERRA-2 six-hourly re-analyses

• Land Initial Conditions

• MERRA-2 CLSM

• Nudging

• Replay capability adds a forcing term to constrain the 12km run to MERRA-2 p, t, winds, and humidity (q)

• Replay allows the model to develop its own internally developed mesoscale while following the large scale trajectory of the underlying MERRA-2 reanalysis (~50km)

Goddard Institute for Space Studies (GISS)

• Part of GSFC Code 600 Sciences and Exploration Directorate

• Key Objective: Prediction of atmospheric and climate changes in the 21st century.

• Program areas at GISS may be roughly divided into the categories of climate forcings, climate model development, Earth observations, atmospheric radiation, atmospheric chemistry, climate impacts, planetary atmospheres and astrobiology, paleoclimate, and other disciplines.

• http://www.giss.nasa.gov/

Monthly temperature anomalies with base 1980-2015, superimposed on a 1980-2015 mean seasonal cycle. (Credit:

NASA/GISS/Schmidt)

A map of the October 2016 LOTI (land-ocean temperature index) anomaly, showing that the Arctic region was much warmer than average. The United States and North Africa were also relatively warm.