NACS_Pre-Proposal_Conference_Tour_at_GSFC_Q&A_Set_2_03172017.pdf

NNA17554082R

NASA Advanced Computing Services (NACS)

Pre-Proposal Conference/Site Tour

Goddard Space Flight Center

Question & Answer Set 2

March 17, 2017

Questions and answers posted after issuance of the draft RFP and prior to issuance of the final

RFP are for guidance only. The final RFP will supersede and replace the draft RFP and all related documentation, including Pres-Proposal Conference/Industry day communications and these questions and answers.

1Q. How long does the Uninterruptible Power Supply (UPS) last?

1A. The UPS in the NASA Center for Climate Simulation (NCCS) is used to ride-through short power outages and can last anywhere from 10 minutes to an hour based on the amount of equipment drawing power.

2Q. Will the Joint Center for Satellite Data Assimilation (JCSDA) system be replaced with other

IBM hardware?

2A. No, the workload has been absorbed into discover, and the hardware is being decommissioned.

3Q. Is filling entire racks due to power limitations avoided?

3A. No. The NCCS does a significant amount of work to balance the power across the phases in the rack mounted Power Distribution Units (PDUs), the room PDUs, and the UPSs.

4Q. How is heat removed from the computer room?

4A. Heat is removed in two ways. For all the compute racks, rear door heat exchangers are used to dump the heat into the facility chilled water directly at the back of the rack. In addition, traditional air handler units are used to remove heat from the room.

5Q. What is used for fire suppression?

5A. Traditional water based sprinkler systems.

Q&A Set 2

6Q. Is equipment refurbished before passing on?

6A. No. We have a standard decommissioning process that we execute prior to passing equipment to other entities. This includes setting devices to factor defaults and scrubbing hard drives.

7Q. How are backups done?

7A. For Discover and ADAPT, only the user home directories are backed up, which are relatively small. Discover also has scratch space for users, called “nobackup”, which is NOT backed up. Users are responsible for backing up their larger data sets to the NCCS mass storage or other locations.

8Q. Does the container only have OpenStack? Is OpenStack also used on the ADAPT servers located in E100?

8A. Yes, the servers in the container are only running OpenStack Currently, the system is in an operational prototype phase and will be put into operations during this year. The ADAPT portion of the science cloud located inside Building 28, room E100 were installed before OpenStack became more readily available and stable. The production servers inside the building are running a combination of KVM, Puppet, LDAP, GPFS, and custom scripts to manage that cloud environment.

9Q. What is done in the container, HPC or enterprise applications?

9A. The servers in the container will be the second version of the ADAPT environment. This will be an OpenStack cloud used for scientific, large scale data analysis. This environment is also being used to explore (in a lower risk environment) some emerging technologies, such as

Software Defined Networking. Through this effort, the NCCS is providing guidance and lessons learned to Goddard for how to create a center-wide private cloud.

10Q. Do we have network connections to other Centers?

10A. The NCCS is connected to the Science and Engineering Network (SEN) (see https://science.gsfc.nasa.gov/606.1/SEN.html for more information). The SEN connects the

NCCS to the Goddard campus area network and to external networks.

https://science.gsfc.nasa.gov/606.1/SEN.html

11Q. Why is the container used?

11A. The container originally belonged to Code 700, and they were vacating it. The NCCS asked and was granted ownership of the container to prototype the OpenStack compute cloud.

This provided the NCCS with some additional power, cooling, and floor space at minimum cost.

12Q. When did the container become NCCS?

12A. The NCCS took ownership of the container in late 2016.

13Q. Will there be HPC or more Enterprise services in the container?

13A. In the existing container, the NCCS is going to run the OpenStack part of our private

ADAPT science cloud. The existing container will not be used for High Performance Computing or more Enterprise work.

14Q. Are there plans to have other containers?

14A. Not currently, but the NCCS is in the process of performing a facility analysis to look at options and capacity planning for future systems. Containers will be a part of that analysis.

15Q. Is the Heating, Ventilation, and Air Conditioning (HVAC) is all Carrier?

15A. The only Carrier unit used is a 50-ton external chiller for the container. Inside the computer rooms in Building 28, Liebert down flow 50-ton units are used. Our onsite facilities staff perform the first level of support for these units, and the NCCS contracts with the Goddard Facility

Management Division (FMD) for maintenance on these units.

16Q. Is SGI Data Management Facility (DMF) and Lustre used?

16A. No, only the SGI DMF is used on the mass storage with SGI’s Clustered XFS (CXFS). On both Discover and ADAPT, IBM’s Spectrum Scale (formally known as GPFS) is used. Lustre is not used.

17Q. What is the NCCS relationship with DataDirect Networks (DDN)?

17A. The NCCS has been a DDN customer for over 10 years. Currently, the NCCS has several

DDN storage systems in operations. However, DDN is just one of many storage vendors whose systems are in use throughout the NCCS. The NCCS deploys storage from multiple storage vendors, including DDN, NetApp, Seagate, Raid Inc., HPE, and more.

18Q. What are the different kind of storage fabrics used?

18A. On Discover and the Mass Storage system, fibre channel is used for the storage fabric. In the ADAPT environment, JBODs are directly connected to servers (no storage fabric is used).

19Q. How does NCCS get Infiniband across the hall?

19A. The IB fabrics do not span the two rooms. The systems in each room are separate IB fabrics connected to the storage system using dedicated GPFS storage nodes for each cluster. The storage nodes are connected to the storage via long fibre channel cables that go between the two rooms.

20Q. How is the equipment monitored?

20A. There are a number of ways by which the equipment is monitored, and this is system dependent. Nagios is used extensively throughout the NCCS, and ganglia is also used. In addition, the facilities are monitored through a Supervisory Control and Data Acquisition

(SCADA) system.

21Q. Does monitoring data go back to a central location?

21A. For the most part yes. Nagios messages and alerts are sent to a central dashboard that sends both email and pages to staff members when there are issues. In addition, system administrators have their own monitoring scripts that may be unique to each system.

22Q. Is SGI numa-link architecture used?

22A. No.

23Q. Is there collaboration of monitoring with NASA ARC?

23A. No, system monitoring is done locally. At a higher level, system utilization and general status are combined with Ames on a monthly basis for reporting purposes.

24Q. Is there any Cray equipment?

24A. No, the NCCS has not had Cray since the early 1990's.

25Q. Is moving equipment around part of this contract?

25A. Yes, it is within scope of the contract. When equipment comes into the center, the vendor is responsible for bringing in the equipment. During operations, equipment may need to be moved within the data center, which will be the responsibility of the local contract staff. During decommissioning, the local contractors may play a part in moving equipment, but that will depend on how the systems are being decommissioned.

26Q. How is it determined to excess equipment?

26A. There are a number of factors involved, but it basically comes down to a cost benefit analysis; can the NCCS support more science and engineering output using the existing equipment versus a replacement. Many factors play a part in this decision, including facilities

(power, cooling, floor space), cost of replacement, maintainability, etc.

27Q. What about equipment maintenance?

27A. Most equipment is purchased with an initial 3-years of maintenance (but not always).

Before the initial maintenance period expires, the NCCS will evaluate whether or not it is cost effective or necessary to continue maintenance. Typically for compute systems, the NCCS will not extend maintenance. However, for most infrastructure and disk environments, the NCCS keeps these under maintenance for the entire time they are used in operations.

28Q. How does NCCS maintain configurations of racks and floor space?

28A. There are a number of tools used to maintain configurations, including spreadsheets, Visio diagrams, various documents, etc. The NCCS also uses Confluence as a central repository for relevant configuration and other documents.

29Q. Where is the shipping and receiving located?

29A. Building 28 has two loading docks, but they are non-standard height. Close coordination and communication with the vendor and shippers is needed each time a substantial shipment is made to the NCCS.

30Q. What did Code 700 use the container for?

30A. Originally it was part of the Nebula project, but it was also used as a test bed for advanced technology.

31Q. Does the NCCS store satellite data?

31A. The NCCS is not responsible for storing satellite data for the agency. Storing and disseminating observation data is the responsibility of the EODIS program. Some of the applications running on the NCCS systems do require observation data, which is copied into the system by the users. While there may be some observation data stored within the NCCS for scientific use, the NCCS is not responsible for the long-term data management of that data.

32Q. Does the NCCS have privileged access to satellite data?

32A. No. However, given the location of the data and the high-performance computing center, the NCCS does have fast network access to data stored at Goddard.

33Q. How are the disks sanitized?

33A. There are a combination of methods used to sanitize disks, including overwriting the disks with multiple passes. The NCCS also has a degausser if necessary.

34Q. What data sources are used in the applications? Are satellite data used?

34A. Yes, remote observations from satellites, airborne, and in-situ observations (such as ground stations and ocean buoys) are all used by the NCCS user community. However, the

NCCS does not have a direct link or a direct feed to obtain observation data, nor does the NCCS archive that data.

35Q. What is the name of the software that was running the hyperwall/viswall in S214?

35A. Digital Earth PC (DEPC) a custom software written by the Scientific Visualization Studio

(SVS).

36Q. How many climate models will be included in the Coupled Model Intercomparison Project version 6 (CMIP6) simulations?

36A. Approximately 60 or so models or scenarios will be run by the international community. At

NASA, the Goddard Institute of Space Studies (GISS) will use their ModelE code to run simulations on the NCCS Discover system to contribute to CMIP6.

37Q. How long will it take to run the GISS’s climate simulations for CMIP6?

37A. Their simulations for CMIP5 took approximately 6 months, and the NCCS expects their

CMIP6 simulations to take a similar amount of time.

38Q. What model produced the data for the GISTEMP visualization?

38A. The GISTEMP (GISS Surface Temperature Analysis) data is based on observed temperatures, not based on a model simulation.

39Q. [Visualization(s) that included MODIS fires:] What’s the source for the burning in Africa, South America, in the southern hemisphere – aren’t some of them man-made?

39A: Yes.

40Q. What is the resolution of the 15 screens of the Viswall/Data Visualization Theater?

40A. The current monitors are about 2 years old and are 1080p. The NCCS will be looking to replace them with higher-resolution monitors in the next couple years.

41Q. How many computer systems “drive” the Viswall screens?

41A. Currently, there are three systems that can drive the Vismall. The legacy system is scheduled to be decommissioned soon and has one server per screen. A new system is operated by the Scientific Visualization Studio (SVS), which also has one system per screen. The NCCS is running a single server attached to the wall with connections to all the screens. That single server is running the SAGE2 software.

42Q. How is data moved between the NCCS HPC systems and the Viswall?

42A. The NCCS does not have a direct link between the Viswall and the HPC storage systems.

Therefore, data is moved over high speed networks between the Viswall and HPC systems. At present, there must be “purposeful” data moves, because the data is not simply NFS (or similar) exported to the Viswall. The Viswall and the NCCS HPC systems are in different IT security zones.

43Q. Is the restriction on current model resolution (running on the NCCS Discover system) due to technology issues, or due to regulations?

43A. There are no regulatory issues with increasing the current model resolution on the Discover

HPC system. The NCCS works closely with its user community to ensure that the systems are scheduled appropriately to the users’ priorities. Therefore, the scheduling of the Discover system closely matches the users’ needs and can change regularly.

The NCCS looks for opportunities for the user community to extend their research beyond normal day-to-day work. Often times, this occurs when the NCCS gets new computational systems delivered. At the time of the SCU10 integration (~30,000 cores), the NCCS worked closely with the GMAO to enable them to run the high resolution 1.5 km research run (the highest resolution atmospheric simulation run by a US model at that time). This allows for the user community to test the scaling of their code and assess current modeling capabilities at extreme scale.

The GMAO at NASA Goddard runs an operational research GEOS forecast 4 times daily at a resolution of around 12 km. This forecast is used for a wide variety of NASA applications and also has external NASA users. One of the differentiators of GEOS is their simulation of aerosols in the atmosphere. This is critical to flight and mission planning for NASA and is also used in such applications as predicting the surface solar radiation intensity used for solar energy predictions.

NCEP and ECMWF have operational forecasts at very similar resolutions, with the ECMWF being closer to 10 km at this time. Both NCEP and ECMWF will be increasing the resolution of their operational forecasts over the next 5 years.

44Q. Does the NCCS have any research partnerships with universities?

44A. Yes, the NCCS currently supports two NSF I/UCRC (Industry–University Cooperative

Research Centers Program) programs: one at George Mason University and one at University of

Maryland Baltimore County.

45Q. Is there a relationship with National Oceanic and Atmospheric Administration (NOAA)?

45A. Yes, through a couple of ways. The Global Modeling and Assimilation Office (GMA), NCCS’s largest single user group, works closely with NOAA. Recently, NOAA made the decision to use the Finite Volume Cubed-Sphere (FV3) dynamical core in their next generation weather forecasting model. The FV3 was co-developed by GFDL and Goddard and has been used in the GEOS code for many years; GMAO is working closely with NOAA future developments. In addition, the NCCS supports the JCSDA system.

46Q. Do the GMAO models run on NCCS systems use only satellite data for input?

46A. The models use other types of observations as well, e.g., from operational surface, buoy, and weather balloon observations. The GMAO retrieves input data from the various sources, including the Goddard Distributed Active Archive [DAAC, a.k.a. Goddard Earth Sciences Data and Information Services Center (GES DISC)]. Note that the NCCS does not perform the function of a DAAC.

47Q. Is there a special relationship between the NCCS and the Goddard DAAC?

47A: The NCCS is both a producer and consumer of the DAAC data. High speed campus area networks connect the NCCS systems to the DAAC. Recently, the NCCS has been working more closely with the DAAC to store model data within the NCCS and also have it registered within the DAAC’s metadata system.

48Q. How does the NCCS prioritize the different computing jobs when there is competition for resources?

48A: As background, target utilization for the NCCS Discover cluster is 75%, although utilization has exceeded 75% for the last 6 to 8 months. NCCS users submit allocation requests, and receive yearly awards for NCCS HPC resources, from NASA Headquarters’ Science

Mission Directorate. When there is contention for resources, the NCCS works closely with the affected users, and ask them how they want to prioritize their jobs among the body work they want to run.

49Q. Is there other cloud type work going on?

49A. CISTO has supported the Science and Engineering Directorate Virtual Machine

Environment (SED VME) for quite some time. This is a relatively small VMware environment designed for persistent applications. The NCCS does not support this system.

In addition, the NCCS is working closely with CISTO and Code 700 at Goddard to create a prototype Goddard Private Cloud of similar architecture to ADAPT and managed by OpenStack.