Appendix_B_-_NOAA_Facilities_CONOPS_Vers02_05172017_Update.pdf

Facility Concept of Operations

Version 2.0

March 1, 2017 i

Table of Contents

1. Facility Overview

1.1. ESRL Overview

1.2. GFDL Overview

1.3. NESCC Overview

2. Facility Details

2.1. Points of Contact

2.2. Computer Room Physical Layout

2.3. Power Systems and Infrastructure

2.4. HVAC Infrastructure

2.5. Fire Detection and Suppression Systems

2.6. Power and Cooling Limitations (Facility-Wide)

2.7. Room Layout Drawings Without Equipment

2.7.1. ESRL 1

2.7.2. ESRL 2

2.7.3. GFDL Computer Room

3. Current Facility Characteristics

3.1. Current Room Layouts with Equipment and Dates

3.1.1. ESRL 1

3.1.2. ESRL 2

3.1.3. GFDL Computer Room

3.1.4. NESCC HDDC-A West Computer Room

3.1.5. NESCC HDDC-A East Computer Room

3.1.6. NESCC 359 Computer Room

3.2. Current and Projected Resource Usage from Other HPC Equipment

3.2.1. ESRL1 Resource Usage

3.2.2. ESRL2 Resource Usage

3.2.3. GFDL Computer Room Resource Usage

3.2.4. NESCC HDDC-A Computer Room Resource Usage

3.2.5. NESCC Room 359 Computer Room Resource Usage

ii

4.0 Contractor Responsibilities

4.1 Processing New Contractor Personnel

4.2 Procedures for Requesting Facility Modifications

4.2.1 Facility Modification Lead Times

4.3 Procedures for Projecting Electrical Utility Usage

4.4 Contractor Responsibility for Emergency Preparedness

5.0 Facility Policies and Procedures

5.1 Physical Security

5.1.1 Access Policies and Procedures

5.1.2 Computer Room Access Controls

5.1.3 Additional Security Controls

5.2 Preventive Maintenance Policy

5.3 Data Center Work Rules

1. Facility Overview

1.1. ESRL Overview

ESRL 1 and ESRL 2 are located in the David Skaggs Research Center (DSRC), 325 Broadway, Boulder, CO. DSRC is owned by the

Government and managed by the General Services Administration (GSA). Both computing facilities are managed by the Global

Systems Division (GSD) of the Earth System Research Laboratory (ESRL). Due to the large number of visitors, the following website was developed to assist with locality information with regard to the site: http://boulder.noaa.gov/. This website will display maps, local information, driving directions, etc. DSRC is located on an access-controlled government campus with 24/7 Federal Police protection. All contract personnel requiring regular access must submit to a background check and will be issued an “affiliate” ID badge.

ESRL was formed to pursue a broad and comprehensive understanding of the Earth system. This system comprises many physical, chemical and biological processes that need to be dynamically integrated to better predict their behavior over scales from local to global and periods of minutes to millennia. At ESRL we are working toward a greater stewardship of the Earth through a number of themes aimed at understanding the Earth system processes and changes. To learn more about the work of ESRL, please visit its website at: http://esrl.noaa.gov/. There is also more current information in the ESRL Quarterly Newsletter located at:

http://esrl.noaa.gov/news/quarterly/.

Management of the building is contracted out by GSA to a building management company. This contract covers landscaping, janitorial, building modification and repair, and maintenance of major building systems. GSD contracts out for maintenance of specific UPS systems, fire suppression systems and cleaning services.

ESRL 1, located in room GA405 of DSRC, currently holds four of NOAA’s supercomputer systems (tJet, uJet, sJet and xJet) operated under a contract with CSRA. ESRL 2, located in room 2B201 of DSRC, holds one of NOAA’s supercomputer systems (vJet) and associated supercomputing support systems. This system is also operated under a contract with CSRA.

ESRL 2 also holds computer systems supporting various other GSD programs and projects not associated with NOAA High

Performance Computing.

Both computer facilities are monitored M-F, 6am to 5pm by on-site System Support Technicians. A DCIM systems provides remote monitoring afterhours and weekends. Each room is access controlled by the use of proximity keypads and other controls.

http://boulder.noaa.gov/ http://esrl.noaa.gov/ http://esrl.noaa.gov/news/quarterly/

1.2. GFDL Overview

The Geophysical Fluid Dynamics Laboratory (GFDL), within the Office of Oceanic and Atmospheric Research of the U.S.

Department of Commerce’s National Oceanic and Atmospheric Administration, is located in Princeton, New Jersey in a two-building complex on the B-Site of Princeton University’s James V. Forrestal Campus. NOAA leases the two buildings and the 6.25 surrounding acres in a triple-net-lease arrangement with Princeton University. The current lease extension runs through September 30, 2011 and the final lease extension on the current lease agreement runs through September 30, 2016. GFDL was founded in 1955 in

Washington, DC, and moved to its present location in 1968. High performance computing has been at the heart of the research of

GFDL since its inception.

GFDL develops and uses mathematical models and computer simulations to improve our understanding and prediction of the behavior of the atmosphere, the oceans, and climate. Since its inception in 1955, GFDL has set the agenda for much of the world’s research on the modeling of global climate change and has played a significant role in the World Meteorological Organization, the

Intergovernmental Panel on Climate Change assessments, and the U.S. Global Change Research Program (formerly the U.S. Climate

Change Science Program). GFDL scientists focus on model-building relevant for society, such as understanding global and regional climate change and hurricane research, prediction, and seasonal forecasting. To learn more about the work of GFDL, please visit its website at: http://www.gfdl.noaa.gov; there you will also find directions to GFDL (http://www.gfdl.noaa.gov/visiting), local hotel information (http://www.gfdl.noaa.gov/hotels), and security requirements for site access (http://www.gfdl.noaa.gov/building-access-requirements).

The two buildings that comprise the GFDL complex are connected by two links, or covered corridors, share a common 1,000 ton chilled water plant, and 3.75 MVA primary electrical service. The Main Building and Computer Building were constructed in 1968 and 1980, respectively. The Main Building is a three-story steel and glass structure of about 55,000 square feet. The Computer

Building is a slab-on-grade single-story concrete and brick structure of about 18,000 square feet. The Computer Building houses

GFDL’s Computer Room, which contains the NOAA Research and Development High Performance Computing System equipment used primarily for Climate Computing. The physical characteristics of the GFDL Computer Room are documented in the tables and images that follow, but, briefly, it is 10,000 gross square feet, comprised of a 7,052 gross square foot raised floor area and two 1,476 gross square foot hard pan areas. The Main Building, the original home of the GFDL supercomputer prior to the construction of the

Computer Building, has approximately 3,900 gross square feet of contiguous raised floor area on its 1st floor that has been repurposed and subdivided since the supercomputer is now housed in the Computer Building. If a compelling need existed, the Main Building raised floor area could be reclaimed, rehabilitated, and recommissioned as a data center.

Management of the GFDL complex is the sole responsibility of the Government. The Government executes this responsibility through a patchwork of agreements and contracts between Princeton University, Princeton Forrestal Center (the leasing agent for the Forrestal http://www.gfdl.noaa.gov/ http://www.gfdl.noaa.gov/visiting http://www.gfdl.noaa.gov/hotels http://www.gfdl.noaa.gov/building-access-requirements http://www.gfdl.noaa.gov/building-access-requirements

Campus), 2020, LLC (a government contractor that provides facilities maintenance and project management support), and other contracts that provide additional support (such as Johnson Controls for the chillers, Carrier for the Automated Building Controls

System, and SSI for the Computer Room Fire Detection and Suppression System, for example). Management, as defined here, includes all aspects of facilities management, including lawn care and landscaping, janitorial services, preventative maintenance, major maintenance, repair, and building modification.

1.3. NESCC Overview

The NOAA Environmental Security Computing Center (NESCC) is located in the Robert H. Mollohan Research Center, 1,000

Galliher Dr, Fairmont, WV 26554. The building is owned and operated by Vertex Non-Profit Organization and is managed by the

General Services Administration (GSA). NOAA also holds a contract with GSA for the O&M Contractor to perform regularly scheduled maintenance and respond to emergencies for electrical, mechanical, security and controls systems. Management of the

NESCC is the sole responsibility of the Government.

There are three (3) Data Centers located at the NESCC to support the High-Performance Computing systems.

• HDDC-A (8,500 sq.ft)

• HDDC-B (6,100 sq.ft) Unfinished

• Room 359 (6,000 sq.ft)

HDDC-A currently houses the former supercomputer (Zeus) which was commissioned in 2011 and the new supercomputer (Theia) which was commissioned in 2014. Room 359 houses the support systems for the active supercomputer (tape storage, etc). Zeus was decommissioned in 2015.

2. Facility Details

2.1. Points of Contact

Description ESRL 1 ESRL 2 GFDL Computer Rm.

Room

HDDC-A

Points of Contact Primary: Scott Nahman Alternate: Jeff VanBuskirk

Primary: Scott Nahman Alternate: Jeff VanBuskirk

Primary: Rob Taylor Alternate: Steve Mayle

Primary: Matt Allen Alternative:

Cameron Shelton

Primary: Matt Allen Alternative:

Cameron Shelton

2.2. Computer Room Physical Layout

Description ESRL 1 ESRL 2 GFDL Computer Rm.

Room

HDDC-A

Dimensions of Raised Floor

2162 sq. ft. 60’x60’=3600 sq. ft. 86’x82’ = 7,052 sq. ft. 6,000 sq. ft. 8,500 sq. ft.

Dimensions of Adjacent Non- Raised Floor

N/A N/A South side: 18’x82’ North side: 18’x52’

N/A N/A

Ceiling Height (Raised floor to drop ceiling)

10 Feet 8.5 Feet 9.5 feet 10 feet 14 feet

Maximum allowable equipment height

82” (Based on passage way ceiling height restriction)

84” (Based on 18” below drop ceiling per fire code)

96” (Based on 18” below drop ceiling per fire code)

84” (Based on egress door height)

96” (Based on egress door height)

Raised Floor Height (Slab to Top of Raised Floor)

24 Inches 12 Inches 24 Inches (except 48” trench for chilled water in middle of room)

24 inches 48 inches

Raised Floor Loading Characteristics

Tate ConCore SF 1250 Bolted Stringer:

Concentrated Load:

1250 lbs.; Uniform Load:

300 lbs./ft²; Ultimate Load: 3850 lbs.; Rolling Load: 1000 lbs. (10 Passes) Floor Specs (adjacent hallways): Vinyl tile over concrete. Floor tiles are

Tate ConCore SF 1250 Bolted Stringer:

Concentrated Load: 1250 lbs.; Uniform Load: 300 lbs./ft²; Ultimate Load:

3850 lbs.; Rolling Load:

1000 lbs. (10 Passes) Raised floor Specs (adjacent hallways):

ConCore SF 1250 Cornerlock:

Nevamar tile and CEI Bolted Stinger: Static Load: 1250 lbs;

Ultimate load: 2900 lbs. Rolling load:

1200 lbs. (10 passes)

Tate ConCore SF 1250:

Bolted Stringer:

Tate ConCore SF 2400:

Bolted Stringer:

2’x2’ and all network and power connections are made via 5” round ports located in solid tiles only.

Concentrated Load: 1250 lbs.; Uniform Load: 300 lbs./ft²; Ultimate Load:

3750 lbs.; Rolling Load:

1000 lbs. (10 Passes) Floor tiles are 2’x2’ and all network and power connections are made via 5” round ports located in solid tiles only.

Delivery Access Path

Loading Dock - Standard, designed to accommodate 18-wheeled semi-trucks.

Access Path Freight Elevator with 8000-lb.

Capacity 25’ from the Loading Dock to the freight elevator and then about 150’ to the computer room, down and up ADA-compliant ramps.

Doorframes along this path are 96 inches at their low point. Hallway low point along this path is 86 inches.

Loading Dock - Standard, designed to accommodate 18-wheeled semi-trucks.

Access Path Corridors - 8” raised floors Doorframes along this path are 83 inches at their low point Freight Elevator with 8000-lb. Capacity. 25’ from Loading Dock to the freight elevator and then about 50’ to the computer room.

Doorframes along this path are 83 inches at their low point

Loading Dock – Standard, designed to accommodate 18-wheeler semi-trucks.

Access Path – Equipment passes through an entry door from the Loading Dock into the Storage Room and then onto a hardpan staging area in the south corner of the Computer Room -a total distance of roughly 30 feet. The two sets of double doors along this path have clearances of 85 inches high by 70 inches wide.

Standard, designed to accommodate 18-wheeled semi-trucks.

Access Path Corridors to the building elevators.

Standard, designed to accommodate 18-wheeled semi-trucks.

Access Path Corridors

2.3. Power Systems and Infrastructure

Rm.

Room

HDDC-A

Transformers 480/277-volt main switchboard, rated for 3000 amps and 65,000 AIC. The gear has

480/277-volt main switchboard, rated for 3000 amps and 65,000 AIC. The gear has

Computer Building substation for HPC equipment: 2000-ampere, 480/277-volt

UT-A1, UT-A2 and UT-A3 transformers serve distribution switchgears CPS-

UT-A1, UT-A2 and UT-A3 transformers serve distribution switchgears CPS-integrated PowerNet monitoring software.

All power is fed from a single utility substation. Power is then distributed via step-down transformers and panel boards.

integrated PowerNet monitoring software. All power is fed from a single utility substation.

Power is then distributed via step-down transformers and panel boards.

main switchboard.

Rating: 1,500 kVA / 1725-kVA (with fan assist).

Computer Building transformer for motor control center and other non-HPC loads: Rating:

500 kVA

A1, CPS-A2,and

CPS-A3,

respectively. Each of CPS switchgears are rated for 3,200A, 277/480V. CPS switchgears are equipped with two automated mains, one main (CPSU) is connected to its respective UT transformer, and the other main is connected to its standby generator

(CPSG).

A1, CPS-A2,and

CPS-A3,

respectively. Each of CPS switchgears are rated for 3,200A, 277/480V.

CPS

switchgears are equipped with two automated mains, one main (CPSU) is connected to its respective UT transformer, and the other main is connected to its standby generator

(CPSG).

Generators None Full generator back up as of 1 July 2017.

None Two Cummins 2000kW generators which feeds the A-2 and A-3 line up

Two Cummins 2000kW generators which feeds the A-2 and A-3 line up

UPS and Power Conditioning

• Power Distribution:

Cutler-Hammer Electrical Distribution Equipment (480Volt, 3 Phase)

• Under floor power distribution is accomplished by flexible liquid-tight conduit receptacles.

• UPS: 800 kVA UPS System (1000 kVA Installed); 16-Minute Runtime (Full Load)

• Other: Transient Voltage Surge Suppressor (TVSS) Protected

• Emergency Power Off

• Power Distribution:

Cutler-Hammer Electrical Distribution Equipment (480Volt, 3 Phase) under floor distribution is accomplished by 50 ft.

flexible conduit power whips, fed from wall-mounted breaker panels.

• UPS: 300 kVA Chloride UPS Systems (425 kVA Installed); (200kVA available to R&D HPCS in 2010); 8-Minute Runtime (Full Load) for systems connected to the EM Generator.

Power Distribution Units (PDUs):

Five (5) United Power 225-KVA PDUs One (1) Liebert 225-kVA

PDU

One (1) Liebert 125-KVA

PDU

One (1) EPE 125-KVA

PDU

Uninterruptible Power Supplies (UPS):

800-kVA UPS MGE cabinet and battery bank Model No. ‘72-130108- 44 EPS8800/44,66’ with Serial No. S06-10615 (Originally installed in 2006) 500-kVA UPS MGE cabinet and battery bank

Power Distribution Four (4) Liebert 225-kVA PDU that supplies 208-120 Volt power feeds from under the floor through liquid tight flexible metal conduit.

• UPS: Three (3) 1000 kVA Rotary UPS System (960 kW);

14-Second Runtime (Full Load) which feed Three separate electrical distribution line-ups.

• Other: Transient Voltage Surge

Power Distribution Eight (8) Liebert 225-kVA PDU that supplies 208-120 Volt power feeds

Two (2) Liebert RPPs that supplies 208-120 Volt power feeds

Fifteen 800-amp Square D distribution panels that directly feeds 480 Volts to the two Super Computers.

Power is distributed from under the floor through liquid tight

(EPO) Switch Protected

• The DSRC Building standard for UPS Systems is the Toshiba Power Protection System.

Suppressor (TVSS) Protected

• Emergency Power Off (EPO) Switch Protected

• The DSRC Building standard for UPS Systems is the Toshiba Power Protection System.

Model No. ‘72-130104-

00 EPS 6500/44, 66’

with Serial No. 69937-01 (Originally installed in 1997) 225-kVA UPS MGE cabinet and battery bank Model No. ‘72-130101-

01 EPS 6225/44, 66’

with Serial No. 200834- 01 (Originally installed in 1998)

Suppressor (TVSS) Protected

• Emergency Power Off (EPO) Switch which is located at the exits of the datacenter. The EPO systems have a manual bypass feature which allows work to be performed in the Data Center.

flexible metal conduit.

• UPS: Three (3) 1000 kVA Rotary UPS System (960 kW); 14-Second Runtime (Full Load) which feed Three separate electrical distribution line-ups.

Suppressor (TVSS) Protected

• Emergency Power Off (EPO) Switch which is located at the exits of the datacenter. The EPO systems have a manual bypass feature which allows work to be performed in the Data Center.

2.4. HVAC Infrastructure

Rm.

Room

HDDC-A

Chilled Water Plant Building operates four

(4) 470-ton chilled-water cooling systems. Three of the systems are on EM generator power, and one is fed strictly from utility. There are four cooling towers, Building operates four

(4) 470-ton chilled-water cooling systems. Three of the systems are on EM generator power, and one is fed strictly from utility. There are four cooling towers, Centralized chilled water plant consisting of two (2) centrifugal and one (1 ) screw-type chillers connected to a common chilled water loop with associated chilled water pumps, NESCC operates three (3) 500 Ton Air-Cooled chiller in a N+1 configuration.

There are Free- Cooling and Chiller Assist mode’s where eight (8) 150-

NESCC operates three (3) 500 Ton Air-Cooled chiller in a N+1 configuration.

There are Free- Cooling and Chiller Assist each sized to match the capacity of a single chiller. There are four primary pumps, four secondary pumps, and four condenser water pumps. The chiller plant is located in a mechanical room that is adjacent to ESRL 1.

Cooling is delivered at 42oF.

A Flat Plate Heat Exchanger is employed when outside temperatures permit, thus allowing shutdown of the primary chillers.

each sized to match the capacity of a single chiller. There are four primary pumps, four secondary pumps, and four condenser water pumps. The chiller plant is located in a mechanical room that is adjacent to ESRL 1.

Cooling is delivered at 42oF.

A Flat Plate Heat Exchanger is employed when outside temperatures permit, thus allowing shutdown of the primary chillers.

condenser pumps and cooling towers.

The plant services both the Computer Building and the Main GFDL Building. The 1000 ton chilled water capacity is produced via the following distribution:

400-ton Carrier centrifugal chiller (designated Chiller #1) 350-ton York centrifugal chiller (designated Chiller #3) 250-ton York screw chiller (designated Chiller #4) Baltimore Air Coil Cooling Towers.

The plant is controlled and monitored through an electronic Building Automation System, ton ea. dry coolers provide cooling to the system to reduce chiller operations. There are three (3) Primary Pumps (40hp), three (3) Secondary Pumps (75hp) and two (2) Tertiary Pumps (20hp) that supply flow for the system.

A 25,000-gal Thermal tank is installed to provide cooling during an outage and can support up to 15 min at full system load.

mode’s where eight (8) 150-ton

ea. dry coolers provide cooling to the system to reduce chiller operations. There are three (3) Primary Pumps (40hp), three (3) Secondary Pumps (75hp) and two (2) Tertiary Pumps (20hp) that supply flow for the system. A 25,000-gal Thermal tank is installed to provide cooling during an outage and can support up to 15 min at full system load.

Room A/C Systems Six (6) 30-Ton Liebert Downdraft Computer Room Air Conditioners (CRACs) Seventeen (17) 4-Ton Liebert Extreme Density Overhead (XDO16) Systems; Ten (10) 2-Ton Liebert XDV’s; fed from two (2) Liebert Extreme Density Pumps (XDP) Cooling systems are powered from UPS backup. (No Generator)

Four (4) 30-Ton Liebert Downdraft Computer Room Air Conditioners (CRACs) Cooling systems are powered from emergency generator backed circuits.

Eight (8) 35-ton Dataflow/APC CRACs Two (2) 60-ton Liebert CRACs One (1) 20-ton Liebert Updraft CRAC Other smaller air handling units - one located in the ceiling of the Printer Room and two in the UPS room

Six (6) 20-ton Liebert CW084D CRAH units. The units are Networked together for rotation and alarms response.

Three (3) Carel Steam Humidifiers installed provide humidity control

Three (3) Custom JCI 150-ton air handlers One (1) Carel Steam Humidifier installed for humidity control.

2.5. Fire Detection and Suppression Systems

Description ESRL 1 ESRL 2 GFDL Computer Rm. Room

HDDC-A

Fire Detection Systems

Addressable “smoke head” alarm system attached to the Building Automation System

(BAS).

VESDA early smoke detection system.

Addressable alarm system (Cerberus) attached to the Building Automation System (BAS).

VESDA early smoke detection system.

Addressable low-voltage alarm system with a Simplex 4100U alarm panel in the room adjacent to the computer room, and VESDA early smoke detection system.

Addressable low-voltage alarm system with a FIKE alarm panel, and VESDA early smoke detection system. The room also has 4,855 lbs of clean agent fire suppression stored in an adjacent room.

Addressable low-voltage alarm system with a FIKE alarm panel, and VESDA early smoke detection system. The room also has 13,068 lbs of clean agent fire suppression stored in an adjacent room.

Fire Suppression Systems

Clean agent fire suppression system (Ecaro) and wet-pipe water sprinkler system with 200°F trip point.

Clean agent fire suppression system (FM-200) and wet-pipe water sprinkler system with 155°F trip point.

Clean agent fire suppression system and Dry-pipe, dual-interlocked water sprinkler system

Clean agent fire suppression system (FM-200) and wet-pipe water sprinkler system with 155°F trip point.

Clean agent fire suppression system (FM-200) and wet-pipe water sprinkler system with 155°F trip point.

Fire Extinguisher Policy

Fire extinguishers located within the data center are near the entry points and charged with CO2. Fire extinguishers throughout the remainder of the building are recessed into hallway cabinets and charged with clean agent

FE-36.

Fire extinguishers located within the data center are near the entry points and charged with CO2.

Fire extinguishers throughout the remainder of the building are recessed into hallway cabinets and charged with clean agent FE-36.

Fire extinguishers located within the data center are charged with CO2, with the exception of one extinguisher in the UPS room that is rated A-B-C for use on the UPS equipment but is not suitable for use on computers.

Fire extinguishers located within the data center are charged with FE-

36. Fire extinguishers throughout the remainder of the building are recessed into hallway cabinets and are A-B-C rated.

Fire extinguishers located within the data center are charged with FE-

36. Fire extinguishers throughout the remainder of the building are recessed into hallway cabinets and are A-B-C rated.

2.6. Power and Cooling Limitations (Facility-Wide)

Power and cooling values listed are maximum room values for HPC equipment. Existing HPC equipment in the room will reduce the power and cooling available for new equipment.

Description ESRL 1 ESRL 2 GFDL Computer Rm. Room

HDDC-A

Maximum HPC System Power

800 kVA 200 kVA 1500 kVA 400 kVA 1144 kVA

Maximum HPC Chilled Water Capacity

260 Tons 80 Tons 344 tons 100 tons 322 tons

2.7. Room Layout Drawings Without Equipment

2.7.1. ESRL 1

Room layout without HPC equipment

2.7.2. ESRL 2

Room layout without HPC equipment

2.7.3. GFDL Computer Room

Room layout without HPC equipment (except for five (5) STK Powderhorn silos)

3. Current Facility Characteristics

3.1. Current Room Layouts with Equipment and Dates

3.1.1. ESRL 1

Facility Configuration as of March 2017

3.1.2. ESRL 2

Facility Configuration as of March 2017

3.1.3. GFDL Computer Room

Facility configuration as of August 2009

3.1.4. NESCC HDDC-A West Computer Room

Facility configuration as of March 2017

3.1.5. NESCC HDDC-A East Computer Room

3.1.6. NESCC 359 Computer Room

3.2. Current and Projected Resource Usage from Other HPC Equipment The following indicates the maximum power and cooling usage by other HPC equipment not under this contract. The HPC resources available for use under the current contract may be obtained by subtracting the usage values in this section from the maximum values in section 2.7.

3.2.1. ESRL1 Resource Usage

Period Power Usage Chilled Water Usage

275 kVA 110 Tons

400 kVA 166 Tons

3.2.2. ESRL2 Resource Usage

Period Power Usage Chilled Water Usage

200 kVA 60 Tons

3.2.3. GFDL Computer Room Resource Usage

Period Power Usage Chilled Water Usage

1000 kVA 300 Tons

3.2.4. NESCC HDDC-A Computer Room Resource Usage

660 kVA 175 Tons

1144 kVA 305 Tons

3.2.5. NESCC Room 359 Computer Room Resource Usage

18 kVA 5 Tons

110 kVA 30 Tons

4.0 Contractor Responsibilities

4.1 Processing New Contractor Personnel

The Government requires new employees of the Contractor (assumed throughout to include subcontractors) to successfully complete required background checks and other clearance procedures prior to reporting for work at the facility. Note that the level of required pre-employment background investigations and clearances for employees are based on the position that they will hold with the Contractor and their level of access to the government facilities and government owned and leased equipment. It is the Contractor’s responsibility to assure that all such checks and clearance procedures are completed in a timely way so that the new employee is able to begin functioning in his/her designated position at the facility when he/she arrives and completes the necessary in-processing for new arrivals. If the required checks and clearances have not been completed satisfactorily when the new employee arrives, he/she will likely be denied access to the facility and the Contractor will be considered to have failed to fill the required position.

4.2 Procedures for Requesting Facility Modifications

The observance of formal modification request procedures helps ensure that facility changes occur smoothly with minimal disruption and downtime. Such procedures may vary depending on the depth and scope of the proposed changes, with some modifications requiring more in-depth planning, vendor involvement, and additional steps to ensure success than others.

The procedures should define all steps for the modifications, reference vendor documentation related to updating the equipment involved, and provide testing procedures for validating equipment performance after the installation is completed. Once modification procedures are defined and validated, the associated procedure should be referenced in all change documentation appropriate to the particular installation.

Project Baseline Submittal

Along with any space change requirement that the Contractor submits to the Government it shall include an overview outlining the changes to the physical space. As indicated in Section

C.6.14.3, the Contractor shall also provide the electrical consumption and cooling requirements for the equipment over the life of the equipment within the contract. In addition, the Contractor shall estimate the capital cost of the proposed change.

4.2.1 Facility Modification Lead Times

Facility modifications for ESRL 1&2 and NESCC are contracted through GSA while at GFDL they will be carried out through Princeton University, which is the landlord of the GFDL

Complex, or through the DOC Procurement Office that services GFDL. Lead times are dependent upon the scope of work being proposed. While the timing of each modification is different, experience has shown that higher cost modifications take longer beginning-to-end. The following table indicates expected lead times:

Proposed Cost of Facility Modification Estimated Lead Time to Completion

<= $500,000 6 Months to Complete

<= $1,000,000 12 Months to Complete

The contractor should plan far enough in advance to allow for contracting and construction delays for all proposed facility modifications. All facility modification must be fully funded prior to addressing the requirements with industry providers.

4.3 Procedures for Projecting Electrical Utility Usage

As indicated in Section C.6.14.3, the Government does not have additional funds to pay for electrical utility costs associated with its data center operation and therefore intends to pay for the power to operate and cool all HPC equipment under this contract from the overall pool of funds available by year for this effort. In order to provide sufficient funds to pay projected utility costs, the Government requires accurate projections of power and cooling consumption for any equipment to be installed and operated at Government facilities over its expected contract life.

ESRL

The metering of electrical utility usage is based upon actual consumption. The HPC Contractor will be held responsible for future installations in accordance with Section C.6.14.3 of the RFP.

GFDL

GFDL is responsible for paying the cost for all electricity that it uses. The lab currently monitors all electric power that is used to operate the HPC equipment under the current Raytheon contract through meters installed on the input side of the UPS units that power only the HPC equipment under the Raytheon contract. Tracking of the direct electricity consumption by the HPC equipment is straightforward. On the other hand, determining the electricity required for cooling the equipment is more problematic, since the GFDL Chilled Water Plant is used to cool the entire GFDL complex, rather than the data center alone. Because of this, the Contractor shall provide the Government with a projection of both the sustained electric consumption of each

HPC device to be installed as well as a projection of the cooling required, in units of BTUs, over the projected life of the equipment. The Government will evaluate the projected electric and cooling requirements of the equipment in order to determine the amount of funds necessary to pay for electricity for operating and cooling the HPC equipment over its projected life.

NESCC

NESCC is responsible for paying the cost of electricity that is uses in the data centers and support areas. NOAA currently monitors the HPC usage through branch circuit monitoring. The

O&M Contractor through GSA reports all electrical usage to NOAA on a monthly basis.

The Contractor shall provide the Government with a projection of both the sustained electric consumption of each HPC device to be installed as well as a projection of the cooling required, in units of BTUs, over the projected life of the equipment. The Government will evaluate the projected electric and cooling requirements of the equipment in order to determine the amount of funds necessary to pay for electricity for operating and cooling the HPC equipment over its projected life.

4.4 Contractor Responsibility for Emergency Preparedness

The Contractor shall ensure that its employees at the facility are well trained in the emergency preparedness procedures for the facility. Procedures for shutting the system down during emergencies and bringing the system back up after such an emergency shutdown shall be developed and tested on a regular basis.

In addition, the Contractor employees must participate in fire drills and any other emergency preparedness drills that take place at the facility and, if requested, serve on emergency response and/or disaster recovery teams.

5.0 Facility Policies and Procedures

5.1 Physical Security

5.1.1 Access Policies and Procedures

The Government-operated facilities that are the subject of this document have physical access controls in accordance with Department of Justice and Department of Commerce guidance.

Although local access policies vary to some degree, they are generally consistent with NOAA’s

Silver Spring Metro Complex visitor access procedure, which can be found at:

http://www.osec.doc.gov/osy/noaa/1.htm. The Government stipulates varying degrees of advanced notice and background investigation for visitors depending on the length of stay, citizenship, level of computer access requirement, and physical access requirement. For

Contractor staff that do not require unescorted machine room access or system administrator level access to the computing system, the contractor shall provide the COTR with at least two weeks advanced notice, and the Contractor employee must complete the necessary security processing paperwork found at http://www.easc.noaa.gov/Security/webfile/erso.doc.gov/PSI/ASSURANCE.htm within three days of arriving at the Government facility. For Contractor staff that requires system administrator level computer access, unescorted machine room access, or both, the Contractor is referred to the requirements of a “Critical-Sensitive Position” at http://www.easc.noaa.gov/Security/webfile/erso.doc.gov/PSI/Sensitive.htm,; note that, instead of completing an SF-86, the Contractor staff member shall complete an SF-85P through the E-QIP system and two FD-258 fingerprint card instead of an SF-87. The Contractor shall provide the

COTR with at least 30 days advanced notice and the Contractor staff member may not begin to work, until he/she completes and submits all of the required forms and satisfactorily completes a pre-employment background investigation. If the Contractor assigns a foreign national to work on this contract, that individual shall submit to the Department of Commerce foreign national reporting requirements, which are described at: http://deemedexports.noaa.gov/sponsor.html.

5.1.2 Computer Room Access Controls

The Government considers its computer rooms to be critical spaces and controls access to them, typically through use of identification card reading systems. Contract staff that have not submitted to a background investigation will not be permitted to have unescorted access to the computer rooms, and escort is subject to the availability of the Government to spare an individual to provide escort services.

http://www.osec.doc.gov/osy/noaa/1.htm http://www.easc.noaa.gov/Security/webfile/erso.doc.gov/PSI/ASSURANCE.htm http://www.easc.noaa.gov/Security/webfile/erso.doc.gov/PSI/Sensitive.htm http://deemedexports.noaa.gov/sponsor.html

ESRL

Room access within the DSRC will be granted upon successful completion and approval of the required access paperwork.

GFDL

Access to GFDL, including general building access, Computer Room access, or both, may be granted upon successful completion and approval of the required background check, see 5.1.1.

above, and a GFDL Computer Account Request Form (http://www.gfdl.noaa.gov/cms-filesystem-action/administrative/computer_account_request.pdf) or a GFDL Building Access

Request Form (http://www.gfdl.noaa.gov/cms-filesystem-action/administrative/building_access_request.pdf). GFDL will permit identity card access to individuals that are assigned an office at the site and may issue a temporary access card to less frequent contractor staff that have a specific need to be on-site. Visitors and others without approved access will be escorted, subject to the Government’s ability to spare an individual to provide escort services.

NESCC

General building access is provided by the building owner. Access to the Computer Rooms and other supporting areas is granted by the Government upon successful completion and approval of the required background check, see 5.1.1. Visitors and others without approved access will be escorted, subject to the Government’s ability to spare an individual to provide escort services.

5.1.3 Additional Security Controls

Many of the Government’s mechanical room spaces are considered to be critical, like the computer rooms. Access to these rooms is therefore controlled in the same manner as access to the computer rooms.

5.2 Preventive Maintenance Policy

Preventive maintenance on facility infrastructure equipment is the responsibility of the

Government and is performed in accordance with standard practices.

5.3 Data Center Work Rules

The Contractor employees shall read and understand the Data Center work rules. These rules are in place to make personnel aware of the environment and take precautions when entering and working in such of an environment:

Overview

The purpose of this policy is to define and explain the process, procedure and responsibilities for contractors to follow the rules and guidelines, set forth by NOAA and their Contractors, while on the premises. It is the responsibility of NOAA to ensure that all contractors are aware of, understand, and have been trained on the Data Center Work Rules policy and their requirements:

http://www.gfdl.noaa.gov/cms-filesystem-action/administrative/computer_account_request.pdf http://www.gfdl.noaa.gov/cms-filesystem-action/administrative/computer_account_request.pdf http://www.gfdl.noaa.gov/cms-filesystem-action/administrative/building_access_request.pdf http://www.gfdl.noaa.gov/cms-filesystem-action/administrative/building_access_request.pdf

For a contractor to be allowed to perform work within the data center environment to adhere to the following:

• All contractors must be trained and display an understanding of the Data Center

Work Rules. Contractors will adhere to the rules and guidelines set forth by the

Government at all times while on the premises

Personal Behavior

All contractors and vendors are expected to act professionally at all times while on-site. In order to facilitate this goal, all contractors and vendors must adhere to the following:

• All vendors, contractors and other service providers must be appropriately attired and act in a professional manner.

• No firearms, explosive chemicals or devices, or weapons of any type are allowed on the site.

• Smoking is not allowed inside the building.

• Profane language, abusive behavior, being under the influence of alcohol or drugs, sexual comments to or about employees, leering, and other offensive or inappropriate behavior will not be permitted and offenders will be asked to permanently leave the premises.

• Personal entertainment devices (Radios, MP3 players, etc.) are not permitted in

Data Center.

• No food or drink is allowed on the raised floor space. Food and drink is permitted in designated locations only.

Safety

The safety of all personnel performing work on-site is the primary concern. In order to ensure a safe and orderly work environment, all contractors and vendors will be required to adhere to the following:

• Be safe!

• When in doubt, ask!

• Be sure you thoroughly understand what you are going to do before you proceed.

You may not have a second chance to correct a mistake.

• When in doubt be conservative!

• Obey safety cones, barricades, caution tape, or other safety equipment that has been installed to guide you around hazardous areas.

▪ Red tape means DANGER, do not enter for any reason. You must be responsible for a specific task for the work being performed to cross that boundary.

▪ Yellow tape means CAUTION, do not enter without permission from the person performing the work. Do not assume a work area is safe.

• Contractors will not block access ways unless absolutely necessary to complete their work. If blocking an access way is necessary, the contractor will contact the

Government for approval. Safety cones and/or caution tape will be placed around the work area when access ways are blocked.

• Emergency evacuation routes, including stairways and exit doors, are never obstructed or blocked.

• Raised or uneven floor surfaces and other physical obstructions that may pose a tripping hazard are identified using cones or other appropriate warning devices to inform others of the hazard.

• In the event of a fire alarm, all employees, vendors, and contractors are to evacuate the building immediately. The Contractor are responsible for checking worker headcount and reporting to the fire official in charge that their people are out of the building. Do not leave the premises until you have been accounted for and have communicated your departure to your foreman or lead person.

• In the event of physical injury, please notify your Government official and building

Security personnel immediately. In an emergency call 911 or notify Security to do so.

• Follow proper ladder safety guidelines.

• Carrying awkward loads within the Data Center requires a minimum of two people

(one on each end).

• Contractors using hand and power tools and exposed to falling, flying, abrasive, and splashing objects, or exposed to harmful dust, fumes, mists, vapors, or gases are responsible to have and wear the proper PPE (Personal Protective Equipment), such as safety glasses, face shields, protective gloves, etc.

• Hazardous Materials

▪ Materials deemed hazardous must be in an acceptable container, approved by the Facilities Manager, and accompanied with a Material Safety Data

Sheet (MSDS) and a copy posted in the MSDS binder in the front lobby.

▪ Paints, solvents, adhesives, or any other flammable materials must have the approval of the Facilities Manager before they are brought inside the data center. These must be kept to a minimum, and may not be stored in the Data Center. These are considered hazardous materials and must be accompanied by a MSDS.

▪ All solvent waste or flammable liquids, leftover paint, cleaners, oily rags and other cleanup materials, and other materials are to be kept in properly labeled, fire resistant, closed containers until removed from the Data

Center.

▪ Chemical wastes are to be disposed by the Contractor in strict compliance with applicable government regulations.

▪ All spills of materials deemed hazardous must be reported immediately to the Facilities Team and the Government. Spills of certain types of hazardous materials require immediate reporting to governmental agencies. Contractors are to consider any chemical spill as a serious event.

▪ A Contractor violating this policy will be permanently barred from the premises.

▪ A Contractor attempting to “cover up” a chemical spill may be subject to legal action by governmental agencies.

▪ Always make sure you are aware of the location of the fire exits as well as the nearest fire extinguishers. Evacuation maps are posted throughout the

Data Center.

▪ In the event of a fire, earthquake, or other life threatening emergency all employees, vendors, and contractors are to evacuate the building immediately. Do not leave the premises until you have been accounted for and have communicated your departure to your foreman or lead person.

▪ Foremen or lead persons are responsible for checking their worker headcount and reporting to the fire official in charge that their people are out of the building.

▪ In the event of a medical emergency call 911. Remain with the injured person until help arrives.

Work Site Cleanliness

Maintaining a clean work area is vital to both the work being performed as well as personnel safety. All vendors and contractors will be required to:

• It is the responsibility of the contractor performing work on the premises to keep the work site clean and free of hazards.

• Contractors will not block access ways unless absolutely necessary to complete their work. If blocking an access way is necessary, the contractor will contact the

Government for approval. Visual barriers will be installed when access ways are blocked.

• Contractors will not block airflow in the data center with equipment, carts, doors, etc. unless absolutely necessary to complete their work. If blocking airflow is necessary, the contractor shall contact the Government for approval.

• To limit the number of airborne particles, all vacuums used in the data center will have a HEPA discharge filter capable of limiting discharged particles to 0.3 microns.

• Liquids are not allowed on the raised floor areas unless absolutely necessary to complete the required work and are approved by the Government.

• All packing material must be removed from equipment and components in the specified staging areas before being moved onto the Data Center floor.

• Storage of tools and equipment in the Data Center is not allowed.

• Work areas must be clean prior to contractors and vendors leaving for the day and/or upon the completion of the job.

• Any work within a Data Center that could result in the production of dust or smoke must be pre-approved by the Government so appropriate fire detection bypass procedures can be conducted.

Control of Equipment & Stop Work Authority

Due to the critical nature of a data center environment, it is imperative that all work is done in a safe manner. All vendors and contractors are required to abide by the following:

• Any person in the data center, including the Government, has the authority to pause work o Upon being issues a pause work order, the contractor must immediately place the work area in a safe condition o The contractor should immediately contact the Government for guidance.

• If there is a safety issue or work stoppage onsite, it must be reported.

Life Safety Equipment

The safety of all personnel on site is the primary concern of the Government. Failure to follow these guidelines could result in disciplinary actions.

Fire Detection and Suppression Systems

• On activation of a smoke detector intense flashing strobe lights will flash and horns will sound.

• In the event of a fire alarm, evacuate the Data Center immediately. Go directly to the nearest safe exit unless directed otherwise by Security personnel.

• Never prop open a fire rated door. Leaving doors open degrades the survivability of the building and affects computer room temperature, humidity, stability and security.

Emergency Power Off System (EPO)

• EPO buttons are located at the exits on each floor and are carefully labeled.

Operation of an EPO button removes all power within a particular computer room zone and results in a failure of the Data Center. This is an extremely serious event.

• Use the EPO buttons only in the event of a major life-threatening emergency. If possible and prudent, attempt to localize the problem before using the EPO which shuts off power to an entire computer room and has a major impact on the entire company.

• Operation of the EPO button requires pressing one or sometimes two individual buttons. A local alarm may sound at each pushbutton when the cover is removed.

The EPO system alarm may sound when two pushbuttons are simultaneously depressed. Do not press the reset button on the EPO panel. Only qualified Facilities personnel with the approval of Facilities Management can reset an EPO panel.