Attach 3 - MCFES 20-01.pdf

DEPARTMENT OF THE AIR FORCE

HEADQUARTERS SIXTEENTH AIR FORCE (AIR FORCES CYBER) (ACC)

JOINT BASE SAN ANTONIO – LACKLAND TEXAS

MEMORANDUM FOR DISTRIBUTION

FROM: 16 AF/CC

102 Hall Blvd Suite 201 San Antonio TX 78243-7072

SUBJECT: Mission Critical Facility Engineering Standard (MCFES) 20-01, Facility Electrical and Mechanical Infrastructure Redundancy Requirements

1. PURPOSE: This MCFES specifies required facility electrical and mechanical infrastructure support necessary to achieve one of three Resiliency Levels (RLs) based on mission category.

2. APPLICATION: All Sixteenth Air Force (16 AF) units shall comply with this MCFES.

2.1. Existing Facilities. Requirements are mandatory for all 16 AF units in both 16 AF occupied and non-occupied mission critical facilities. Existing facilities shall comply with the requirements herein. Through scheduled assessments or unit requests, 16AF/A4CC will evaluate existing facilities and determine compliance with this MCFES. Where compliance is not achieved, 16 AF/A4CC will develop and provide options to achieve the required RL.

2.2. New Construction/Renovation. All projects at or below 15% design completion shall comply with the requirements outlined herein. Projects beyond 15% design completion shall comply with the requirement in MCFES 15-01, dated 27 Oct 2015.

2.3. Waivers. Any deviations to requirements of this MCFES require a written waiver from 16

AF/A4C.

2.4. Authority. Per Air Force Manual (AFMAN) 32-1084, the AFISRA Civil Engineer, now designated the 16 AF Civil Engineer, has the authority to determine requirements for AFISRA Operations Facilities (category code 141456).

2.5. Conflicts. Where conflicts exist between this document, Unified Facilities Criteria (UFC) documents or other governing guidance, the project design shall comply with the most stringent requirement.

3. REFERENCES: Refer to Attachment 1 for a list of referenced publications.

4. ACRONYMS AND TERMS: Refer to Attachment 2 for a list of acronyms and terms.

5. BACKGROUND:

5.1. Office of Director of National Intelligence (ODNI) specifies Tier 3 (as defined in

ANSI/TIA-942) infrastructure as the minimum standard for intelligence community data centers as published in its document titled, Intelligence Community Data Center Standards and Guidelines. These ODNI guidelines do not address the many diverse missions performed by 16 AF. This document bridges that gap and establishes infrastructure standards and guidelines based on mission importance.

5.2. Missions are classified into four categories based on type of mission executed or the task critical asset tier rating as validated by the AF/A30-AH Critical Asset Risk Management (CARM) Program Office. The facility RL of 1, 2 or 3 is determined by its mission category.

6. FACILITY RL DESCRIPTIONS:

6.1. RL 1 Facility. Electrical and mechanical systems have single distribution pathways, N capacity components and N IEDs serving the critical and non-critical loads. Distribution pathways and capacity components must be completely shut down to perform maintenance and repair work leading to a disruption of critical loads.

6.2. RL 2 Facility.

6.2.1. The electrical system has dual distribution pathways serving the critical loads and single distribution pathways serving non-critical loads. Each pathway must have sufficient capacity to carry the load if the other pathway is unavailable. The electrical system has N+1 capacity components and 2N IEDs serving critical loads. The electrical system has N capacity components serving non-critical loads. Single distribution pathways must be completely shut down to perform maintenance and repair work leading to a disruption of critical loads. Redundant distribution pathways and capacity components can be isolated to perform maintenance or repairs without disrupting critical loads.

6.2.2. The mechanical system has single distribution pathways serving both the critical and non-critical loads. The mechanical system has N+1 capacity components and N IEDs serving critical loads. The mechanical system has N capacity components serving non-critical loads. Single distribution pathways must be completely shut down to perform maintenance and repair work leading to a disruption of critical loads. Capacity components can be isolated to perform maintenance or repairs without disrupting critical loads.

6.3. RL 3 Facility. The electrical and mechanical systems have dual distribution pathways serving the critical loads and single distribution pathways serving non-critical loads. The electrical and mechanical systems have N+1 capacity components and 2N IEDs serving critical loads. The electrical and mechanical systems have N capacity components serving non-critical loads. Each pathway must have sufficient capacity to carry the load if the other pathway is unavailable. Redundant distribution pathways and capacity components can be isolated to perform maintenance or repairs without disrupting critical loads.

6.4. Mission Category Standards. Mission categories and the associated RL are listed in Table

6.1 below. Detailed requirements for electrical and mechanical infrastructure systems are defined in Attachments 3 and 4.

Table 6.1. 16 AF Facility RL Matrix

Mission Category

RL 1

RL 2 RL 3 UPS

Optional

UPS

Required

DCA, Tier 1 TCA Tier 2 TCA, Ops Real Time Mission, Offensive Cyber Ops

Ops Non-Real Time or Specialized Mission, Defensive Cyber Ops

Ops Training

6.5. Requirements for Electrical and Mechanical Systems.

6.5.1. RL 1 Electrical Systems.

6.5.1.1. The electrical system shall have single distribution pathways and N capacity components.

6.5.1.2. Single electrical distribution pathways shall be provided to all IEDs.

6.5.1.3. Generators shall be installed as single or paralleled units to meet critical load and critical infrastructure capacity requirements.

6.5.1.4. Generators shall be rated as Emergency Standby Power according to ISO 8528-1.

6.5.1.5. Load transfers shall be accomplished by an automatic transfer switch (ATS).

Each ATS shall have an internal bypass, as well as a parallel external bypass.

6.5.1.6. The 16 AF Civil Engineer is the approval authority for inclusion of Uninterruptible Power Supply (UPS) systems in RL 1 facilities.

6.5.1.7. An automated central power and environmental monitoring and control system (PEMCS) shall be provided to monitor and control all major electrical and mechanical equipment. Electrical equipment such as main switchgear, generators, UPSs, static transfer switches (STSs), power distribution units (PDUs), ATSs, motor control centers and surge protection systems shall be monitored and controlled. Mechanical equipment such as chillers, air handlers, pumps, computer room air conditioners (CRACs), computer room air handlers (CRAHs), automatic valves, dampers, boilers, fans and generators fuel storage shall be monitored and controlled. Parameters such as temperature, humidity, system pressure and flow rates shall be monitored. PEMCS shall not be connected to external unclassified networks. Key facility personnel shall automatically be notified of alarms via phone, text and email.

6.5.1.8. UPSs shall utilize double conversion technology, be compatible with generators and have an external maintenance bypass.

6.5.1.9. An electrical connection point for portable load banks shall be provided to allow testing of the generator.

6.5.1.10. An electrical connection point for load banks shall be provided to allow testing of the UPS.

6.5.1.11. The earth electrode subsystem shall be designed and tested to provide an impedance to earth ground of 5 ohms or less. A common bonding network shall be provided.

6.5.2. RL 1 Mechanical Systems.

6.5.2.1. The Heating, Ventilation and Air Conditioning (HVAC) system shall have single distribution pathways and N capacity components. Economizer components are allowed but shall not be considered as part of design capacity.

6.5.2.2. HVAC controls shall be designed to resume programmed operations automatically in the event of power loss.

6.5.2.3. Data centers shall be maintained at positive pressure relative to internal and external areas of the facility.

6.5.2.4. In data centers, the cooling system shall maintain server air inlet temperature and humidity levels within the recommended range specified in the latest version of the ASHRAE Thermal Guidelines for Data Processing Environments.

Figure 6.1. ASHRAE Thermal Guidelines for Data Processing Environments, 2015

6.5.3. RL 2 Electrical Systems.

6.5.3.1. RL 2 facilities shall meet all requirements of RL 1. In addition, RL 2 facilities shall meet the additional requirements specified in this section.

6.5.3.2. Medium voltage distribution shall consist of dual pathways and N capacity components with the exception that service entrance transformers shall be 2N. Feed-through transformers shall not be used.

6.5.3.3. Provide a Main-Tie-Main (MTM) switchgear with programmable logic controller (PLC) and touch screen. It shall be used to control the transfers between power sources.

6.5.3.4. Dual electrical distribution pathways shall be provided to all electrical system IEDs.

6.5.3.5. A backup generator N+1 system shall be provided. Each side of the MTM switchgear shall be supplied with backup generator power capable of supporting the entire critical load and critical infrastructure.

6.5.3.6. Permanent load banks for generator testing shall be installed. Load banks shall be sized to ensure UFC 3-540-07 testing requirements are capable of being met using load banks only.

6.5.3.7. A 2N UPS system shall be provided. N UPS units shall be connected to each side of the MTM switchgear. Each UPS shall have a dedicated electrical distribution output panel.

6.5.3.8. Each data center IT equipment rack shall be fed from two PDUs. Each IT equipment rack power strip shall be fed from a different UPS. Single-corded computer equipment shall be provided with a rack-mounted static switch fed from each UPS.

6.5.3.9. Color-coded nameplates shall be used to distinguish distribution pathways and capacity components. Pathways and components originating from the left side of the tie breaker shall be colored blue. Pathways and components originating from the right side of the tie breaker shall be colored red.

6.5.3.10. Electrical receptacles feeding the IT equipment rack power strips shall be 3-phase, locking type. Each receptacle shall be identified with the PDU and circuit number which serves it.

6.5.4. RL 2 Mechanical Systems.

6.5.4.1.RL 2 facilities shall meet all requirements of RL 1. In addition, RL 2 facilities

6.5.4.2. The HVAC system shall have single distribution pathways and N+1 capacity components. Economizer components do not require N+1 capacity and shall not be considered as part of design capacity.

6.5.4.3. Mechanical capacity components shall be distributed as evenly as possible on each side of the MTM switchgear.

6.5.4.4. When external fuel tanks are included, the entire fuel system shall be provided with single distribution pathways and N+1 capacity components.

6.5.5. RL 3 Electrical Systems.

6.5.5.1. RL 3 facilities shall meet all requirements of RL 2. In addition, RL 3 facilities

6.5.5.2. Dual electrical distribution pathways shall be provided to all mechanical IEDs and all mechanical critical infrastructure.

6.5.5.3. Medium voltage distribution shall consist of dual distribution pathways and 2N capacity components. Each pathway shall originate from a different utility substation transformer.

6.5.5.4. The MTM switchgear shall include an additional tie breaker, therefore becoming a double-ended Main-Tie-Tie-Main (MTTM) switchgear.

6.5.5.5. Generators shall meet specifications of Prime Rated Power according to ISO 8528-1.

6.5.5.6. A 2N PEMCS shall be provided.

6.5.6. RL 3 Mechanical Systems.

6.5.6.1. RL 3 facilities shall meet all requirements of RL 2. In addition, RL 3 facilities shall meet the additional requirements specified in this section.

6.5.6.2. The HVAC system shall have dual distribution pathways and N+ 1 capacity components.

6.5.6.3. When heat rejection is performed solely through evaporative cooling, two independent water sources are required or a single water source with on-site water storage capacity capable of operating for 24 hours. When a combination of evaporative cooling and air cooling is utilized, a single water source is sufficient if the air-cooled heat rejection component has N capacity.

6.5.6.4. Each valve shall be installed so it can be isolated to perform maintenance or repairs without disrupting critical loads.

6.5.6.5. When external fuel tanks are included, the entire fuel systems shall be provided with dual distribution pathways and N+ 1 capacity components.

7. POINT OF CONTACT: Direct all questions and waiver requests to Dr. Jeffrey N. Rumrill, Chief, Civil Engineering, 16AFIA4C, DSN 969-2795,jeffrey.rumrill.1@us.af.mil.

-~/fJ l-lpt

TIMOTH~ ~UGH

Lieutenant General, USAF Commander

4 Attachments:

1. Referenced Publications

2. Acronyms and Terms

3. Electrical System Requirements

4. Mechanical System Requirements

DISTRIBUTION (listed on next page)

DISTRIBUTION:

16 AF/A1

16 AF/J2

16 AF/J3

16 AF/A4

16 AF/J5

16 AF/A6

16 AF/A8

16 AF/J9

16 AF/AP

16 AF/FM

16 AF/JA

16 AF/PA

16 AF/IG

16 AF/SE

16 AF/SG

16 AF/HC

16 AF/HO

16 AF/POLAD

16 AF/TD

AFCO

616 OC

AFCYBER-FWD

JFHQ-C (AF)

CO-IPE EUCOM

CO-IPE SPACECOM

CO-IPE STRATCOM

CO-IPE TRANSCOM

AFTAC

9 RW

55 RW

67 CW

70 ISRW

319 RW

363 ISRW

480 ISRW

557 WW

688 CW

38 ES

16 AF MCFES 20-01, 22 October 2020

Atch 1

ATTACHMENT 1

Reference Publications

Air Force:

Air Force Instruction (AFI) 10-2402, Critical Asset Risk Management Program AFMAN 32-1062, Electrical Systems, Power Plants and Generators AFMAN 32-1065, Grounding & Electrical Systems AFMAN 32-1084, Standard Facility Requirements Engineering Technical Letter (ETL) 01-18: Fire Protection Engineering Criteria- Electronic Equipment Installations

Department of Defense (DoD):

Department of Defense Manual (DoDM) 3020.45, Volume 1, Defense Critical Infrastructure Program (DCIP): DoD Mission-Based Critical Asset Identification Process (CAIP) DoDM 3020.45-M, Volume 3, Defense Critical Infrastructure Program (DCIP): Security Classification Manual (SCM) Intelligence Community Directive (ICD) Standards and Guidelines, Office of the Director of National Intelligence (ODNI) Military Handbook 419A, Grounding, Bonding, and Shielding for Electronic Equipment and Facilities UFC 3-410-02, Direct Digital Control for HVAC and Other Building Control Systems UFC 3-470-01, Utility Monitoring and Control System (UMCS) Front End and Integration UFC 3-520-01, Interior Electrical Systems UFC 3-540-07, Operation and Maintenance (O&M): Generators UFC 3-575-01, Lightning and Static Electricity Protection Systems

Industry Standards:

American National Standards Institute (ANSI)/Telecommunications Industry Association (TIA)- 942-B, Telecommunications Infrastructure Standard for Data Centers ANSI/TIA-607-D, Generic Telecommunications Bonding and Grounding (Earthing) for Customer Premises American Society of Heating, Refrigerating and Air-Conditioning Engineers (ASHRAE) Thermal Guidelines for Data Processing Environments Data Center Site Infrastructure Tier Standard: Topology, Uptime Institute, LLC Institute of Electrical and Electronics Engineers (IEEE) 1100-2005, IEEE Recommended Practice for Powering and Grounding Electronic Equipment International Organization for Standardization (ISO) 8528-1, Generator Set Ratings National Fire Protection Association (NFPA) 70, National Electrical Code (NEC) NFPA 780, Standard for the Installation of Lightning Protection Systems

Atch 2 (1 of 3)

ATTACHMENT 2

Acronyms and Terms

Acronyms

ATS - Automatic Transfer Switch CARM - Critical Asset Risk Management CCMD - Combatant Command CRAC - Computer Room Air Conditioner CRAH - Computer Room Air Handler DA - Defense Agency DCA - Defense Critical Asset DCIP - Defense Critical Infrastructure Program DISLA - Defense Infrastructure Sector Lead Agency DoD - Department of Defense DoDM - Department of Defense Manual EMT - Electrical Metallic Tubing EPO - Emergency Power Off HVAC - Heating, Ventilation and Air Conditioning IC - Intelligence Community IED - Intelligent Electronic Device MET - Mission Essential Task NCA - National Command Authority NEC - National Electrical Code NFPA - National Fire Protection Association ODNI - Office of Director of National Intelligence OSD - Office of Secretary of Defense PDU - Power Distribution Unit PED - Processing, Exploitation and Dissemination PEMCS - Power and Environmental Monitoring and Control System SPD - Surge Protective Device STS - Static Transfer Switch TCA- Task Critical Asset UPS - Uninterruptible Power Supply VRLA - Valve Regulated Lead Acid

Terms

Operational Real Time Mission - Cyber operations, weather operations or processing, exploitation and dissemination (PED) operations directly supporting in-progress tactical operations.

Operational Non-Real Time Mission - Cyber operations, weather operations or PED operations occurring, but not in direct support of in-progress tactical operations.

Mission Critical Facility - A facility containing one or more critical loads.

Atch 2 (2 of 3)

Critical Load - All mission essential equipment and devices, which if lost or disrupted, results in the inability to execute the mission for which the RL is earned.

Critical Infrastructure - A distribution pathway, capacity component or IED that serves critical loads.

Capacity Component - An individual electrical or mechanical equipment item that transforms energy or performs work.

Distribution Pathway - An individual electrical or mechanical element, or combination of elements that delivers or directs energy or fluid.

Intelligent Electronic Device (IED) - A microprocessor-based controller that monitors system inputs and initiates a sequence of operations for multiple capacity components or multiple distribution pathways in order to achieve or maintain a desired system state.

N - The total number of capacity components required to satisfy the design load.

N+1 - The total number of capacity components required to satisfy the design load plus one additional capacity component.

2N - Two times the total number of capacity components required to satisfy the design load

N+20% - The total number of capacity components required to satisfy the design load plus 20% additional capacity components. Fractional additional capacity component values shall be rounded up to the next integer (e.g. N=6 would result in 2 additional capacity components for a total of 8 installed capacity components).

Distribution Substation - A substation located near the end user where electricity is received, transformed to the installation distribution voltage and distributed to facilities through multiple medium voltage feeders and switches.

Data Center - An unoccupied space within a facility, with controlled access, dedicated to computer systems that process and store data.

Task Critical Asset (TCA) - An asset that is of such extraordinary importance that its incapacitation or destruction would have a serious, debilitating effect on the ability of one or more DoD or OSD components to execute the capability or mission-essential task it supports.

Defense Critical Asset (DCA) - TCAs of such extraordinary importance to operations in peace, crisis, and war that its incapacitation or destruction would have a very serious, debilitating effect on the ability of the DoD to fulfill its missions.

Tier 1 TCA - A system or asset of which the loss, incapacitation, or disruption could result in mission (or function) failure at the DoD, Military Department, Combatant Command (CCMD), Sub-Unified Command, Defense Agency, or Defense Infrastructure Sector Lead Agency

Atch 2 (3 of 3)

(DISLA) level.

Tier 2 TCA - A system or asset of which the loss, incapacitation, or disruption could result in severe mission (or function) degradation at the DoD, Military Department, CCMD, Sub-Unified Command, Defense Agency, or DISLA level.

Specialized Mission - Mission area category to describe a function that is important to the operations of an individual unit, a single base or to multiple bases, but does not qualify as Tier 1 TCA, Tier 2 TCA or Ops Real Time Mission.

Atch 3 (1 of 8)

ATTACHMENT 3

Electrical System Requirements

1. General Electrical System Requirements.

1.1. Utility Service Entrance and Distribution.

1.1.1. The switchgear and ATS shall not be installed outdoors.

1.1.2. A disconnecting means shall be installed to allow interruption of utility power to facilitate testing of the emergency backup system. The disconnecting means shall be capable of operating under load and installed ahead of the switchgear or ATS serving the critical load and critical infrastructure. The disconnecting means shall be located as close to the switchgear or ATS as possible, but no further upstream than the secondary side of the transformer. If the disconnecting means is not within the service transformer, it shall then be the same type disconnecting means as used by the switchgear's main input interrupting device.

1.1.3. ATSs shall be installed with an internal and an external/wraparound bypass.

1.1.4. All designs shall require the installing contractor to determine and set trip settings and fusing type/sizes based on a coordination study specific to the actual equipment installed. The study shall be approved by a licensed professional engineer.

1.1.5. All panels feeding mission critical loads shall use Type 1 or Type 2 Surge Protective Devices (SPDs) and comply with the requirements of UFC 3-520-01. Locate SPDs as close as practical to the breaker with a maximum lead length of 1.5 feet (450 mm).

1.1.6. Neutral conductors shall not be sized less than current carrying conductors. Single phase branch circuits shall not share neutral conductors.

1.2. Generators.

1.2.1. Generators shall be designed to account for the facility load profile (e.g. nonlinear loads).

1.2.2. Generators shall meet ISO 8528-1 performance class G3. If transient limits are provided and justified, generators shall meet performance class G4.

1.2.3. Paralleled generators shall be capable of manual synchronization in the event of failure of the automatic synchronization controls.

1.2.4. Provide a minimum of 7 days of dedicated on-site fuel storage or have agreements

Atch 3 (2 of 8) in place that ensure refueling of generator integral/day tanks for 7 consecutive days.

When the 7-day requirement is accomplished by a delivery source, provide each generator set with a minimum integral/day tank with 24-hour capacity based on the full-load fuel consumption rate of the engine. RL 3 systems shall have 7 days on-site storage capacity.

1.2.5. Fuel storage systems shall be connected to the PEMCS.

1.2.6. Test generators in accordance with UFC 3-540-07. UFC 3-540-07 requires full generator system tests twice per year with the electrical service entrance power switched off upstream of the facility. During these semi-annual tests, the runtime duration shall be increased 3 hours above the 1-hour UFC requirement for a total of 4 hours.

1.3. Uninterruptible Power Supply (UPS)

1.3.1. UPS systems shall be static type.

1.3.2. UPS systems shall be capable of automatic and manual internal bypass and shall be provided with external means (wrap-around bypass) to bypass the system and avoid interruption of power in the event of system failure, maintenance or replacement.

1.3.3. Rack-mounted and desktop UPS units shall not be used on circuits already supported by a centralized UPS. Rack-mounted UPS units serving the computer room shall be tied to the computer room Emergency Power Off (EPO) system so that the UPS units do not continue to provide power if the EPO is activated.

1.3.4. The UPS shall be programmed to limit the battery charging rate to no more than 25% of the maximum charging rate when the system is being powered by the backup generator. Electrical systems shall be sized to accommodate UPS battery recharging load.

1.3.5. The UPS battery capacity shall be sized to provide a 15-minute runtime at full-rated UPS capacity.

1.3.6. UPS systems with external battery cabinets and strings shall have a battery monitoring system capable of recording and trending individual battery jar voltage and impedance. Temperature shall be monitored and measured per manufacturer’s specifications.

1.3.7. HVAC systems serving UPS and battery rooms shall comply with the requirements of the respective RL. The HVAC system shall be designed to maintain environmental conditions specified by the UPS and battery manufacturers.

1.3.8. Under certain circumstances when the floor space within a served facility is not sufficient, the UPS, associated battery strings and other power distribution equipment may be housed in modular shelters set away from the main facility. Such shelters shall

Atch 3 (3 of 8) be sized to accommodate the equipment and allow for the required clearances. The shelters shall be installed on engineered slabs and shall be equipped with fire alarm and fire suppression systems. Shelters shall be provided with lightning protection and grounding systems as described in this standard. Shelters shall include interior and exterior lighting and security hardware.

1.4. Data Center Power Distribution.

1.4.1. PDUs shall be used for power distribution to racks.

1.4.2. Each PDU shall be provided with a main input circuit breaker.

1.4.3. Each PDU distribution section or panel shall be provided with a main circuit breaker.

1.4.4. PDUs shall include K-13 rated harmonic mitigation transformers (HMT) with copper windings and a 200% rated neutral bus.

1.4.5. Overhead power distribution shall be used in data centers. PDUs shall contain factory-installed sub-feed breakers to supply power to overhead plug-in distribution busways.

1.4.6. Each sub-feed breaker shall be labeled with the busway it feeds. Each busway shall be labeled with the PDU and sub-feed breaker from which it is fed.

1.4.7. Each PDU shall be equipped with a fully configured display.

1.5. Operations Floor Power Distribution.

1.5.1. PDUs shall be used for power distribution to critical electronic equipment and workstations on operations floors.

1.5.2. Each PDU shall be provided with a main input circuit breaker.

1.5.3. PDUs without transformers shall be fed with a 200% rated neutral feeder conductor.

1.5.4. Transformers in PDUs shall be K-13 rated HMT with copper windings and a 200% rated neutral bus.

1.5.5. Factory installed distribution panels shall be an integral part of the PDU. Each PDU distribution panel shall be provided with a main circuit breaker.

1.5.6. Each branch circuit originating from a PDU shall be labeled at the receptacle with the PDU, panel board identifier and circuit breaker number. Each branch breaker in the PDU panel shall be labeled with the receptacle, rack or workstation it feeds.

Atch 3 (4 of 8)

1.5.7. Flexible metal conduit shall not be used for under-floor power distribution.

1.5.8. Each PDU shall be equipped with a fully configured display.

1.6. Emergency Power-Off (EPO) Controls.

1.6.1. EPO controls shall be designed to minimize accidental activation.

1.6.2. EPO controls shall be supervised by the fire alarm control panel per National Fire Protection Association (NFPA) 75. Fire alarm panel shall indicate trouble condition upon loss of EPO control/system power.

1.7. Building Grounding and Lightning Protection Systems.

1.7.1. A building perimeter grounding loop shall be provided consisting of a minimum 4/0 American Wire Gauge bare copper wire buried 3 feet deep and at least 6 inches below the frost line and 3 feet from the building wall, with 10 foot x 3/4 inch copper-clad steel ground rods spaced no closer than every 20 feet along the grounding loop. Ground rod spacing between any two ground rods shall be at least the sum of their driven lengths.

Test wells shall be provided at the four corners of the loop. Intermediate test wells shall be provided along the sides where building walls exceed 300 feet in length. Building steel shall be bonded to the grounding system at every other column. All bonding connections to building steel and all connections to the grounding loop shall be exothermic weld.

1.7.2. Grounding systems and bonding shall conform to TIA-607-D (or latest revision) and be designed to satisfy the equipment manufacturers' grounding requirements.

1.7.3. All electrical equipment shall be grounded and bonded in accordance with NEC.

1.7.4. No portion of the grounding system shall exceed 5 ohms to true earth ground as measured by the four-point fall-of-potential method.

1.7.5. The requirement for lightning protection systems for RL 1 facilities shall be based on a lightning risk assessment in accordance with NFPA 780. The lightning protection systems, if installed, shall be bonded to the building grounding system/ring as required by code and as required for maximum equipment protection. Lightning protection systems shall comply with the requirements of UFC 3-575-01. A lightning protection system is required for RL 2 and RL 3 facilities.

1.7.6. SPDs shall be installed at all levels of the power distribution system that serve critical loads and critical infrastructure.

Atch 3 (5 of 8)

Table 1.1. Electrical RL Requirements

DESCRIPTION RL 1 RL 2 RL 3

MV PRIMARY DISTRIBUTION

Distribution Substations 1 1 2

Number of Feeders From Each Distribution Substation

1 2 1

SERVICE ENTRANCE

Facility Service Entrance Transformers

1 2 2

Service Disconnecting Means Located Ahead of the Critical Load Transfer System

Required Required Required

MAIN SERVICE ENTRANCE EQUIPMENT

Main Service Switchboard/Switchgear

Single Double-Ended/Main- Tie-Main

Double-Ended/ Main- Tie-Tie-Main

SPD Required at Every Level of Distribution

Feeding Critical Loads and Critical

Infrastructure

Required at Every Level of Distribution

Feeding Critical Loads and Critical

Infrastructure

Required at Every Level of Distribution

Feeding Critical Loads and Critical

Infrastructure ATS for Critical Load Transfer

Allowed Not Allowed Not Allowed

Tie-Breaker N/A Power Circuit Breaker (Draw-out Type)

Power Circuit Breaker (Draw-out Type)

Utility and Generator Feeder Circuit Breaker

Power Circuit Breaker (Draw-out Type)

Power Circuit Breaker (Draw-out Type)

Power Circuit Breaker (Draw-out Type)

Critical Load Circuit Breaker

Power Circuit Breaker (Draw-out Type)

Power Circuit Breaker (Draw-out Type)

Power Circuit Breaker (Draw-out Type)

GROUNDING

Building Perimeter Grounding Loop

Optional Required Required

Lightning Protection System

If Required by NFPA 780 Analysis

Required Required

Grounding Grid in Data Center

Required Required Required

Atch 3 (6 of 8)

BACKUP GENERATORS

Generator Configuration

N N+1 Each Side of the MTM

Switchgear Shall be Supplied with Backup

Generator Power Capable of Supporting the Entire Critical Load and Critical

Infrastructure

N+1 Each Side of the

MTTM Switchgear Shall be Supplied with

Backup Generator Power Capable of

Supporting the Entire Critical Load and

Critical Infrastructure

Generator Type Emergency Standby Power

Emergency Standby Power

Prime Rated Power

Integral/Day Fuel Tank

24 Hours at Full Load 24 Hours at Full Load 24 Hours at Full Load

External Fuel Supply Documented Delivery Capability or On-Site

Storage for 7 days

Documented Delivery Capability or On-Site

Storage for 7 days

On-Site Storage for 7 days

PEMCS Integration Required Required Required Generator Testing Comply w/ UFC 3-

540-07 (Except Simulate Utility Outage Once per

Quarter)

Comply w/ UFC 3- 540-07 (Except Simulate Utility Outage Once per

Quarter)

Comply w/ UFC 3- 540-07 (Except Simulate Utility Outage Once per

Quarter)

Load Bank Installation Electrical Connection for Portable Load

Bank Required

Permanently Installed Load Bank Required

Permanently Installed Load Bank Required

Atch 3 (7 of 8)

UPS

UPS System N 2N 2N

UPS Internal Maintenance Bypass

Required Required Required

UPS External (Wrap- Around) Bypass

Required Required Required

UPS Power Distribution to Critical Loads

Through PDUs with Harmonic Mitigation or K-rated Transformers

Through PDUs with Harmonic Mitigation or K-rated Transformers

Through PDUs with Harmonic Mitigation or K-rated Transformers

UPS Room EPO Required Required Required

UPS Maintenance Contract

Required Required Required

UPS Room Cooling N N+1 N+1

Load Bank Connection Point

Required Required Required

UPS BATTERY SYSTEM

Battery Type Valve Regulated Lead Acid (VRLA)

VRLA VRLA

Battery Backup Time 15 Minutes at UPS Rated Capacity

15 Minutes at UPS Rated Capacity

15 Minutes at UPS Rated Capacity

Battery Monitoring System

Required Required Required

PEMCS Integration Required Required Required

Data Center EPO Required Do Not Install Unless Required by Authority

Having Jurisdiction (AHJ) or User

Do Not Install Unless Required by AHJ or

User

Battery Room EPO Required Required Required

Battery Room Cooling N N+1 N+1

Atch 3 (8 of 8)

POWER DISTRIBUTION UNITS (PDU)

PDU Transformer K-13 Rated, HMT with Copper Windings

K-13 Rated, HMT with Copper Windings

K-13 Rated, HMT with Copper Windings

PDU Configuration 200% Rated Neutral, Dedicated Main

Circuit Breaker for Each Panel Board

200% Rated Neutral, Dedicated Main

Circuit Breaker for Each Panel Board

200% Rated Neutral, Dedicated Main

Circuit Breaker for Each Panel Board

Power Distribution from PDUs to Data Center Racks

Overhead Busway Overhead Busway Overhead Busway

Power Distribution from PDUs to Ops Floor Computers

Branch Circuits Under Raised Floor

Branch Circuits Under Raised Floor

Branch Circuits Under Raised Floor

PDU EPO Required Required Required PDU Display Required Required Required

PEMCS & ELECTRICAL SYSTEM IED

Minimum Monitoring Points Shall Include

Utility Transformers, Main Distribution

Switchboard, Switchgear, UPS, Battery Room, Generators, Generator

Fuel Tanks, ATS, PDUs, STSs, CRAHs, AHUs, Chiller Plant

Utility Transformers, Main Distribution

Switchboard, Switchgear, UPS, Battery Room, Generators, Generator

Fuel Tanks, ATS, PDUs, STSs, CRAHs, AHUs, Chiller Plant

Utility Transformers, Main Distribution

Switchboard, Switchgear, UPS, Battery Room, Generators, Generator

Fuel Tanks, ATS, PDUs, STSs, CRAHs, AHUs, Chiller Plant

External Alarm and Status Notification Means

Phone, Text and Email Phone, Text and Email Phone, Text and Email

PEMCS Architecture Continuous Polling in Closed Loop Using Industry Standard Protocols (Non-

Proprietary)

Continuous Polling in Closed Loop Using Industry Standard Protocols (Non-

Proprietary)

Continuous Polling in Closed Loop Using Industry Standard Protocols (Non-

Proprietary)

PEMCS Central Processing Unit

N N 2N

UPS Power to PEMCS Single path (When Available)

Single Path Dual Path

IEDs N 2N 2N

UPS Power to IEDs Single path (When Available)

Dual Path Dual Path

Atch 4 (1 of 4)

ATTACHMENT 4

Mechanical Systems Requirements

1. General Mechanical Systems Requirements.

1.1. Airflow Management.

1.1.1. Data centers shall be provided with adequate outside air to maintain positive pressure relative to adjacent spaces and satisfy ASHRAE Standard 62.1.

1.1.2. Airflow shall be managed such that the inlet temperature and humidity supplied to each piece of IT equipment falls within the recommended thermal guidelines defined by ASHRAE in Figure 7.1. Cooling efficiency and capacity shall be maximized by preventing rack inlet and exhaust air from mixing.

1.1.3. Racks shall be installed in rows with fronts of cabinets/racks facing each other to create cold aisles. The rear of cabinets/racks shall face each other to create hot aisles.

Full hot or cold aisle containment shall be utilized.

1.1.4. Perforated floor tiles shall only be installed in the cold aisles. Perforated tile airflow shall match the IT equipment airflow demand.

1.1.5. Blanking plates shall be installed within racks to prevent re-circulation of hot exhaust air within the racks.

1.1.6. When a raised floor is used as a plenum, all penetrations shall be sealed to minimize air leakage.

1.1.7. Raised floor height shall be a minimum of 24 inches. Underfloor obstructions between CRAC/H discharge and the perforated floor tiles shall be minimized

1.1.8. Data centers with white space greater than 1000SF or having a cooling design load greater than 100kW shall be validated using computational fluid dynamics modeling.

Model shall validate all rack inlet temperatures conform to the recommended range of ASHRAE Thermal Guidelines for Data Processing Environments at design load with N capacity components. Model shall be submitted for review with the initial design submission or prior to 50% design completion, whichever is sooner

1.2. Computer Room Air Conditioning/Handling Units.

1.2.1. Ducting shall not be used with CRAC and CRAH units, however return chimneys are allowed.

Atch 4 (2 of 4)

1.2.2. If present, ceiling tiles shall be non-flaking or non-dusting and vapor resistant. If a gaseous fire suppression system is used, ceiling tiles shall have hold down clips.

1.2.3. CRAC/H units shall be installed in opposed pairs and located perpendicular to hot aisles. Power shall be distributed to units such that loss of power to a panel or bus does not result in loss of power to two or more adjacent units.

1.2.4. CRAC/Hs shall be networked and centrally controlled to maintain equipment inlet temperatures within the recommended thermal guidelines defined by ASHRAE in Figure 7.1.

1.3. Piping and Pumps.

1.3.1. Water and drain pipes not supporting mission critical areas shall not be routed through or above the facility mission critical areas.

1.3.2. Valves shall be installed to permit isolation of individual capacity components.

1.3.3. Floor drains or sump pumps shall be installed in data centers.

1.3.4. All plumbing shall be insulated.

1.3.5. Leak-detection systems shall be installed in raised floor environments.

1.4. Fire Protection.

1.4.1. Data center fire protection shall comply with ETL 01-18: Fire Protection Engineering Criteria – Electronic Equipment Installations.

1.4.2. The power to all electronic equipment shall be automatically disconnected upon activation of a gaseous agent total flooding suppression system.

Atch 4 (3 of 4)

Table 1.1. Mechanical RL Requirements

HEATING, VENTILATION AND AIR CONDITIONING (HVAC)

Temperature and Humidity Control for Data Centers

Maintain Temperature and Humidity Within the Recommended Range Specified in the

ASHRAE Thermal Guidelines for Data

Processing Environments

Maintain Temperature and Humidity Within the Recommended Range Specified in the

ASHRAE Thermal Guidelines for Data

Processing Environments

Maintain Temperature and Humidity Within the Recommended Range Specified in the

ASHRAE Thermal Guidelines for Data

Processing Environments

Electrical Service to Mechanical Equipment

Single Path Single Path Dual Path

HEAT REJECTION COMPONENTS AND PATHWAYS

Chilled Water Piping Distribution System

Single Path Single Path Dual Path

Condenser Water Piping Distribution System

Single Path Single Path Dual Path

Chilled Water Pumps N N+1 N+1

Condenser Water Pumps

N N+1 N+1

Air-Cooled Chillers N N+1 N+1

Water-Cooled Chillers N N+1 N+1

Cooling Towers N N+1 N+1

Direct Expansion Systems

N N+1 N+1

AIR HANDLING SYSTEMS

Air Handling Units N N+1 N+1 CRAC/CRAH Units N N+20% N+20% In-Row Cooling Units N N+20% N+20% In-Rack Cooling Units N N+20% N+20%

(Fans and Coils) (Fans and Coils) Fan Coil Units N N+1 N+1 Data Center Aisle Air Containment

Required Required Required

Atch 4 (4 of 4)

PLUMBING FOR WATER-COOLED HEAT REJECTION

Condenser Makeup Water

Single Water Source, On-Site Backup

Storage Not Required

Single Water Source, On-Site Backup

Storage Not Required

1. Dual Water Sources, or

2. Single Water Source and 24-Hour On-Site Storage, or

3. Single Water Source and Air-Cooled Backup with N Capacity

Data Center Floor Drainage

Floor Drain or Sump Pump

Floor Drain or Sump Pump

Floor Drain or Sump Pump

Data Center Leak Detection

Required Under Raised Floors

Required Under Raised Floors

Required Under Raised Floors

GENERATOR FUEL SYSTEM

External Storage Tank N N N+1 Fuel Delivery Pumps N N+1 N+1 Fuel Piping Distribution System

Single Path Single Path Dual Path

DATA CENTER FIRE SUPPRESSION

Wet Pipe Fire Sprinkler System

Required Required Required

Pre-Action Fire Sprinkler System

May Substitute for Wet Pipe Fire

Sprinkler System if All Conditions of ETL

01-18 are Satisfied

May Substitute for Wet Pipe Fire

Sprinkler System if All Conditions of ETL

01-18 are Satisfied

May Substitute for Wet Pipe Fire

Sprinkler System if All Conditions of ETL

01-18 are Satisfied Supplemental Clean Agent Fire Suppression System

Allowed Allowed Allowed

Smoke Exhaust System

Required Required Required

Ultrasensitive Smoke Detection System

Required Required Required

MECHANICAL SYSTEM IED

IEDs N N 2N

UPS Power to IEDs Single Path (when available)

Single Path Dual Path