Attachment 6 - Infrastructure Standards for Telecommunication Spaces Version 4.0.pdf

OFFICE OF

INFORMATION

AND TECHNOLOGY

Infrastructure Standard for Telecommunications Spaces

(ISTS)

Version 4.0

June 1, 2023 | Infrastructure Operations (IO)

OFFICE OF INFORMATION AND TECHNOLOGY

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 1

Revision History Table 1: Revision History

Date Reason for Changes Version Author 2007 Original 0.1 Michael Julian, RCDD

2009 Revision 0.2 Michael Julian, RCDD

2012 Revision 1.0 Michael Julian, RCDD

12/07/2016 Enterprise Data Center Infrastructure Collaboration Team (EDICT) Revision

1.1 EDICT

2/05/2018

– Expanded revision including the addition of Administrative and Operations Standards

– Name changed to include IT Support Infrastructure to reflect broader scope

1.2 EDICT

6/11/2018

– Minor update to allow for deeper network racks in cooperation with IO’s request

– Expanded allowance for fiber cabinets, cassettes, patch panels, and racks

– Changed National Data Center Operations & Logistics (NDCOL) facility ID numbers to VA Record ID numbers

– Clarified reference to OIT responsibility for the enterprise-wide technical framework and IT architecture services

2.1 EDICT

4/10/2019

– Name updated from “VA Enterprise Facility IT Support Infrastructure Standard” to “Infrastructure Standard for Telecommunications Spaces”

– Minor editorial changes resulting from initial review by the Office of Construction and Facilities Management (CFM) Facilities Standards Service (FSS) for publication in Technical Information Library (TIL)

– Removed operational information not required for building planning and design for publishing of the Standard on the TIL. Information redacted will be published as part of the Operations and Maintenance Standards for Telecommunications Spaces. In the interim, refer to the June 11th, 2018 version 2.1 of The Standard on the Baseline Configuration Management (BCM) Standards site.

– 508 compliance updates

2.2 EDICT

CFM

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 2

Date Reason for Changes Version Author

8/21/2020

– Editorial changes resulting from full review by CFM FSS for publication in CFM’s TIL

– Updated Telecommunications Room (TR), entrance room, and electrical power standards

– Added grounding and bonding, transformers, intra-building/inter-building cabling, lighting, and lightning protection system sections, and Request for Variance process instructions

– Created VA-modified Data Center Facility Environment Conditioning Standards figure to replace ASHRAE’s less stringent requirements

– Reorganized sequence of chapters for a more logical flow of data center design. Refer to the Table of Contents and the associated links to quickly find any subject of interest

– Relocated Figures, Tables, Definitions, References, and Acronyms sections to front matter for consistency with Telecommunication Industry Association (TIA) formats

– Added appendices to include Request for Variance form and OIT Design Guide Template Design Package

3.0

EDICT

CFM

7/1/21

– 4.2.8 Bonding and Grounding: changed “supplementary” ground path to “dedicated” so it cannot be confused with SBNs, SRGs, Mesh-BNs, etc.

– 4.4.4 Fiber Optic Cable: relaxed authorization for Universal Cassettes to be used when breaking out to LC connectors

– Appendix B Sheet 2: corrected split package air conditioner heat rejection requirement to be 17,000 BTU/h (5 kW) per rack

– Appendix B Sheet 3: corrected layer issue masking attributes and specifications on certain lines in the table

– Appendix B Sheet 21: corrected layer issue masking a Power Distribution Units (PDU) on left side of Plate

3.1 EDICT

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 3

Date Reason for Changes Version Author

5/1/23

– Clarified TR sizing and rack count for small clinical environments and non-medical spaces per serving area

– Added Telecommunications Enclosure (TE) specifications

– Added antenna entrance room specifications

– Added IT Equipment Power Cord section

– Harmonized the Environmental Control Equipment Requirements section with the HVAC DM

– Added a Fans and Supplemental Airflow Augmentation Device section

– Added a Physical Access to Telecommunications Spaces section

– Enhanced the Structured Cabling section to differentiate between campus structured cabling and data center structured cabling

– Updated the Request for Variance form

– Reorganized Appendix B Design Templates into sections to assist with navigation to specific topics;

Telecommunications Spaces (Architectural), Telecommunications Enclosure (TE), Electrical Distribution and Bonding, Telecommunications Distribution (Elevations, Cabling, and Routing), and Basis of Design Equipment.

– Added sheets for TEs, antenna entrance rooms, Power Distribution Unit (PDU), Uninterruptible Power Supply (UPS), temperature and humidity sensors, horizontal distance study (Diamond Analysis), electrical one-line drawings, cable routing, telecommunications spaces rack elevations, and the rule of thirds.

– Added new switch models to the bonding page, clearances around racks and power panels, and separated out the fiber from the copper media into individual sheets.

– Changed the light fixtures from 2 ft to 4 ft and replaced the 40 in. Chatsworth Network Cabinet with the Great Lakes Cabinet, and the Chatsworth Terraframe with the ZetaFrame replacement model.

4.0 EDICT

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 4

Table of Contents Revision History Table of Contents 1 Introduction

1.1 Purpose

1.2 Scope

1.3 Administration

1.3.1 Request for Variance from Infrastructure Standards

1.3.2 Request for Change to Infrastructure Standards for Telecommunications Spaces

1.3.3 Contact Information

1.4 Standards Overview

1.4.1 Background

1.4.2 Future

1.5 Objectives

1.6 Authority

2 IT Support Space Classification

2.1 Core Data Centers

2.2 Mission Support Centers

2.3 Campus Support Centers

2.4 Network Support Centers

3 Information Technology Support Space Planned Redundancy Levels

3.1 Core Data Center and Campus Support Center Planned Redundancy Levels

3.2 Mission Support Center Planned Redundancy Levels

3.3 Network Support Center, Telecommunications Room, Entrance Room, and Antenna Entrance

Room Planned Redundancy Levels

4 Infrastructure Standards

4.1 Architectural Standards and Space Design

4.1.1 Building Specifications for Common Telecommunications Spaces

4.1.2 Data Center Layout Standards

4.1.3 Telecommunications Rooms

4.1.4 Telecommunications Enclosures

4.1.5 Entrance Rooms

4.1.6 Antenna Entrance Rooms

4.2 Electrical Power and Grounding Standards

4.2.1 Topology for Power Distribution Standards

4.2.2 Uninterruptible Power Supply Specifications

4.2.3 Transformers

4.2.4 Branch Power Circuit to Rack/Cabinet

4.2.5 Zone Power Distribution Units

4.2.6 Rack Mounted Uninterruptible Power Supplies

4.2.7 Vertical Rack Power Distribution Units

4.2.8 IT Equipment Power Cords

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 5

4.2.9 Bonding and Grounding

4.2.10 Lighting

4.2.11 Metering

4.3 Mechanical and Environmental Conditioning Standards and Monitoring

4.3.1 Introduction

4.3.2 Environmental Operating Envelope Conditions

4.3.3 Data Center Facility Environment Conditioning Standards

4.3.4 Data Center Facility Environment Conditioning Standards (Amendments and Exceptions)

4.3.5 Environmental Control Equipment Requirements

4.3.6 Airflow Control

4.3.7 Monitoring

4.4 Telecommunications Standards

4.4.1 Structured Cabling

4.4.2 Unshielded Twisted Pair

4.4.3 Unshielded Twisted Pair Patch Panels

4.4.4 Unshielded Twisted Pair Patch Cord

4.4.5 Fiber Optic Cable

4.4.6 Fiber Distribution Cassettes

4.4.7 Fiber Patch Cords

4.4.8 Fiber Distribution Panel/Cabinet

4.4.9 Cable Support Infrastructure

4.4.10 Work Area Outlets

4.4.11 Horizontal Distribution

4.4.12 Fiber Optic Backbone Distribution

4.4.13 Copper Backbone Distribution

4.4.14 Server Cabinets

4.4.15 Network Equipment Racks

4.4.16 Network Equipment Cabinets

4.4.17 Telecommunications Enclosure Layout Standards

5 Administration Standards

5.1 Data Center Position Identification

5.1.1 Cross-Reference to Other Identification Systems

5.1.2 Standardized Data Center Facility Type Identification

5.1.3 Standardized Data Center Naming Convention

5.2 Information Transport Systems Equipment and Component Labeling

5.2.1 Label Materials

5.2.2 Support Infrastructure Identification

5.2.3 Transport Media and Interface Identification

5.2.4 Power Distribution Identification

5.2.5 Data Communications Cabling Labeling

5.2.6 IT Equipment Rack/Cabinet Labeling

5.2.7 Power Whip IT Equipment In-Rack/In-Cabinet Power Distribution Labeling

5.2.8 Power Distribution to IT Labeling

5.2.9 Physical Label Format

5.3 Management Standards

5.3.1 Color Coded Identification

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 6

5.3.2 Management of Telecommunications Cabling

5.3.3 Management of Airflow

5.3.4 Physical Access to Telecommunications Spaces

5.3.5 Custodial Services

Appendix A: Request for Variance A.1 Request for Variance from Infrastructure Standards A.2 Request for Change to Infrastructure Standards for Telecommunications Spaces

Appendix B: OIT Design Guide Templates B.1 Design Narrative B.2 Main Computer Rooms B.3 Entrance Rooms B.4 Antenna Entrance Rooms B.5 Telecommunications Rooms B.6 Telecommunications Enclosures

Appendix C: Figures, Tables, & Other References C.1 Figures C.2 Tables C.3 Definitions C.4 References C.5 Acronyms C.6 Contributors

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 7

1 Introduction

1.1 Purpose

The Infrastructure Standard for Telecommunications Spaces (ISTS) serves as the master reference document of criteria for Department of Veterans Affairs (VA) facility Information Technology (IT) support infrastructure. It defines the technical requirements necessary to maintain optimum reliability and efficiency within VA facilities and computing centers.

1.2 Scope

The ISTS shall be applied to all aspects of IT support infrastructure at all VA owned, operated, and leased spaces. “All aspects” includes, but is not limited to planning, design, construction, sustainment (operations, maintenance, repair), restoration, modernization, and administration.

“All VA spaces” includes but is not limited to VA Central Office (VACO) facilities and all field facilities managed by Veterans Health Administration (VHA), Veterans Benefits Administration (VBA), and the National Cemetery Administration (NCA).

IT Support Infrastructure is defined as all passive telecommunications and IT equipment, and supporting physical space and equipment, which includes the following:

• Copper and fiber cabling Information Transport Systems (ITS)

• IT equipment and systems (servers, storage, network switching and distribution, etc.)

• Telecommunications spaces, Telecommunications Rooms (TR), entrance rooms, antenna entrance rooms, data centers, horizontal distribution pathways, vertical distribution pathways, etc.

• Telecommunications equipment and systems (local and wide area network, telephone, cable television, physical security IT system, etc.)

• Physical infrastructure equipment and systems necessary to support active equipment operations at the required availability and sustainability levels (power distribution, environmental conditioning, monitoring and metering equipment, grounding, etc.)

1.3 Administration

The Standard shall be implemented without modification, except when required by an Authority Having Jurisdiction (AHJ), Federal, state, or local laws and codes. Additional requirements of AHJs shall be provided to enterprise Data Center and Infrastructure Engineering (DCIE) for evaluation of project application and possible amendment of the Standard.

Projects in the planning, design, or implementation phases impacting telecommunications distribution, space allocation, layout, electrical distribution, or mechanical systems shall contact DCIE for assistance in telecommunications space design and standards compliance.

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 8

Existing IT support infrastructure that does not conform to the ISTS shall be brought into compliance during routine tech refresh, lifecycle replacement, upgrades, new installations, or renovations of existing space.

Facility managers should take every opportunity to bring telecommunications spaces into compliance with the ISTS through incremental changes between scheduled sustainment activities.

Area Managers shall ensure that all contracting officers, IT personnel, and engineering staff are aware and make use of the ISTS.

All staff that routinely access telecommunications spaces require initial basic and annual telecommunications training. Area Managers and supervisors shall assign the approved annual list of OIT telecommunications courses corresponding to the role of the staff member.

1.3.1 Request for Variance from Infrastructure Standards

Requests for variance for allowable deviations from specific requirements of any standards published within may be authorized where a proposed alternate method of installation or operation will provide equal or higher safety, reliability, redundancy, or sustainability objectives. Requests for variance shall be submitted for consideration to DCIE (VAITESEDatacenterEngineering2@va.gov) using a Request for Variance form. If a variance is granted, it shall be limited to the particular site and installation covered in the application and will NOT be considered as a precedent for other installations.

Note: This is a separate process from VHA’s process and form for requesting variances from other TIL standards and design manuals.

1.3.2 Request for Change to Infrastructure Standards for Telecommunications Spaces

Requests for changes shall also be submitted to DCIE (VAITESEDatacenterEngineering2@va.gov) for consideration using a Request for Variance form. If approved, the proposed change(s) will be ratified through the Enterprise Baseline and Standard Review (EBSR) process and submitted to the Enterprise Change Advisory Board (CAB) for final approval (if required) before publication.

1.3.3 Contact Information

Transmit all communications to DCIE (VAITESEDatacenterEngineering2@va.gov).

1.4 Standards Overview

The ISTS includes standard specifications, decisions supporting the standard specifications, guidelines, or recommendations for implementing the standard specifications, and supplemental factors to consider when evaluating subject components.

mailto:VAITESEDatacenterEngineering2@va.gov https://dvagov.sharepoint.com/:b:/r/sites/VHAOCAMESEHRM/VHA Site Assessment Resources/Guidance Documents From Solution Delivery/Request for DCIE Variance (UPDATED 03APR2023).pdf?csf=1&web=1&e=fsdIqb mailto:VAITESEDatacenterEngineering2@va.gov https://dvagov.sharepoint.com/:b:/r/sites/VHAOCAMESEHRM/VHA Site Assessment Resources/Guidance Documents From Solution Delivery/Request for DCIE Variance (UPDATED 03APR2023).pdf?csf=1&web=1&e=fsdIqb mailto:VAITESEDatacenterEngineering2@va.gov

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 9

The Infrastructure and the Administration portions of the ISTS define the topologies and specifications for facilities and systems in the VA enterprise.

The Infrastructure Standards also provide guidance on procuring components that meet the standard specifications, guidance on integrating them with existing components, and explanation of how the subject components fit into the VA Enterprise Architecture.

The Administration Standards define the acceptable and recommended specifications for Inside Plant (ISP) naming conventions, including racks, equipment enclosures, fiber and Unshielded Twisted Pair (UTP) transport media, power distribution for active and passive elements, cable plant, and data center naming conventions.

1.4.1 Background

A standard is a set of rules or requirements that are determined by a consensus opinion of subject matter experts and prescribe criteria for a product, process, test, or procedure. The general benefits of a standard are quality, interchangeability of parts or systems, consistency, and reduction of lifecycle costs.

Telecommunications infrastructure standards are based on business needs provided through or supported by IT services. IT services are designed to support business processes and are constructed from software, hardware, and infrastructure components.

Establishing and enforcing standards for the selection and configuration of these supporting components improves the capacity, sustainability, maintainability, reliability, and availability of IT services within projected economic constraints in alignment with business needs.

1.4.2 Future

At the time of this publishing, a separate document, the Operations and Maintenance (O&M) of Telecommunications Spaces Standard is being developed as a companion piece to the Standard.

The O&M Standard will describe facility computer room operation specifications, management of telecommunications cabling, and management of airflow.

1.5 Objectives

This Standard provides specifications for use across all VA facilities to support:

• Requirement development

• Solution design

• Solution evaluation

• Solution procurement and bid evaluation

• Evaluation of architectural specifications

• Standardization of passive ISP and Outside Plant (OSP) infrastructure

• Standardization of design criteria for telecommunications spaces, facilities physical infrastructure systems, and operation and maintenance of systems supporting VA services

• Standardization of naming and identification conventions for VA data centers

• Standardization of ISP and rack power infrastructure naming conventions

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 10

• Standardization of labeling and identification conventions for physical plant systems and components

• Standardization of operations

• Operational solution procurement and bid evaluation

• Evaluation of maintenance, repair, and replacement specifications

• Standardization of lifecycle sustainment activities across all VA data centers

• Standardization of the operational, organizational, and aesthetic results of Sustainment, Restoration, and Modernization (SR&M) activities

1.6 Authority

Office of Information and Technology (OIT) is the chartered organization responsible for the enterprise-wide technical framework and IT architecture services for VA systems and projects.

Centralized responsibility promotes one technology vision across VA, which supports system optimization, integration, and interoperability throughout the enterprise.

One way in which OIT executes this responsibility is determining technical standards for deployed infrastructure technologies. These standards are also known as configuration baselines. The authority of the ISTS derives from the approved enterprise configuration baselines.

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 11

2 IT Support Space Classification VA has four segregated IT Support Space types to support the enterprise distributed computing environment.

• Core Data Centers (CDC) provide enterprise-level support services for VA-wide and cross-component systems and applications.

• Mission Support Centers (MSC) are highly specialized data centers that provide unique, focused services that are best provided from smaller, mission-focused facilities that cannot or should not be supported by CDCs or Campus Support Centers (CSC) due to the direct association MSCs have with mission-specific infrastructure or equipment.

• Campus Support Centers (CSC) operate local data centers to support VA tenants located on a VA campus and within geographically supported areas. CSCs provide services that cannot be provided (technically or economically) from a CDC. Only one CSC is authorized per VA campus.

• Network Support Centers (NSC), provide lower-level application and IT services distribution across the geographic breadth of the VA’s computing enterprise, enabling special functions and distributed users to satisfy the specific mission. Note: Telecommunications Rooms typically follow NSC rating classification and redundancy requirements.

There shall be a single data center per VA campus. Additional computer rooms used to house and operate IT server or storage equipment are not authorized. This follows with the converged physical telecommunications infrastructure networking model where backbone is distributed from the data center to TRs and other distributed telecommunications spaces, and horizontal distribution supports end-user requirements in the serving zones of these distributed telecommunications spaces. The data center on campus is the only IT support space with the redundancy and resiliency characteristics to support end-user server and storage requirements appropriately.

2.1 Core Data Centers

Core Data Centers (CDC) are facilities that provide shared and distributed enterprise services to supported sites. CDCs host cloud services to provide complete end-to-end IT services (including but not limited to application, backup, and storage) to the supported VA sites.

The CDC architecture is considered mission-essential and is designed, operated, and maintained to provide availability consistent with current VA national data center standards for supported IT applications. CDCs are the target architecture component for IT services that span multiple Department components and user groups (services and applications used by many or all users, such as messaging, productivity, and collaboration).

CDCs may be VA-owned, leased, or government or commercial outsourced (external cloud) data centers. CDCs meet (or are planned to meet) all published CDC physical infrastructure requirements to ensure enterprise reliability, redundancy, high availability, and Disaster Recovery (DR) requirements.

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 12

CDCs may also provide services designated to be provided by an NSC if no local NSC to provide those services is available. CDCs may also provide MSC-type application support.

2.2 Mission Support Centers

Mission Support Centers (MSC) provide element-specific and single-instance special enterprise services that may be inappropriate for consolidation to CDC facilities. MSCs are the target architecture for specialized-function systems and applications, particularly those that support a specific but limited local or distributed user group (services allowing the Department to function but with a limited user base such as internal financial management, or applications requiring centralized processing for a small number of user facilities, such as consolidated pharmacies).

MSC physical requirements supporting the computing environment may be more lenient than those for CDCs. Specific requirements will vary based on the supported functions, criticality of the mission, and similar criteria. For example, MSCs designated as enterprise Test and Development environments will not require the same level of high-availability physical and network redundancy as a mission-critical production environment that directly supports patient care.

MSCs may also provide services designated to be provided by an NSC if no local NSC to provide those services is available.

2.3 Campus Support Centers

Campus Support Centers (CSC) provide geographically specific, operational IT services in support of campus services to Veterans and VA employees that cannot be effectively consolidated or provided over cloud architectures to the campus.

CSCs are intended to provide for collocation of IT equipment and systems from all provider organizations including Veterans Affairs Medical Centers (VAMC), VBA Regional Offices, Office of Information Field Office (OIFO), Research & Development (R&D), Police, Security, local Facilities Management Service (FMS), Health Informatics, Clinical Engineering (CE), etc. into the minimum number of operational spaces on a campus; that is, consolidated data centers providing environmental and area network support appropriate to all IT equipment needed by tenant organizations on a particular campus to perform their missions.

CSCs shall provide NSC services for their supported campuses.

2.4 Network Support Centers

Network Support Centers (NSC) provide local IT services that cannot be effectively consolidated or provided over a network-supported distributed architecture to the local operational site.

Local means to the immediate facility and to facilities within the local commuting/networking area for the purposes of this architecture; NSCs may provide application support for all VA facilities in a metropolitan area, for example.

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 13

An NSC is the appropriate target environment where application architecture and network requirements require a closer data center point of presence to enable end users effective, efficient system access.

Table 2: Data Center Classification Standards

ID Primary Attribute Secondary Attribute Specification

1 Core Data Center (CDC)

Description

CDCs are centralized data centers that provide enterprise-level services to geographically-distributed VA organizations to support business functions. CDCs shall maintain complete High Availability (HA) and DR.

Planned ANSI/TIA-942-B Rating 3

Examples

· Information Technology Center (ITC)

· Regional Data Center (RDC)

Mission Support Center (MSC)

Description MSCs are stand-alone data centers that provide specific enterprise IT functionality to VA organizations and business functions.

Planned ANSI/TIA-942-B Rating 2 or 3

Examples

· Test/Dev data centers at Rating 2

· Centralized Mail Outpatient Pharmacy (CMOP)

· Consolidated Patient Account Center (CPAC)

3 Campus Support Center (CSC)

Description

CSCs are stand-alone data centers that provide specific IT functionality to a VA campus supporting all VA tenants on that campus and in the geographically supported area.

Planned ANSI/TIA-942-B Rating 3

Examples

· VA Medical Center (VAMC)

· CBOCs 50,000 ft² and over

· Health Care Center (HCC)

4 Network Support Center (NSC)

Description NSCs are local data centers that provide local IT application and network support to local operating locations.

Planned ANSI/TIA-942-B Rating 1

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 14

ID Primary Attribute Secondary Attribute Specification

Examples

· CBOCs under 50,000 ft²

· VBA Regional Offices

· TRs

· Non-healthcare facility entrance rooms

(healthcare facility entrance rooms and antenna entrance rooms inherit the Rating 3 redundancy requirements of their CSC)

· Other server rooms of limited capacity design (not “shallow rooms”)

· Administrative office space server rooms

Evaluation Factors

• Data center facility appropriately and centrally categorized by function and usage

• Capability gaps between existing and planned ANSI/TIA942B Rating requirements identified.

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 15

3 Information Technology Support Space Planned Redundancy Levels

VA IT support spaces shall be designed and operated in accordance with the ANSI/TIA-942-B Ratings for each data center type, except as detailed specifications are provided in this and other VA data center facilities standards.

These specifications define the minimum system redundancy levels for the physical plants and facilities physical infrastructure systems that support enterprise data centers. This Standard shall be used to quantify and qualify designs of physical infrastructure supporting data center facilities and spaces when those spaces are being designed, built, retrofitted, refreshed, and updated.

The minimum design redundancy levels for new and replacement systems in IT support spaces are detailed below. These designs balance system implementation and operation cost against availability needs for each type of space. In planning physical infrastructure system projects, reduction of potential single points of failure is generally considered more critical than increasing the redundancy level of a system.

From a design perspective on a system+system (2N) design, the smaller capacity of the two individual systems is considered the overall system capacity. For instance, a 2N system with a 500 kW side and a 750 kW side has a maximum capacity of 500 kW.

Redundancy is represented relating to the need (N). If no redundancy is required, the system simply has a redundancy level of N. To gain the simplest form of redundancy, one additional system is added represented by N+1. For example, if two Computer Room Air Conditioners (CRAC) are needed to meet the cooling need of the data center, then to obtain an N+1 redundancy level a third CRAC must be added. To provide further redundancy a 2N system could be put in place. In the CRAC example, four CRACs would create a 2N system.

Furthermore, for a 2N+1 system, five CRACs would be required to meet this level of redundancy. Finally, for a 2(N+1) system, six CRACs would be required.

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 16

Implementation Guidance

Where N+C redundancy is required for environmental control systems in CDCs, be aware that design considerations including total number and capacity of CRAC units, spacing of units in planned designs, and geometries of computer room spaces may dictate that one, two, three, or more units more than the N requirement may be necessary to ensure appropriate environmental control in all contingency situations. Computational Fluid Dynamics (CFD) modeling is required to understand the airflow and determine the effect of teaming various combinations of CRAC units and adjustment of settings.

Figure 1: Generic Redundancy Level Standards for Physical Plants, Facilities, and Infrastructure Systems

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 17

Table 3: Data Center Planned Redundancy Levels

ID Primary Attribute Secondary Attribute Specification

Facility Power Input Sources

Core Data Center (CDC)

N+1 redundant feed: one commercial power source (also known as utility), one generated power (also known as emergency)

Mission Support Center (MSC) N+1 redundant feed (one commercial, one generated) for a Rating 3 facility or single feed (N) for a Rating 2 facility

Campus Support Center (CSC) N+1 redundant feed (one commercial, one generated)

Network Support Center (NSC) N (one commercial)

2 Emergency Power Generation Source

Core Data Center (CDC) N+1

Mission Support Center (MSC) N for Rating 2 facility and N+1 for Rating 3 facility.

Campus Support Center (CSC) N+1

Network Support Center (NSC)

· Provide emergency power to TRs IAW the VA Electrical Design Manual when supporting medical center with a CSC

· Not required but recommended to support other medical and non-medical facilities

Uninterruptible Power Supply (UPS) Technical Power Systems

Core Data Center (CDC) 2N

Mission Support Center (MSC)

N+1 (distributed redundant modules or block redundant system with dedicated battery string for each module)

Campus Support Centers (CSC)

2N

Network Support Center (NSC)

N (single UPS system, single battery string)

4 Electrical Distribution (UPS to IT equipment)

Core Data Center (CDC) 2N (mirrored A/B distribution from UPS to rack in support of all IT systems) https://www.cfm.va.gov/til/dManual/dmElec.pdf

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 18

ID Primary Attribute Secondary Attribute Specification

Mission Support Center (MSC) 2N (mirrored A/B distribution from UPS to rack in support of all IT systems)

Campus Support Center (CSC) 2N (mirrored A/B distribution from UPS to rack in support of all IT systems)

Network Support Center (NSC)

N

5 Environmental Support (HVAC)

Core Data Center (CDC)

· N+1. If more than five CRACs, then N+C where C=1 for every five to eight CRACs needed

· Brief interruptions to electrical power (<10 min) will not cause loss of cooling but may cause temperature to elevate within operational range of critical equipment

Mission Support Center (MSC)

· N+1

· Brief interruptions to electrical power (<10 min) will not cause loss of cooling but may cause temperature to elevate within operational range of critical equipment

Campus Support Center (CSC)

· N+1

· Brief interruptions to electrical power (<10 min) will not cause loss of cooling but may cause temperature to elevate within operational range of critical equipment

Network Support Center (NSC) N

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 19

3.1 Core Data Center and Campus Support Center Planned

Redundancy Levels

Table 4: Core Data Center and Campus Support Center Planned Redundancy Levels

ID Primary Attribute Secondary Attribute Specification

1 Facility Power Input Sources

System Redundancy Components

N+1 redundant feed (one commercial, one generated)

Distribution Paths One active, one standby (one commercial feed, one standby generation, Automatic Transfer Switch (ATS))

System (N) Capacity

Total facility design capacity (either commercial feed or generator can provide 100 % of the facility’s design load)

2 Emergency Power Generation Source

System Redundancy Components N+1

Distribution Paths Single path ATS

System (N) Capacity Total facility design capacity (generator source can provide all critical mechanical and electrical support for the facility)

3 UPS Technical Power Systems

System Redundancy Components 2N

Distribution Paths Two simultaneously active (A/B configuration)

System (N) Capacity

· Each UPS system (A/B) rated for 100 % of data center design load

· Modular systems sized for ultimate design load but only populated to current load needs is preferred

UPS Technology Two delta conversion or double conversion UPS systems

Electrical Distribution (UPS to IT equipment)

System Redundancy Components

2N (mirrored A/B distribution from UPS to rack in support of all IT systems)

Distribution Paths Two simultaneously active (A/B distribution from UPS)

System (N) Capacity Maximum design loading of all components with one distribution path offline

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 20

ID Primary Attribute Secondary Attribute Specification

5 Environmental Support (HVAC)

System Redundancy Components

· N+1. If more than five CRACs, then N+C where C=1 for every five to eight CRACs needed

· Brief interruptions to electrical power (<10 min) will not cause loss of cooling but may cause temperature to elevate within operational range of critical equipment

Redundancy Configuration One alternate

Power Source

· Each CRAC and its associated heat rejection equipment for that CRAC powered from the same source

· Sets of heat rejection systems diversely powered (from differing sources)

· Design power sources to provide maximum reasonable avoidance of single points of failure

System (N) Capacity Total facility design capacity

Implementation Guidance

Uninterruptible Power Supply (UPS) systems shall be sized for the ultimate design load of the telecommunications space. For initial cost, maintenance, reliability, and efficiency reasons, modular UPS systems built out to provide sufficient UPS and battery capacity to meet near-term requirements are preferred over full-capacity systems.

Both the A/B UPS systems shall be powered from generator-backed sources.

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 21

3.2 Mission Support Center Planned Redundancy Levels

Table 5: Mission Support Center Planned Redundancy Levels

ID Primary Attribute Secondary Attribute Specification

1 Facility Power Input Sources

System Redundancy Components

N+1 Redundant Feed (one commercial, one generated) for a Rating 3 facility or Single feed (N) for a Rating 2 facility

Distribution Paths One active, one standby (one commercial utility feed, one standby generation, ATS)

System (N) Capacity Total facility design capacity (either source can provide 100 % of the facility’s design load)

Emergency Power Generation Source

System Redundancy Components

· N+1 for a Rating 3 facility

· N for a Rating 2 facility

Distribution Paths Single path ATS

System (N) Capacity

Total facility design capacity (generator source can provide all critical mechanical and electrical support for the facility) for a Rating 3 facility or sized for UPS and mechanical system without redundancy for a Rating 2 facility

3 UPS Technical Power Systems

System Redundancy Components

N+1 (distributed redundant modules or block redundant system with dedicated battery string for each module)

Distribution Paths Two simultaneously active (A/B configuration)

System (N) Capacity

· Each UPS system (A/B) rated for 100 % of data center design load

· Modular systems sized for ultimate design load but only populated to current load needs are preferred

UPS Technology Delta conversion or double conversion UPS systems

System Redundancy Components

2N (mirrored A/B distribution from UPS to rack in support of all IT systems)

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 22

ID Primary Attribute Secondary Attribute Specification

Electrical Distribution (UPS to IT equipment)

Distribution Paths

· Two simultaneously active (A/B distribution from UPS systems to Power Distribution Units (PDU) to the Master Power Distribution Unit (MPDU))

· Redundant, diverse path, and follows rectilinear pathways when not contained in a cable tray

System (N) Capacity 5 kW capacity for each enclosure

5 Environmental Support (HVAC)

System Redundancy Components

· N+1

· Temporary loss of electrical power will not cause loss of cooling, but may cause temperature to elevate within operational range of critical equipment for a Rating 3 facility or N+1 redundancy for mechanical equipment)

· Loss of electrical power can cause loss of cooling in a Rating 2 facility

Redundancy Configuration One alternate

Power Source

· Each CRAC and its associated heat rejection equipment for that CRAC powered from the same source

· Sets of heat rejection systems diversely powered (from differing sources)

· Design power sources to provide maximum reasonable avoidance of single points of failure

System (N) Capacity Total facility design capacity

Implementation Guidance

UPS systems shall be sized for the ultimate design load of the telecommunications space. For initial cost, maintenance, reliability, and efficiency reasons modular UPS systems built out to provide sufficient UPS and battery capacity to meet near-term requirements are preferred over full-capacity systems.

Both the A/B UPS systems shall be powered from generator-backed sources (for Rating 3 facilities).

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 23

3.3 Network Support Center, Telecommunications Room, Entrance Room, and Antenna Entrance Room Planned Redundancy Levels

Note: Healthcare facility entrance rooms and antenna entrance rooms inherit the TIA-942-B Rating 3 redundancy requirements of the CSC data center facility that they support.

Table 6: Network Support Center, Telecommunications Room, Entrance Room, and Antenna Entrance Room

Planned Redundancy Levels

ID Primary Attribute Secondary Attribute Specification

1 Facility Power Input Sources

System Redundancy Components N (one commercial)

Distribution Paths One path

System (N) Capacity Total facility design capacity (commercial feed can provide 100 % of the facility’s design load)

Emergency Power Generation Source

System Redundancy Components

· N+1 when supporting medical center with a CSC, not required, but recommended supporting other medical facilities and non-medical facilities

· N required at all sites with facility generation capabilities

Distribution Paths

· Minimum one path when supporting medical center with a CSC, not required, but recommended supporting other medical facilities and non-medical facilities

· N required at all sites with facility generation capabilities

System (N) Capacity When provided, based on planned facility ultimate load

3 UPS Technical Power Systems

System Redundancy Components

· N (single UPS system, single battery string;

may be individual rack-mount UPS systems for each rack) for NSCs and TRs

· N for entrance rooms supporting NSCs and Rating 2 MSCs

· 2N for entrance rooms and antenna entrance rooms supporting CSCs, CDCs, and Rating 3 MSCs

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 24

ID Primary Attribute Secondary Attribute Specification

Distribution Paths

· One path

· Two simultaneously active (A/B configuration) for entrance rooms and antenna entrance rooms supporting CSCs , CDCs, and Rating 3 MSCs

System (N) Capacity

· UPS system(s) rated for 100 % of critical IT system design load (5 kW per enclosure)

· Modular systems sized for ultimate design load but only populated to current load needs are preferred

UPS Technology

· Rack-mounted line interactive UPS systems (mechanical ATS optional)

· Facility UPS

Electrical Distribution to IT equipment

System Redundancy Components

· 2N

· A side shall be UPS-backed, B-side may be commercial

· 2N both sides UPS-backed for entrance rooms and antenna entrance rooms supporting CSCs, CDCs, and Rating 3 MSCs

Distribution Paths Two paths simultaneously active (A/B configuration)

System (N) Capacity 5 kW capacity for each enclosure

5 Environmental Support (HVAC)

System Redundancy Components

· N

· N+1 for entrance rooms and antenna entrance rooms supporting CSCs, CDCs, and Rating 3 MSCs

Redundancy Configuration

· None

· One alternate for entrance rooms and antenna entrance rooms supporting CSCs, CDCs, and Rating 3 MSCs

System (N) Capacity Requirement varies based on the number of racks

Implementation Guidance

UPS systems shall be sized for the ultimate design load of the telecommunications space. For initial cost, maintenance, reliability, and efficiency reasons modular UPS systems built out to provide sufficient UPS and battery capacity to meet near-term requirements are preferred over

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 25

full-capacity systems. Modular UPS systems may not be available to meet rack-mounted applications.

Healthcare facility entrance rooms and antenna entrance rooms inherit the TIA-942-B Rating 3 redundancy requirements of the CSC data center facility that they support.

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 26

4 Infrastructure Standards

4.1 Architectural Standards and Space Design

VA telecommunications spaces, including entrance rooms, antenna entrance rooms, TRs, and data centers shall be designed and operated in accordance with the ANSI/TIA-942-B ratings for each data center classification, except as detailed specifications are provided in this and other VA data center facilities standards.

4.1.1 Building Specifications for Common Telecommunications Spaces Table 7: Building Specifications

ID Primary Attribute Secondary Attribute Specification

1 Shell

Security As per Physical Security and Resiliency Design Manual (PSRDM)

Access Route Leads to location outside computer room for access control

Egress Observes life safety code

Door Height (telecommunications spaces) 8 ft minimum

Door Height (Mechanical Room) 9 ft minimum

Door Width

· CSC, CDC, and MSCs- A 6 ft double doorway opening is required with or without a mullion. If center mullion is used, it must be removable.

· NSC- 3 ft (nominal)

Finished Floor to Lowest Structural Obstacle (clear space)

12 ft minimum

Slab to Floor Above (MSC, CSC, CDC) 16 ft minimum

Raised Floor (in new data center construction only if CFD analysis determines raised floor is the best cooling solution)

· 24 in. minimum above slab

· Height determined by CFD analysis https://www.cfm.va.gov/til/PhysicalSecurity/dmPhySec.pdf https://www.cfm.va.gov/til/PhysicalSecurity/dmPhySec.pdf

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 27

ID Primary Attribute Secondary Attribute Specification

Floor Surface

· Electrostatic Discharge (ESD) coating for raised floor systems

· Static dissipative coating or material for slab floors

Bonding Per manufacturer requirements

Backboard

· 4 ft x 8 ft AC grade ¾ in. trade size fire-rated plywood backboard painted high-gloss white with two coats of fire-resistant paint for service provider/security/video/etc. around three perimeter walls of TRs, entrance rooms, antenna entrance rooms, and as required for computer rooms

· Reserve an additional 12 in. dedicated space from backboards to accommodate equipment mounted on them

2 Raised Floor Tile

Dimensions 24 in. x 24 in. nominal

Standard Tile Placed by default everywhere perforated tiles and grates are not required by CFD

Perforated Tile, High-Capacity Grates, and other High Airflow Tiles

· Only placed in the cold aisle in front of equipment inlets where CFD modeling predicts the optimum effectiveness

· Positioned so that tiles are flush to the front of the row with directional airflow aimed towards the air inlets on IT equipment in the racks

· Can be removed without interference from racks or cabinets

4.1.2 Data Center Layout Standards

Data centers will be designed in accordance with the specifications below, VA standards, and industry best practices listed in References.

CFD modeling shall be used to determine the most effective and cost-effective layout and containment solution for a given space (including new design/build, capability expansion, and lifecycle replacement). All data center designs for new data centers, expansion, or significant modification of existing data centers shall be approved through DCIE (VAITESEDatacenterEngineering2@va.gov).

Airflow separation of hot and cold air shall be a prime consideration when designing the floor layout and when selecting and procuring IT equipment. Equipment shall be specified, procured, configured, and installed so that the equipment draws cool air from the front face of the rack mailto:VAITESEDatacenterEngineering2@va.gov

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 28

and flows from the front of the rack to the rear or top of the rack and exiting either the rear door of the rack or the top through a Vertical Exhaust Duct (VED). Do not create conditions for exhaust to be recirculated back into cold air supply spaces.

Table 8: Data Center Floor Layout Standards

ID Primary Attribute Secondary Attribute Specification

1 Aisles

Orientation

· Front of racks will be oriented toward cold aisle as in Figure 2 below

· Front of racks will be aligned with each other to create a continuous linear aisle

· If a raised access floor is used, front of racks will be flush with edge of floor tiles

Width

· 4 ft minimum in cold aisles

· 3 ft minimum in hot aisles (4 ft recommended)

Cold Aisle

· Power cords if run in an access floor (with data cabling) will be placed in cable trays in cold aisles

· Overhead power cords can be run either in the hot or cold aisles

Hot Aisle

· Data cables if run in an access floor will be placed in cable trays in hot aisles

· Power cords if run in an access floor (where data cabling is run overhead) will be placed in the hot aisle

· Overhead power cords can be run either in the hot or cold aisles

· Busways will be placed in the hot aisles

Containment Hot or cold aisle containment will be determined based on data center layout, HVAC configuration, and CFD analysis

2 Support Equipment

Computer Room Air Conditioners (CRAC) Placement determined using CFD analysis

Floor Stands Manufacturer recommended floor stands for heavy equipment when raised access floor is used

Power Distribution Units (PDU)

· Floor mounted PDUs are not to be placed within cold aisles

· A-side and B-side PDUs will be aligned in a regular order to indicate power source

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 29

ID Primary Attribute Secondary Attribute Specification Branch Circuit Panelboards (BCP) (also known as Remote Power Panels (RPP))

· Not to be placed within cold aisles

· A-side BCP/RPPs aligned in a row

· B-side BCP/RPPs aligned in a row

Uninterruptable Power Supplies (UPS)

Placed in separate dedicated spaces or within the computer room IAW CFD

3 Equipment Rows

Length No more than 14 racks/cabinets per row

Orientation No rows that terminate along a wall

Cabinet/Rack/Cooling/ PDU/RPP Placement

· End of row racks or network cabinets shall be used for the Horizontal Distribution Area (HDA) to accommodate structured cabling

· Gaps between adjacent racks and cabinets within each row are closed to eliminate internal paths of bypass and recirculation of air flows

· Equipment cabinets will be used between the HDAs

Clearance

· Ensure 3 ft minimum clearance from the front, back, and one side of each rack row to walls. The fourth side must not abut a wall and needs to leave 12 in. clearance

· 4 ft minimum (6 ft recommended) distance from air conditioning and power distribution equipment (does not apply to in-row cooling/PDUs/RPPs)

· Per manufacturer requirements

· Per AHJ for egress

INFRASTRUCTURE STANDARD FOR TELECOMMUNICATIONS SPACES | 30

Figure 2: Hot Aisle/Cold Aisle Approach

Implementation Guidance

This Standard applies to new data center and telecommunications space construction as well as expansion, modification, and operation of existing facilities.

Engineered floor stands are required for CRACs, UPS battery cabinets, and other equipment capable of causing floor loading issues in raised floor environments.

VA shall not design or build “shallow rooms” or similar small spaces. Shallow rooms are intended only for access to vertical chases and riser cables installed there. Where a distribution TR is necessary in a location (i.e., the third floor of the west wing of a hospital), VA shall construct and operate out of a standardized TR.

4.1.3 Telecommunications Rooms

Telecommunications Rooms (TR) are a subset of the NSC classification archetype with less stringent environmental control requirements that:

• Serve as the transition point between the horizontal and backbone cabling systems

• Contain all networking, security, video distribution, paging, and any other system that serves the TR zone

There shall only be a single TR per serving zone. Additional TRs (or similar) are not authorized to support specific functions (e.g., FMS or CE). Likewise, TRs shall not be physically split or divided to separate organizational functions or for physical security purposes and shall not be designed, implemented, or operated. This follows with the converged physical telecommunications infrastructure networking model where backbone is distributed from the data center to TRs and other distributed telecommunications spaces and horizontal distribution supports end-user requirements in the serving zones of these distributed telecommunications spaces. The TR is the only IT support space in the serving zone that is supplied with backbone media suitable to support IT and IT-like system functions and designed with the redundancy and resiliency characteristics to support end-user equipment requirements appropriately.

I…