05A10 Appendix A10 Generic Extra-Small CSC Design Narrative.pdf

OFFICE OF

INFORMATION

AND TECHNOLOGY

Design Narrative Generic Extra-Small Data Center for Main Computer Rooms (MCRs) and other Campus Support Centers (CSCs)

14 May 2021 Data Center & Infrastructure Engineering Solution Delivery

Greenfield Design Narrative – Extra-Small Data Center for MCRs and other CSCs | 1

Table of Contents Table of Contents 1................................................................................................................................... Background 2......................................................................................... Campus Support Center (CSC) Overview

2.1 Main Computer Rooms (MCRs)

2.2 Other Campus Support Centers (CSCs)

3.........................................................................................................................Design Description

3.1 Architectural

3.2 Electrical

3.3 Mechanical

3.4 Telecommunications

IT Equipment Enclosures Cable Tray Structured Cabling Fiber Backbone

4...................................................................................... Other Telecommunications Infrastructure

4.1 Entrance Rooms

4.2 Telecommunications Rooms

4.3 Fiber Backbone

5...................................................................................................................... Contact Information

Greenfield Design Narrative – Extra-Small Data Center for MCRs and other CSCs | 2

1 Background There is a recognized need to provide a generic Campus Support Center (CSC) data center design that can be applied without significant modifications to future construction projects.

The existence of a standardized, standards-based design will prevent non-compliant, incorrectly sized, unbalanced, and unsustainable data center designs from being produced by VA mission partners, reducing design project timelines and costs, and preventing unnecessary rework and change orders.

The Office of Information Technology (OIT) Solution Delivery (SD) Data Center & Infrastructure Engineering team has developed the accompanying design package (Generic Extra-Small Data Center Design Package for Main Computer Rooms (MCRs) and Other Campus Support Centers (CSCs)) for Architect/Engineer (A/E) partners to use in their design drawing development for new construction projects across VA.

This design narrative describes the generic CSC design utilizing a standardized TIA-942-B Telecommunications-Electrical-Architectural-Mechanical (TEAM) approach that will allow A/E designs to separate requirements by discipline. Disciplines not directly related to the telecommunications infrastructure (e.g. structural, plumbing, fire protection) are not addressed.

2 Campus Support Center (CSC) Overview CSCs provide geographically specific, operational IT services in support of campus services to Veterans and VA employees that cannot be effectively consolidated or provided over cloud architectures to the campus. Information Technology (IT) services at CSCs support multiple organizations, including Clinical Engineering (CE, a.k.a. biomed), Facility Management Services (FMS), VA police, Research & Development (R&D), Veterans Benefits Administration (VBA) Regional Offices, Office of Information Field Offices (OIFOs), and others.

Design models for CSCs are intended to provide for collocation of IT equipment and systems from all provider organizations into the minimum number of operational spaces on a campus;

that is, consolidated data centers providing environmental and area network support appropriate to all IT equipment needed by tenant organizations on a particular campus to perform their missions. There should only be a single data center on a campus that hosts all of the locally necessary IT hardware to operate the site; similarly, all IT hardware necessary to operate the site should be physically located in the single data center described.

2.1 Main Computer Rooms (MCRs)

The most common type of CSC in the VA inventory is the Main Computer Room (MCR) that provides IT services to a VAMC campus.

Greenfield Design Narrative – Extra-Small Data Center for MCRs and other CSCs | 3

Legacy MCRs in in-patient VA Medical Centers typically support between 25kW and 75kW of critical load, with significant additional loads anticipated as VA modernizes electronic health record systems and deploys additional medical systems increasingly dependent upon IT systems.

Analysis of projected IT trends and needs has led VA to standardize on a single CSC design for major VA outpatient clinical facilities (known variously as super-CBOCs (Community Based Outpatient Clinics, ACCs (Ambulatory Care Centers), and HCCs (Health Care Centers)) with a footprint larger than 50,000 square feet (and with 48 or fewer Telecommunications Rooms (TRs), which will satisfy anticipated telecommunications support needs for a twenty (20) year data center life cycle (coinciding with the expected typical lease length for many of these facilities)

Utilize this ‘extra-small’ design when indicated by OIT Data Center & Infrastructure Engineering.

2.2 Other Campus Support Centers (CSCs)

Consult OIT Data Center & Infrastructure Engineering for the appropriate size and design of CSCs during project scoping for other facility classifications.

3 Design Description This design narrative describes a generic, standardized CSC suitable for insertion into new construction at selected VA outpatient clinical facilities with a footprint larger than 50,000 square feet with fewer support requirements to provide all necessary IT services for all tenants.

The standards-based design will result in a TIA-942 Rating 3 CSC that complies with the requirements of the VA’s Infrastructure Standard for Telecommunications Spaces.

The ultimate design load for this facility is 41.2kW, which is sufficient to support all existing loads, anticipated equipment consolidations, deployment of Cerner EHRM equipment, and provides for reasonable amounts of growth over the planned 20-year lifecycle of the CSC.

Physical space, electrical capacity and distribution, heat rejection, and telecommunications elements have been holistically designed and balanced in this design.

Include all described elements (except for active server/storage/switch equipment and patch cords) in the scope of the construction project.

3.1 Architectural

Any modifications to the generic design will require the design team to conduct additional Computational Fluid Dynamics (CFD) analysis to show that the new design functions as intended at all design loading and contingency conditions. Contact the Data Center & Infrastructure Engineering team for additional information as required.

Greenfield Design Narrative – Extra-Small Data Center for MCRs and other CSCs | 4

For maximum flexibility, there are two (2) basic designs for the generic extra-small CSC. The ‘square’ extra-small design requires a 24’x34’-4-1/2” (775 square foot) footprint without column interruptions. The ‘rectangle’ extra-small design requires a 20’x39’-2-3/4” (785 square foot) footprint without column interruptions. Insertion of columns will require an increase in the allocated square footage, and is not recommended. This space is sufficient to house both the IT equipment and the endpoint electrical and mechanical support equipment necessary to support the critical load. The minimum height of the space is twelve (12) feet, and sixteen (16) feet are suggested. Provide a minimum 6’ width 8’ height double door to the MCR either without a mullion or with a removable one.

Future CSCs will be slab-floor facilities, so raised access floor systems are not required. There shall not be a suspended ceiling in future CSCs. In this design the CSC will have contained return air ducting providing airflow separation.

Depending on the generic design selected, the generic extra-small CSC supports either a single row or split, two-row system of server and networking equipment, a 2N UPS system, and an N+1 heat rejection system sized to support the ultimate design load for the CSC. Dimensions and layout for the generic extra-small CSC are shown on sheets 4 and 9 of the Generic Extra- Small Data Center Design Package.

There is insufficient physical space in the generic extra-small design for other telecommunications functions (e.g. carrier circuit demarc/Entrance Room functions) to be collocated in the data center.

3.2 Electrical

The ultimate design for the critical load in the generic extra-small CSC is 41.2kW, which provides for six (6) IT equipment server cabinet positions at Standard Density. UPS systems sized to support this ultimate design load are necessary. 2N UPS redundancy in the CSC is a requirement for this classification of facility.

Provide power to the CSC 2N Uninterruptible Power Supply (UPS) systems from separate distribution panels both on the generator-backed facility electrical bus.

In order to provide the 2N redundancy necessary for critical electrical distribution for the mission critical CSC, Data Center & Infrastructure Engineering specifies installation of a matched pair of modular UPS systems suitable for all capacity levels up to the ultimate design. Adoption of the modular technology will decrease initial capital costs to procure these systems, as well as provide higher operating efficiencies for the UPS systems over their lifecycle. As the critical load in the MCR increases with additional IT workload over time, the systems can be gradually built up (using operations, not capital funds) to support the new critical loads.

Individual facilities will have varying levels of critical load at the time of CSC commissioning.

Procure the UPS systems to have sufficient modular UPS and battery elements to support the load at the time of commissioning with N+1 internal elements in each UPS. (Example: The

Greenfield Design Narrative – Extra-Small Data Center for MCRs and other CSCs | 5 modular UPS system has 10kW UPS modules. At the time of commissioning, the critical load is 16kW. The smallest number of UPS modules that can support this load is 2 (20kW capacity).

Provide three (3) 10kW modules in each UPS to provide internal N+1 capability.)

In order to achieve the 41.2kW ultimate load, Data Center & Infrastructure Engineering recommends specification of a modular UPS system similar to the Eaton 93PM (93PM-L for 208V applications), expandable in 10kW increments with 120/208V output, initially built out to support the planned critical load at commissioning. At fully-built out condition, the battery cabinet is estimated to be able to provide around 15 minutes of runtime, which exceeds the 10-minute requirement of the VA Infrastructure Standard for Telecommunications Spaces.

Procure sufficient batteries in these battery cabinets to provide at least 10 minutes of runtime at initial critical load. This will result in a UPS unit that is approximately 83-85” wide, depending on the specific configuration. The total physical space required for each UPS system to support the recommended ultimate design load is estimated to be equivalent to less than four (4) IT equipment cabinets, with 36” front and 0” rear clearance. (Other makes and manufactures of UPS may have rear clearance requirements and not be compatible with all extra-small designs.)

The generic extra-small CSC uses busbar electrical distribution to support the critical load to minimize lifecycle expansion and operation cost and complexity. Power distribution from the UPS systems to the IT equipment rows is via overhead 120/208V busbars as shown in sheets 24 and 25 of the Generic Extra-Small Data Center Design Package. In the ‘square’ layout, each busbar requires the ability to power 20.6kW of critical load on either busbar, per equipment row. In the ‘rectangle’ layout, each busbar requires the ability to power 41.2kW of critical load on either busbar. (A-side and B-side busbars in the ‘square’ layout may be combined depending upon busbar manufacturer offerings, which changes the supported load requirement to 41.2kW.) Assume a 30% phase load imbalance when conducting amperage calculations for busbar sizing.

Distribute 120/208V output from the A/B UPS units to circuit breakers in the UPS distribution cabinets to support each A/B busbar.

Determine an identifying color scheme for the “A” and “B” side distribution elements. Label each circuit, busbar, and busbar tap (receptacle element) in accordance with the requirements of the Infrastructure Standard for Telecommunications Spaces.

There are ten (10) total IT equipment enclosures in the generic extra-small CSC layout. In the busbar distribution model each individual Standard Density enclosure receives a single 3-phase 20A busbar tap outfitted with a single L21-20R receptacle from each A/B redundant bus as shown in the diagram below. This will provide power to the entire MCR (ultimate design) in the standard configuration defined in the Infrastructure Standard for Telecommunications Spaces.

(There are no High Density enclosures specified in the generic extra-small CSC design.)

Greenfield Design Narrative – Extra-Small Data Center for MCRs and other CSCs | 6

Provide and install twenty (20) standardized 3-phase Rack PDUs per sheet 26 of the Generic Extra-Small Data Center Design Package. Each Standard Density IT equipment enclosure receives a mirrored set of two (2) Rack PDUs. Additional detail on these pieces of equipment is shown in the diagram below.

Provide and install A/B color-coded IEC-320 power cords for installed IT equipment to connect to the standard 3-phase Rack PDUs. The use of IEC-320 C13/14 and C19/20 power cords allows auto-ranging PSUs to automatically pull 208V power, which can decrease the power consumption of the equipment in the range of 2-3%. The A side power cords shall be black and the B side a distinctly different color.

A complete grounding system is required for the generic extra-small CSC. Ensure that Main Distribution Area (MDA) & Horizontal Distribution Area (HDA) network racks in the CSC each receive a Rack Bonding Busbar (RBB) to be mounted in the top rear 1RU. Typical bonding connections that comply with TIA-607-C are shown on sheet 27 of the Generic Extra-Small Data Center Design Package.

Greenfield Design Narrative – Extra-Small Data Center for MCRs and other CSCs | 7

3.3 Mechanical

Any modifications to the generic design will require the design team to conduct additional Computational Fluid Dynamics (CFD) analysis to show that the new design functions as intended at all design loading and contingency conditions. Contact the Data Center & Infrastructure Engineering team for additional information as required.

Heat loads for IT equipment enclosures vary by the function and planned density of the intended equipment. There are six (6) Standard Density (5kW) server cabinets in the generic extra-small CSC design as shown on sheets 4 and 9 of the Generic Extra-Small Data Center Design Package. Each of two (2) HDA network cabinets generates 2.8kW of heat load. Each of the two (2) MDA network cabinets each generate 2.8kW of heat load. The total critical load heat generation at ultimate design load is 41.2kW.

The estimated total heat generation of the generic extra-small CSC at ultimate design is 45.5kW. The installed heat rejection system shall be sized to support 50kW of heat load. N+1 heat rejection system redundancy in the CSC is a requirement for this classification of facility.

Provide and install two (2) matched downflow Computer Room Air Conditioner (CRAC) units set on 12” floor stands as shown on sheets 4 and 9 of the Generic Extra-Small Data Center Design Package. The placement of these units will require the A/E to select units that have maintenance access requirements that can be met in the available space. Selection of units with front-only access, for example, may be necessary.

Team the units to minimize energy consumption during regular operation, which means operating both units at the minimum fan speeds necessary to supply the required airflow for the critical load except in contingency conditions.

The generic extra-small CSC is designed to operate providing cooling supply air to the room at 72°F from the CRAC units, with a 20°F T. This provides a near-optimal environmental condition (77°F) for the UPS batteries to maximize their life.

Approximately 160CFM per critical load kW is necessary at this T level; therefore, at the ultimate operating load (41.2kW) the system should be set to provide 7,500CFM (3,750CFM per unit in regular conditions and 7,500CFM in N+1 contingency contions). Maintaining a CRAC Supply Air Temperature (SAT) of 72°F allows the CRAC units to supply cooling air to the IT equipment inlets between 72°F and 79°F, which meets the requirements of the Infrastructure Standards for Telecommunications Spaces.

The airflow requirement for the CRAC units, based on the ultimate operating heat load to be rejected, is 8,00CFM. Units modeled in the Generic extra-small CSC CFD Report accompanying the design package are 53.2kW Liebert DS053 units meeting the requirements listed above.

Ensure that controls for the system can specify Supply Air Temperature (SAT) setpoints and fan speed/CFM percentage. Typical systems are based solely on Return Air Temperature (RAT) setpoints, which are not appropriate for this design.

Greenfield Design Narrative – Extra-Small Data Center for MCRs and other CSCs | 8

At ultimate design loading, the facility can be operated with both CRACs operating at 50% fan speed (72°F setpoints), or one unit operating at 100% (75°F setpoint). Due to the fan affinity law, in normal operating conditions at commissioning, it will be more efficient to operate both units together at a lower fan speed each than to operate one unit at higher fan speeds to supply the necessary amount of cooling air. Example: If the critical load at commissioning is 20kW, ~4,150CFM will be required. Operate both units at 2,160CFM each (27%) to minimize operational energy consumption.

Power supplied to the new CRAC units may be either 208V or 480V, at the discretion of the facility based on the availability of generator-backed distribution. Input voltage of 480V is typically preferred for motor efficiency. Ideally, select units with redundant compressors that can be powered from redundant sources, and ideally have each unit powered from multiple generator-backed sources that do not share a common upstream single point of failure (e.g., from different distribution panels).

Server and network cabinets are provided with Vertical Exhaust Ducts (VEDs) elements allowing IT equipment exhaust air to be directly vented to a common overhead return air ducting system.

Install a return air ducting system as shown on sheets 19, 20, and 22 of the Generic Extra-Small Data Center Design Package. The Computational Fluid Dynamics (CFD) model used to analyze this design indicates that the minimum duct size necessary to evacuate the heat in contingency conditions at ultimate design load is 36” width x 30” height, with common return ducts over the CRAC units 100” or 97” (depending upon design selected) width x 30” height. Mount the return air ducting system as high as feasible in the space, but at a minimum height of 9’-0” to provide adequate clearance for electrical and telecommunications infrastructure equipment.

Provide and install blanking panels in every open RU of each IT equipment enclosure (server cabinets, HDA racks, and MDA racks) that does not have other equipment. This will dramatically reduce bypass and recirculation airflow issues in the MCR. An estimated 400RU of blanking panels will be necessary initially.

3.4 Telecommunications

IT Equipment Enclosures Provide six (6) server cabinets and four (4) network cabinets with Vertical Exhaust Ducts (VEDs) meeting the requirements of sheets 28 and 29 of the Generic Extra-Small Data Center Design Package.

Outfit each server cabinet, HDA network rack, and MDA network rack as shown on sheets 16 and 17 of the Generic Extra-Small Data Center Design Package. MDA and HDA network racks will require side panel cutouts and grommets to allow for horizontal distribution between the cabinets. This is not a standard feature and will need to be explicitly called out.

Greenfield Design Narrative – Extra-Small Data Center for MCRs and other CSCs | 9

Cable Tray

Install a ceiling-mounted cable tray system as detailed in sheets 5, 10, and 15 of the Generic Extra-Small Data Center Design Package to allow bidirectional (diverse path) distribution to each equipment cabinet. The minimum size of the main tray is 4”x12” to hold the 144 Cat 6A UTP cables necessary for structured cabling distribution (this size of tray can hold up to 199 standard Cat 6A cables at 50% fill). Telecommunications cable trays will extend to the walls in two places to allow for diverse backbone routing from the MDAs to campus TRs.

Structured Cabling

Install a structured cabling system within the MCR. Utilize OM4 in a mesh between MDAs and the HDAs, Cat 6A copper unshielded twisted pair (UTP) from MDAs to their respective HDAs, and a combination of OM4 fiber and Cat 6A copper UTP from the HDAs to each IT equipment enclosure location. All distribution components, such as the media and the media interfaces, must be component compliant to the same performance specifications (Cat 6A or OM4). Utilize only pre-terminated connections on installed structured cabling elements as shown in the graphic below.

Determine the total cable lengths for fiber and copper connections between the MDAs and respective HDAs (backbone), and between the HDAs and respective terminations in the server cabinet Equipment Distributors (EDs) (horizontal), accounting for both horizontal and vertical distances. Procure and install both the backbone and horizontal structured cabling elements.

Specifications for each element shall be:

Backbone fiber – MPO-MPO (Type A) 12-fiber OM4 fiber backbone between the fiber cassettes in MDA A and HDA B, and between the fiber cassettes in MDA B and HDA A.

Greenfield Design Narrative – Extra-Small Data Center for MCRs and other CSCs | 10

(A total of four (4) 12-strand cables for a total of 24 duplex LC connections are required on each path.)

Horizontal fiber – MPO-MPO (Type A) 12-fiber OM4 fiber backbone between the fiber cassettes in each redundant A/B HDA to the corresponding A/B fiber cassettes in each server cabinet. (A total of four (4) are required in each Standard Density server cabinet.)

Horizontal copper – See media recommendations on sheet 18, plate 3 of the Generic Extra-Small Data Center Design Package.

In IT equipment server cabinets, reserve the top rear 5RU of space for structured cabling landing (the Equipment Distributor, or ED). See sheet 16, plate 2 of the Generic Extra-Small Data Center Design Package. Reserve the top rear 1RU in each network rack for the Rack Bonding Busbar (RBB).

Provide fiber and copper UTP patch equipment as specified above and in the accompanying design package. Necessary equipment will include fiber breakout boxes to land new campus fiber backbone bulk cable, fiber cassettes (MPO rear to LC-LC front), angled Cat 6A UTP patch panels, and horizontal and vertical cable management devices.

See the Generic Extra-Small Data Center Design Package for details and to develop a bill of materials (BOM) for these patch equipment elements.

All installed cabling and equipment will require labeling compliant with the current version of the Infrastructure Standard for Telecommunications Spaces.

Fiber Backbone All campus backbone fiber must be re-run to the CSC location as part of the CSC construction project. The fiber backbone refresh work must be planned and accomplished to be complete and ready to transition to during the commissioning of the CSC.

Field-terminate the new backbone fiber in the CSC in the redundant A/B MDAs and in the supported TRs.

4 Other Telecommunications Infrastructure This design narrative specifically addresses the requirements for the CSC. Other telecommunications infrastructure elements are necessary to support the facility, and are generically described in this section.

4.1 Entrance Rooms

Carrier circuits shall be terminated in Entrance Rooms (ERs) located in the same building as the CSC. Relocate circuits as necessary depending on the situation. Provide two (2) ERs for the building containing the CSC, with one carrier circuit entering each from different pathways. The

Greenfield Design Narrative – Extra-Small Data Center for MCRs and other CSCs | 11

ERs shall be located within 50 feet of the exterior of the building and separated by not less than 20 meters (66 feet).

See sheet 4 of the OIT Design Guide Templates (Appendix B of the Infrastructure Standard for Telecommunications Spaces), for construction and outfitting guidance for these redundant ERs.

Follow guidance for the Network Support Center (NSC) telecommunications space classification in the Infrastructure Standard for Telecommunications Spaces for electrical distribution and environmental control requirements for these spaces.

A grounding system compliant with the requirements of TIA-607-C is required for each ER.

Connect the ERs to the CSC with redundant, diverse path fiber optic cabling (24-strand OM4 multimode and 12-strand OS1 single-mode per path). Provide rack-mounted fiber breakout boxes on each end and field terminate the bulk cable installed.

4.2 Telecommunications Rooms

Fiber backbone from the CSC terminates in Telecommunications Rooms (TRs) distributed around the buildings on campus, which provide horizontal network distribution to end-user Work Area Outlets (WAOs). All campus backbone fiber must be re-run to the CSC location as part of the CSC construction project.

TR construction and outfitting requirements are detailed in sheet 5 of the OIT Design Guide Templates (Appendix B of the Infrastructure Standard for Telecommunications Spaces). Follow guidance for the Network Support Center (NSC) telecommunications space classification in the Infrastructure Standard for Telecommunications Spaces for electrical distribution and environmental control requirements for these spaces.

A grounding system compliant with the requirements of TIA-607-C is required for each TR.

Connect the TRs to the CSC with redundant, diverse path fiber optic cabling.

From the CSC to each TR in the same building, install an “A” path and a “B” path each consisting of 24 strands of OM4 multimode fiber and 12 strands of OS1 single-mode fiber. For campus building TRs following protected interior pathways at a total distance of less than 400m, follow the same guidance.

For TRs in unconnected campus buildings, provide an “A” path and a “B” path each consisting of 24 strands of OS2 single-mode fiber.

Provide rack-mounted fiber breakout boxes on each end and field terminate the bulk cable installed.

4.3 Fiber Backbone

All campus backbone fiber must be re-run to the CSC location as part of the CSC construction project. The fiber backbone refresh work must be planned and accomplished to be complete and ready to transition to during the commissioning of the CSC.

Greenfield Design Narrative – Extra-Small Data Center for MCRs and other CSCs | 12

Develop a backbone distribution plan (logical and physical) to provide diverse pathways for the backbone between the CSC and all campus TRs. Provide updated logical and physical maps of the campus, documenting backbone pathways.

All installed cabling and equipment will require labeling compliant with the current version of the Infrastructure Standard for Telecommunications Spaces.

5 Contact Information OIT Solution Delivery (SD) Data Center & Infrastructure Engineering VAITESEDatacenterEngineering2@va.gov

---Nothing Follows---