SOW Attachment 04 - Chillicothe VAMC Assessment Recommendation 20180803 EDICT Report.pdf

OFFICE OF

INFORMATION

AND TECHNOLOGY

Engineering Assessment & Recommendations Chillicothe VA Medical Center Data Center

August 3, 2018 | EDICT Enterprise Data Center Infrastructure Collaboration Team

Engineering Assessment & Recommendation – Chillicothe VA Medical Center | 1

Table of Contents Table of Contents Figures 1 Background 2 Executive Summary 3 Main Computer Room

3.1 Electrical Distribution System

3.2 Mechanical (Air Conditioning) System

4 Additional Chillicothe IT Support Spaces

4.1 Telecommunications Room Concerns

4.2 Biomed Server Room Concerns

5 Campus Project IT Work 6 Recommendations

6.1 Telecommunication Rooms

Environmental Conditioning Environmental Monitoring Grounding Bus Bars Campus Area Network (CAN) Connectivity Maintenance Condition

6.2 Main Computer Room

Physical Layout Lifecycle Condition Network Systems Electrical Systems Mechanical Systems

6.3 Graphics: MCR Recommendation Phases

Engineering Assessment & Recommendation – Chillicothe VA Medical Center | 2

Figures Figure 1. Main Computer Room (MCR) Current & Future Plan Views

Figure 2. Phase 0 Main Computer Room (MCR) – As-Is Conditions

Figure 3. Phase 0 Main Computer Room (MCR) – As-Is Conditions

Figure 4. Phase 1 Main Computer Room (MCR) – No-Cost Changes

Figure 5. Phase 2A Main Computer Room (MCR) – Low-Cost Changes

Figure 6. Phase 2A Main Computer Room (MCR) – Low-Cost Changes

Figure 7. Phase 2B Main Computer Room (MCR) – Low-Cost Changes

Engineering Assessment & Recommendation – Chillicothe VA Medical Center | 3

1 Background The Enterprise Data Center Infrastructure Collaboration Team (EDICT) conducted an engineering survey of the Chillicothe VA Medical Center (VAMC) computer room spaces during the week of 28 May 2018. The mission for the survey conducted by the EDICT team was to make observations, assessments, and recommendations for how to provide a more stable, sustainable computer room operating environment appropriate for the supported facility.

In addition to the Main Computer Room (MCR) located in OIT building 18, there are multiple smaller computer operating spaces, primarily supporting biomedical engineering, around the hospital campus due to the legacy nature of the facilities and the distributed nature of clinics executing the medical mission.

The Chillicothe campus is planning an interconnectivity upgrade project to enable the deployment of Voice Over IP (VOIP) telephone systems. The condition of the Telecommunications Rooms (TRs) in buildings around campus supporting both administrative and medical missions creates concerns as to whether the campus connectivity project can meet its intended objectives, and whether the campus network system will be sustainable. EDICT will make recommendations that can decrease the performance risks to the future VOIP and sustainability of the VA network on the campus.

The Chillicothe facility is a 24x7 hospital campus, not a Community Based Outpatient Clinic (CBOC) or satellite facility. While there are no surgical beds, the campus has many chronic as well as acute clinics and services, including mental health, a community living center, domiciliary, Vocational Rehabilitation and Education (VR&E), acute psych, drug addiction, gastrointestinal (GI), sleep, and cardio labs, and similar.

The Facility Maintenance Service (FMS) organization supporting the campus is a large organization with many internal shops. There is a significant amount of construction ongoing and planned, resulting in many competing priorities for effort and funding. Historically there have not been a lot of direct communications between OIT and FMS/hospital leadership; the local OIT End User Operations (EUO) staff reported that the situation had improved in the past year, but that there was a lot of damage to repair to bring the organizations into what we would consider the necessary cooperative, mutually supportive relationship.

The Entrance Facility (EF) for the campus is located in the OIT building. There is a single Local Exchange Carrier (LEC) central office (Horizon), so the campus Wide Area Network (WAN) connections to the VA network do not have the level of redundancy that having dual circuit providers would indicate. There is not a fiber ring around the campus; instead, there are point-to-point connections between buildings, in general. This type of campus Local Area Network (LAN) is not ideal for connectivity to support modern IT operations, including advanced telephony systems.

Engineering Assessment & Recommendation – Chillicothe VA Medical Center | 4

2 Executive Summary The Enterprise Data Center Infrastructure Collaboration Team (EDICT) was requested to make an engineering assessment and provide recommendations on to identify how to best resolve deficiencies and capability gaps in how OIT’s infrastructure supports the Chillicothe medical center campus. The legacy nature of the campus and existing infrastructure present challenges to the sustainable provision of IT services supporting VHA operations. The recommendations presented are intended to provide guidance and a path towards ensuring that IT’s identified needs are satisfactorily addressed, so that the support mission OIT provides to the campus will continue to be successful.

OIT’s functions support critical VHA medical and medical support missions. EDICT and EUO want to ensure that the local IT and supporting infrastructures are appropriate, reliable, available, and sustainable so that VHA can meet the needs of the Veteran.

The assessment will identify existing conditions, unbalanced conditions, and operational risks that can be addressed through implementation of a phased gap closure program by OIT and VHA. Recommendations will address critical operating risk mitigations, as well as offer suggestions on how to address service provision and efficiency optimization topics for the campus Main Computer Room (MCR) and the Telecommunication Rooms (TRs) supporting the end users around the campus.

3 Main Computer Room The Main Computer Room (MCR) for the Chillicothe VAMC campus is located on the second floor of building 18. It is a dedicated space that was retrofitted from other functions specifically to be a 1990s-era computer room. There are currently 16 IT equipment racks and cabinets, relatively lightly loaded (average load 1.65kW), that support primarily OIT systems. The site has installed a minimalistic overhead cable tray system extending to the primary IT equipment cabinet locations, leaving the 12” underfloor plenum clear for power and cooling air distribution.

Engineering Assessment & Recommendation – Chillicothe VA Medical Center | 5

The suspended ceiling of the room only extends approximately a foot above the tops of the IT equipment cabinets, and the ceiling-mounted equipment (lights, smoke detectors, sprinkler heads) is not arranged based on an IT equipment layout. There are large windows on each of the three exterior walls with simple metal horizontal blinds.

3.1 Electrical Distribution System

Power distribution to the MCR is on a dedicated system. A single utility feed, through a campus generator plant, supports the OIT building.

The MCR is supported by a new dedicated generator, which provides power to the entire OIT building, not just the MCR. The campus reportedly also has a campus-wide generator capability, which we did not have an opportunity to investigate in detail. FMS related that the MCR generator had a 4-day supply of fuel at the unit (local tank capability), with multiple resupply vendors available and a 60,000 gallon campus fuel storage capability for backup. The OIT generator is estimated to be a 250kVA unit; it has a 600 gallon fuel tank, which was half full at the time of the survey.

A single 45kW Eaton 9390 UPS installed in December 2005 supports the critical load in the MCR. The system does have an external maintenance bypass capability. The UPS with integral battery cabinets, generator switchgear, utility mechanical distribution panel, and UPS distribution panel are all located in a room in the basement of the OIT building. The room is open to the outside through a louvered exhaust fan duct, meaning that the equipment operates at roughly the same temperature and humidity as the air outside the building. There are two small split system air conditioners in the space, but they cannot effectively cool the heat sink of the outside air.

Current loading on the UPS was measured at 22.8kW (25.2kVA, 0.90 PF) at 208V line-to-line.

Amperage draws are not well balanced on the output side, being 74.6, 49.2, and 85.8 amps respectively. Note that despite the light loading of the space, the single UPS is currently approximately 50% committed.

The site reportedly does have a UPS maintenance contract, which includes full battery replacement every 3 years (last replacement date 1/26/15). The operating environmental conditions for the UPS are not ideal, so this level of services may be more reasonable than it seems at first blush; typically, batteries would be monitored and replaced on a 7-10 year cycle, except when monitoring indicated individual cells needing replacement.

We relayed concerns that the maintenance contract was likely not providing effective services based on the condition of the air filter for the UPS system (right).

Engineering Assessment & Recommendation – Chillicothe VA Medical Center | 6

FMS opened the electrical panels supporting the MCR for us, allowing us to take readings. The electrical distribution is not well documented onsite; panel schedules exist and reference floor locations, but they do not appear to be completely accurate or represent the distribution in an organized manner. It appears that the UPS feeds panel B, which in turn feeds subpanel A, which in turn feeds subpanel C. Subpanel C does not appear to have any loads on it at the current time. The total load on these panels supporting the MCR critical load was measured at approximately 24kW, roughly aligning with the UPS output readings. Detailed circuit loading information was collected via clamp-on ammeter for additional analysis.

As there is only a single source of power to the MCR from these panels, fed from a single UPS, the site does not have redundant electrical distribution paths. We believe, but could not confirm due to the lack of electrical documentation, that the A/B feeds at each rack are all fed from the same panels. Any semblance of electrical redundancy by having dual rack PDUs appears to be illusionary.

The existing single UPS is undersized for the physical space and potential growth of the local IT mission, and the power support architecture is not compliant with the TIA-942 Rating 3 requirement for the Campus Support Center classification of the facility. Based on physical size and layout, the MCR could eventually be populated with approximately 30 racks of equipment, the UPS systems (sized for ultimate design load) should be sized at around 150kW. (Note that this is not a recommendation, as the electrical distribution and air conditioning capacities must be matched). Matching the UPS system to the planned future 70kW N+1 air conditioning capacity would allow the installation of 14 each 5kW (full capacity) racks. EDICT believes that this will satisfy foreseeable local IT mission needs, including consolidation of all campus IT equipment to the MCR, for the 10-year time horizon.

EDICT will recommend the installation of a second UPS system at the 70kW level, replacement of the existing 45kW system with a (matched) 70kW system, as well as replacement of the power distribution between the existing UPS-backed Main Distribution Panel (MDP) and the workloads in the MCR using a modern, metered and manageable Power Distribution Unit (PDU) system. This electrical work should be programmed for execution by or prior to EOFY2022.

3.2 Mechanical (Air Conditioning) System

The MCR is cooled by two 10-ton (125,000BTUh) Liebert DH125 air-cooled DX CRAC units located in an adjacent mechanical space with an open underfloor supply area and through-wall vented returns. Cooling air is distributed through an approximately 12” height access floor, and supplied to IT equipment cabinets via 6”x18” grates installed in cutouts in the solid floor tiles (rather than using standard 25% open perforated tiles). The underfloor is only used for cooling air and power distribution, and therefore presents very few obstacles to air distribution and pressurization.

The current IT equipment load in the MCR (~24kW) equates to 6.8 tons of air conditioning.

There is adequate cooling in both the current and planned future environmental conditioning states for the space.

Engineering Assessment & Recommendation – Chillicothe VA Medical Center | 7

There are two additional Stulz 5-ton air-cooled DX CRAC units designated for installation in the MCR, but they are not currently operational. Once they are installed, the computer room will have an acceptable level of air conditioning redundancy. The two Stulz CRAC units sitting uninstalled and boxed in the MCR are there because the installation contractor defaulted on their installation contract. The gaseous fire suppression system installed on the same contract cannot be used until these CRAC units are installed. The site estimates that there is another 6 months before the defaulted contract will be fully closed out; at that point FMS will conduct the installation of the units in-house.

A survey of the exterior of the building indicated that the penetrations from the outside into the subfloor area of the MCR for the Stulz condenser piping were not sealed, and conditioned air was blowing directly to the outside. We saw the same thing with a basement window/vent directly from the telco room; both of these are major energy efficiency issues that should be remedied.

The current cooling capability of the MCR is 10 tons (35kW) at N+1 redundancy. Once additional redundant CRAC units are installed, the cooling capability of the space will be approximately 20 tons (70kW) at N+1 redundancy (assuming a potential failure of one of the larger 10-ton units). This is significantly greater capacity than the existing single non-redundant 45kW UPS can support.

4 Additional Chillicothe IT Support Spaces The EDICT team visited a number of spaces in the OIT building and in other facilities while we were on station, to obtain perspective on what could be considered representative of the IT support spaces, as well as to see some of the areas where existing situations put the ability of OIT to provide IT services to the campus at risk.

Engineering Assessment & Recommendation – Chillicothe VA Medical Center | 8

• B18 Basement room O12: We looked at a telecom room below the computer room in the main OIT building. This building is located at the bottom of a gradual hill, so water infiltrates when there is significant precipitation. This space has had historical water infiltration, mushroom, and mold problems, but is considered a critical facility as it acts as the Telephone Equipment Room (TER) for the campus. There does not seem to be any effective manner to remediate these issues in the legacy building.

• B18 Room O14: The UPS and generator ATS room. This space is described in the electrical distribution section above.

• B212 Room 15, in a support services office. This Telecommunications Room (TR) is a shallow closet (shown on the left) with two 3850 switches; the temperature in the closet was 96F. There is no ventilation or air conditioning in this TR.

Shallow closets were generally used for passive cross-connects on telephone systems, and are not recommended for the type of active equipment used in the VA network. Risk mitigation solutions are generally all that can be offered due to the existing cable plants. Shallow closets are not generally suitable for active IT equipment because of their depth; there is insufficient space to mount the switches, which are shown being supported by tied off pieces of copper cabling rather than by a rack mounting assembly.

• B212 Room 125. Another TR. The general condition of this space was better than the others that we observed on campus, and the room had a dedicated split system air conditioner installed above the door. Achieving an environmental support level similar to this space for other TRs could be considered a reasonable and realistic goal for the campus.

• B212 Room 114. Another TR, located in an old shower locker room (still attached to the shower). This room has no ventilation or cooling, high humidity due to its south-facing location and old windows, and is in very poor physical condition. The temperature in this space, recorded by the NetBotz security camera, was 85F. We were concerned about the maintenance condition of this space; the conditions were not conducive to the long-term operation of IT equipment.

Engineering Assessment & Recommendation – Chillicothe VA Medical Center | 9

• B28 Canteen TR. This was another typical shallow TR, located behind the storeroom of the store. The temperature in this space was only 83F; see picture of the floor in this space to discern the maintenance condition of the typical TR.

• B28 Room 201 TR. A converted office upstairs from the canteen store has a disconnected window AC, a wall-mounted fan with its motor burnt out (an overheating and fire risk to the entire building). FMS pulled the fan from the power source, but has not addressed environmental concerns with the space or the equipment in it.

• B27 Room 22G, Biomed cardiology server room, with approximately ½ rack of Accelera Muse and EMG servers. The space is served by a window-mounted air conditioner set at a 63F supply temperature. We identified a wifi temperature and humidity room sensor in this space, which we were told was monitored by FMS. Installation of this type of sensor for gross environmental condition monitoring is a good idea, and should be expanded upon in other IT support spaces on campus.

• B27 Room 14A, Biomed telemetry server room. The rack of IT equipment in this space may be considered severable. This space was serviced by a split-system air conditioner.

4.1 Telecommunications Room Concerns

In general, EDICT has concerns about the environmental and maintenance conditions in the TR spaces that support campus IT operations. While TRs are not intended to be highly redundant clean room environments, there are basic operational requirements that will help ensure that OIT is able to provide the level of IT and telephone support that an active VA hospital campus requires, in both patient-facing and back office administrative spaces.

While Class B support spaces can be operated up to 95F, the recommendation for a high temperature for active IT equipment including switches is 89F. This threshold has been exceeded already in a majority of the TRs that we surveyed, with several exceeding the 95F threshold for Class B spaces. As noted in section 5 below, we expect the power consumption and resulting heat load in these TRs to double as a result of the VOIP project’s installation of new switches providing Power over Ethernet (PoE) to the new digital phones, which will further increase the temperatures in these spaces.

Engineering Assessment & Recommendation – Chillicothe VA Medical Center | 10

A solution to provide heat rejection for these spaces is clearly necessary in order to ensure that the local area network remains viable for the campus. Since many of the spaces currently used as TRs do not provide adequate physical access to or support for the IT gear, a standardized footprint and requirements to support at least one four-post open IT rack should be developed for campus TRs. Shallow closets, interstitial spaces, and other legacy improvised locations for IT switching equipment should be gradually replaced or retrofitted. It should be noted that the TIA-1179 Healthcare Facility Telecommunications Infrastructure standard addresses IT facility requirements in Health Care facilities; it requires a minimum rectangular floor area of 140 square feet for TRs.

4.2 Biomed Server Room Concerns

The conditions and environmental support in the distributed biomed computer room spaces was better than in the average TR that we had the opportunity to visit.

We discussed the federal government’s Data Center Optimization Initiative (DCOI) briefly with the biomed representatives to ascertain their opinions on the consolidation mandate, which will have VA move severable biomedical server equipment from these small, distributed rooms into the MCR, which is intended to be a shared computer room space to support all campus IT needs (OIT, biomed, FMS, R&D, OIG, police, and similar). They expressed a primary concern of their IT equipment being located far away from the medical point-of-care, and the potential issues of not being able to access it to interface or reboot as necessary, particularly during medical procedures.

Implementation of an internal Service Level Agreement (SLA) between the local OIT End User Operations (EUO) team and biomed might be appropriate for discussion, in order to balance the VHA’s operational needs with VA’s mandates from Congress and OMB. With the MCR located in a different building a quarter mile or farther from hospital buildings, both having staff with 24x7 access to the space and having 8x5 10-minute response remote hands support for more critical support (e.g. server reboot on a remote imaging device server, with a patient on the table) might be considered reasonable operational requirements.

5 Campus Project IT Work EDICT attempts to look at supported field installations as holistically as possible during our engineering assessments. The MCR is not an island, and connectivity to the work area outlet level is within the scope of the team and the OIT infrastructure standards.

A new Voice Over Internet Protocol (VOIP) telephone system is currently being designed for campus-wide deployment. The apparent budget for the system is $5M, which is likely insufficient to meet the project’s needs without severe value engineering, which will not allow the system to meet its intended objectives or lifecycle.

Engineering Assessment & Recommendation – Chillicothe VA Medical Center | 11

Details on the inter-building connectivity were not available to the team during the survey;

anecdotal data indicated that the maintenance holes between facilities are already full or nearly so, but the media installed and its condition was not known to the EUO staff. Abatement of abandoned legacy cabling in the connecting ducts may be possible, and EDICT would encourage the scope of this project to include ensuring appropriate and future-ready primary and backup transport media exist to support campus IT needs.

EDICT will recommend that as part of a campus VOIP deployment significantly more (4x) optic fiber should be pulled between buildings than will be necessary just for this project. The majority of the project costs for cabling installation will be setup and labor; doubling the amount of installed cabling material has a very minor cost implication, and the dark fiber capabilities will be necessary to keep up with future technological increases in data transport requirements.

A VOIP system typically deploys telephone devices that are powered by the copper cabling running from the nearest Telecommunication Room (TR) to the user’s work area outlet using Power over Ethernet (PoE) technology. This requires the switches in the TRs to have PoE distribution capability. 24-port distribution switches observed as currently installed in most TRs are not PoE, and are estimated to have a heat load of 175W each in those TRs; there are two of these in most TRs currently. When PoE is deployed to support VOIP and other PoE devices (access control, security cameras, et cetera) we would expect the heat load of each switch to increase to 350kW – doubling the existing heat load in spaces that are already generally running in the mid-90s Fahrenheit, with no heat rejection systems installed.

EDICT will attempt to provide recommendations that will protect the critical switches that will be delivering both the telephone and network connectivity to end-user locations. As noted in the additional IT spaces section of this assessment, many of the campus TRs do not have appropriate environmental conditioning to ensure reliable operation of the IT equipment located in them. The legacy nature of the buildings on campus generally means that TRs were established in whatever spaces were available, rather than in purpose-built spaces that meet industry best practice recommendations; we recognize these limitations and will offer advice on how to improve those conditions, balancing project cost and effort against the OIT standards developed to maximize system availability and to minimize system lifecycle costs.

6 Recommendations

6.1 Telecommunication Rooms

Essentially, there are a large number of noncompliant shallow rooms and closets used as TRs on the campus. Campus buildings were built long before IT was a consideration, and there has not been an opportunity or effort to provide appropriate IT support spaces in buildings around the campus to date. The EDICT recommendations address how to retrofit and effectively provide power and cooling for TR spaces to provide an appropriate level of support for these small but

Engineering Assessment & Recommendation – Chillicothe VA Medical Center | 12 critically important spaces, which is particularly important as the power density in them will be increasing significantly with the deployment of PoE equipment for the VOIP system.

EDICT needs to make recommendations for how to support sustainable TR operations in non-standard spaces. Mounting 4-post racks along walls is an option; it is not as optimal as the recommended and templated solutions for TRs, but where we face issues with space that cannot be remedied, we will support functional solutions. Installation of telecom grounding bus bars in each TR is recommended to protect the overall system. We are not familiar with the grounding and bonding available (or not available) in the campus buildings; a basic grounding system for each building, if not currently installed, is highly recommended. In discussions with local EUO staff, it was contended that VA 0730 states that installing louvers on the doors to TRs was not allowable. EDICT has reviewed this document to make a determination about the validity of this statement. The TRs cannot be isolated in this manner when there are no other manners in which to provide environmental conditioning for the installed equipment.

Environmental Conditioning

VA Handbook 0730/4, Appendix B, establishes physical security requirements for VA spaces including OIT computer rooms and TRs. Requirement D, Other Room Access Means, addresses the physical security requirements for penetrations in walls and doors: All vents, ducts, and similar openings in excess of 96 square inches (620 cm2) that enter or pass through space shall be protected with either bars or grills. If one dimension of the duct measures less than six inches (150 mm) or duct is less than 96 square inches (620 cm2), bars are not required; however, all ducts must be treated to provide sufficient sound attenuation. If bars are used, they must be ½ inch (12.7 mm) diameter steel welded vertically and horizontally six (6) inches (150 mm) on center; if grills are used, they must be of 9-gauge expanded steel.

OIT’s interpretation of this guidance is that ventilation penetrations in TRs would be acceptable, so long as they meet the requirements stated above. An example ventilation penetration that would be compliant could be 12” tall by 24” wide (288 square inches), with a louver cover on the exterior and expanded steel secured to the interior. A typical 9-gauge ¾” expanded steel has an open area of 68%, providing an open area for the example penetration of 196 square inches.

Engineering Assessment & Recommendation – Chillicothe VA Medical Center | 13

The preferred manner of ensuring appropriate environmental conditions in TRs is to provide them with dedicated cooling equipment, typically a small split system package air conditioning device sized to match the anticipated heat load in the space. Where this is not feasible, an alternative is to provide building air conditioning supply and return ducts in the space;

however, due to zone control of variable air systems and gross control of constant air systems, these will generally not provide the necessary level of heat rejection, requiring augmentation.

A simple cooling augmentation solution for TRs is to provide additional airflow, using fans to draw in cooler air from outside the TR and exhaust air at the top of the space. Commercial solutions for this are available, but less expensive solutions built from easily available components could also be used. The space outside these TRs is generally conditioned for human comfort, which is well within the acceptable supply air temperature range. Exhaust can be sent into the above-ceiling void space to avoid overheating human-occupied spaces in many situations. An example of a commercial solution using fans mounted in the door of an IT equipment closet is shown to the right.

Environmental Monitoring EDICT observed that some TRs (and other IT support spaces) had networked temperature and humidity sensors. In TRs, these tended to be on APC NetBotz camera devices; what those connect to and who monitors them is not known. In existing biomed server spaces, Ekahau wifi sensors were installed that reportedly feed to an FMS Building Management System (BMS).

Both types of solutions are appropriate, if they feed to a system that monitors the local environmental conditions, alarms when conditions in a space exceed a set threshold, and generates a maintenance/repair response that protects the IT equipment supporting the operational mission.

EDICT recommends that the MCR, TER, and all TRs be provided with an environmental monitor connected to the BMS system, establishment of a monitoring and alarm program for these sensors by FMS, and establishment of written response procedures for alarm conditions.

Grounding Bus Bars

EDICT observed that some IT spaces had grounding bus bars, but that others did not. IT equipment needs to be operated connected to an earth ground system in order to allow systems to relieve faults associated with variations in voltage. Installation of grounding bus bars connected to building-level earth ground systems, inspected and tested regularly to ensure compliance with Article 250 of the National Electrical Code (NEC), achieving an impedance to

Engineering Assessment & Recommendation – Chillicothe VA Medical Center | 14 ground of 5 Ohms or less (based on the sensitivity of the equipment operated in the facilities) is recommended.

Campus Area Network (CAN) Connectivity

New fiber optic cabling will need to be run between buildings on campus to support the VOIP implementation project. This is the opportune time to abate older cabling that has outlived its usefulness and/or been abandoned. Consideration should be given to deploying the VOIP distribution optic fiber into an integrated and robust loop architecture instead of simply mirroring the current ad-hoc and inefficient connectivity paths. Instead of installing the bare minimum amount of fiber to meet the VOIP project’s needs, recognize that bandwidth requirements are continuing to increase in all telecommunications mediums, and install four times as much as is required for the VOIP project. This will provide adequate dark fiber for medical systems to transport increasingly large images and videos, and for practitioners to support future telehealth efforts.

Prior to committing to the campus-wide VOIP deployment, an assessment should be made of the installed horizontal media within the buildings to ensure that it can support the necessary PoE class for the proposed telephony solution. End-to-end matching of components and capabilities is essential in a deployment of this type.

Maintenance Condition

EDICT recognizes that there are resource constraints in all operations and maintenance (O&M) organizations, which will lead to suboptimal conditions. However, some of the TR spaces that we visited appeared not to have been cleaned in (literally) years, and having inoperative environmental conditioning equipment puts the facilities as well as the networks that IT is tasked with supporting at risk.

O&M efforts, even at a minimal level with due regard for lifecycle replacement, planned preventative maintenance cycles, and available channels for achieving contingency repairs and replacements, must be planned at the campus level.

6.2 Main Computer Room

Physical Layout EDICT is not recommending major layout changes to the MCR at this time. The existing row orientation can be maintained at the design loading in the facility, although alignment of the rows with the floor tile grid and potential changes to the ceiling-mounted equipment need to be addressed. The recommended master layout for the MCR for the next planning cycle is shown in the figure below; these are the layouts that were used in fluid dynamics simulations discussed in the mechanical section below.

Engineering Assessment & Recommendation – Chillicothe VA Medical Center | 15

The existing positioning and clearance to fire protection equipment is not ideal, but altering these characteristics would be a major undertaking. The site will have to validate with the Authority Having Jurisdiction (AHJ) that the current layout and proposed minor changes do not alter the current approvals.

Leave the suspended ceiling in the MCR in place so that the above-ceiling concrete ceiling does not need to be encapsulated. Typically increasing the in-room height provides better separation of the cold and hot airstreams, and allows a higher delta-T across the CRAC units which directly translates to energy efficiency gains, but in this constrained space we believe that the risks and work do not balance with the potential payoff.

Cable trays and lighting in the room should be modified to match the new IT equipment layout.

See the network recommendations section below for additional guidance. Do not block hot aisles with cable trays; instead, use cable trays over the back half of the racks in each row.

Attempt to place light troffers in both cold and hot aisles so that technicians have sufficient illumination to work by.

Relabel the data center and equipment therein in accordance with enterprise-standard guidelines. These guidelines use a spreadsheet format with alpha characters increasing from left to right and numeric characters increasing from top to bottom, from the first full access floor tile, from the NW corner of the space. Specific implementation instructions for labeling the data center and equipment therein are provided in the administrative standards portion of the VA Enterprise Facility IT Support Infrastructure Standard.

Figure 1. Main Computer Room (MCR) Current & Future Plan Views https://vaww.vashare.oit.va.gov/sites/itops/svcs/sma/BCM/Standards%20Lists/va%20enterprise%20facility%20it%20support%20infrastructure%20standard_final%20approved_5%20feb%2018.pdf

Engineering Assessment & Recommendation – Chillicothe VA Medical Center | 16

Install a contamination control mat at the entrance to the MCR to reduce contamination brought in on workers’ shoes. This will significantly reduce the cleaning requirements in the computer room space, particularly underfloor. There is a maintenance requirement associated with this equipment or consumable component installation that will have to be added to the custodial service work plan. An example of an acceptable low-maintenance product being used in VA Core Data Centers is the Dycem Cleanzone contamination control mat.

Lifecycle Condition Establish a lifecycle replacement plan to program funds and plan project work to ensure that facilities infrastructure equipment supporting the MCR and IT support spaces around the campus (particularly TRs) remain viable. Facilities should program lifecycle equipment replacements based on the expected useful life of equipment such as generators, UPS systems, battery strings, and CRAC units.

Network Systems Both core switches supporting the campus are located in the same IT equipment cabinet. EDICT recommends splitting these so that at a minimum they are located at opposite ends of the Main Distribution Area (MDA) row. Splitting up other Wide Area Network (WAN) equipment, such as routers, accelerators, and firewalls similarly would be a best practice to achieve a level of internal physical redundancy.

A redundant MDA provides the ability to use two independent distribution paths for network connectivity within the MCR. A simple example of dual distribution paths from separated core switches is shown to the right. Dual distribution paths will require additional sections of cable tray system for redundancy.

Install a structured cabling solution that aligns with the requirements of the VA Enterprise Facility IT Support Infrastructure Standard.

Electrical Systems EDICT makes a number of specific recommendations for the electrical distribution systems necessary for the IT facility to meet its intended purpose of supplying continuously available IT services to the hospital customers. These recommendations are intended to allow the local site to achieve compliance with published enterprise infrastructure standards as well as to meet electrical metering requirements that are mandated from the Congressional and OMB levels.

The existing electrical distribution system supporting the MCR is severely constrained in its capacity (45kW), has a single non-redundant UPS system and distribution bus, the distribution path uses legacy distribution panels that severely limit abilities to make changes to distribution or to deploy mandated metering solutions, and these distribution panels are located in the https://vaww.vashare.oit.va.gov/sites/itops/svcs/sma/BCM/Standards%20Lists/va%20enterprise%20facility%20it%20support%20infrastructure%20standard_final%20approved_5%20feb%2018.pdf https://vaww.vashare.oit.va.gov/sites/itops/svcs/sma/BCM/Standards%20Lists/va%20enterprise%20facility%20it%20support%20infrastructure%20standard_final%20approved_5%20feb%2018.pdf https://vaww.vashare.oit.va.gov/sites/itops/svcs/sma/BCM/Standards%20Lists/va%20enterprise%20facility%20it%20support%20infrastructure%20standard_final%20approved_5%20feb%2018.pdf

Engineering Assessment & Recommendation – Chillicothe VA Medical Center | 17 unsecured hallway outside the MCR security perimeter, with simple key locks that are easily bypassed.

Computational fluid dynamics (CFD) models in the following sections do not discuss electrical distribution. Updating the electrical distribution is necessary in order to achieve the balance desired between the electrical and mechanical support systems; doing one without addressing the other will not allow the systems to achieve the objectives intended.

The MCR has a requirement for 2N conditioned power distribution. Based on the design recommended by EDICT, this will require:

• Replacement of the existing (A) 45kW UPS with a 70kW system.

• Addition of a second (B) UPS bus matched to the ultimate design load of the current

MCR (70kW).

Add computer-grade power distribution equipment inside the secured MCR space to create redundant distribution paths from the UPS to the IT equipment cabinet level. If the solution uses 480V UPS output, transform the critical distribution in redundant A/B computer-grade Power Distribution Units (PDUs), and provide underfloor distribution to IT equipment cabinets in power whips in liquid-tight flexible metal conduit (LFMC) rather than the steel flexible conduit currently in use. If UPS systems are designed with 208/120V output, use computer-grade Remote Power Panels (PDUs without the internal transformer, essentially) for A/B distribution. Migrating off of the existing distribution panels (DPs) powered from the existing UPS is critical for system sustainability and to meet ultimate design loading needs.

Specify appropriate metering capabilities for all electrical distribution equipment.

Install metering on circuits for supporting mechanical and electrical equipment sufficient to allow capture and reporting of Power Utilization Effectiveness (PUE). In some circumstances it may be more effective to submeter loads in order to subtract them from an upstream meter reading. Installation of a system to automate capture and aggregation of the data from metering components is strongly recommended, in order to eliminate human data collection and reporting requirements.

Install 3-phase standard power and vertical rack PDUs in IT equipment cabinets in the positions indicated to be retained in long-term and future expansion zones. Uninstall power to the remaining cabinet locations, to be removed from the facility, and abate the installed distribution equipment.

Balance phase distribution. IT equipment loads on the A, B, and C phases need to be balanced to reduce neutral currents, which represent heat loads producing no useful work. Loads should be balanced at the PDU and IT equipment cabinet levels.

Review and update as necessary the UPS support, maintenance, and repair contract to cover the new units.

Engineering Assessment & Recommendation – Chillicothe VA Medical Center | 18

If the existing UPS room space will be used for a replacement UPS, seal the room off from outside air and ensure that appropriate environmental conditioning is available to support the critical distribution equipment.

Mechanical Systems EDICT makes a number of specific recommendations for the mechanical and environmental control systems necessary for the IT facility to meet its intended purpose of supplying continuously available IT services to the hospital customers. These recommendations are explained graphically and in additional detail in the following section where EDICT presents the results of airflow optimization simulations.

• CRAC units supporting the MCR need to have a single controller that can switch between lead and lag units, control temperature and humidity in the space using CRAC unit teaming, and detect system failures to start redundant units.

• Replace all installed air supply grates with standard 25% open perforated tiles. As with other recommendation items, additional guidance on this requirement can be found in the VA Enterprise Facility IT Support Infrastructure Standard.

• Realign IT equipment cabinets with the access floor grid as discussed in the physical layout section above, to optimize the effect of the cooling air on the IT equipment.

• Install blanking panels throughout the data center, in every single rack unit (RU) space that is not filled by IT equipment.

• Ensure that IT equipment exhausts are pointed towards the hot aisle, and not into the cold aisles.

• Unless the opening to the exterior in the UPS room is necessary for a reason that EDICT was unable to identify, seal the opening and apply environmental conditioning to the room to prolong the life and improve the reliability of the installed electrical equipment.

• The new Stultz CRAC units use modern electrically commutated (EC) fans to distribute air, which eliminates the need for variable speed drives (VSDs). The existing computer room CRACs are near the end of their effective service lives. They should be programmed for replacement with speed-controllable replacement equipment and controls that allow them to be teamed with the other CRAC systems in the room.

All changes to the heat rejection system, from equipment selection and fan options to equipment placement, must be modeled and confirmed to be appropriate in Computational Fluid Dynamics (CFD) modeling prior to being procured and implemented.

6.3 Graphics: MCR Recommendation Phases

The graphics shown in this section are intended to allow visualization of the performance characteristics of several alternatives tested using the EDICT Computational Fluid Dynamics https://vaww.vashare.oit.va.gov/sites/itops/svcs/sma/BCM/Standards%20Lists/va%20enterprise%20facility%20it%20support%20infrastructure%20standard_final%20approved_5%20feb%2018.pdf

Engineering Assessment & Recommendation – Chillicothe VA Medical Center | 19

(CFD) tool. Information from these simulations was used to make design recommendation decisions in the preceding sections.

A variety of simulations were conducted for:

• Phase 0: The current (as-is) room,

• Phase 1: No-cost recommendations for changes in the current MCR, including rearranging existing air supply grates to optimize airflow and turning off one CRAC unit, to be used as in reserve,

• Phase 2A: Low-cost recommendations for changes to the MCR, including replacement of current air supply grates with standard 25% open perforated floor tiles and installation/commissioning of the two additional Stulz CRAC units, and

• Phase 2B: Additional low-cost recommendations for changes in the MCR implementing industry best practices, including filling gaps in IT equipment rack rows, aligning rows with floor tiles, installing airflow management devices, and reorienting IT equipment airflows to match the hot-cold aisle configuration.

Phase 3 simulations, including completion of the second IT equipment pod to support additional migrated campus IT workloads and adding aisle containment systems to optimize airflows and operating efficiencies, were not accomplished at this time. Each phase necessarily builds on the work conducted in previous phases, and EDICT will need to ensure that Phase 1 and 2 work has been planned/initiated prior to making further detailed recommendations.

Engineering Assessment & Recommendation – Chillicothe VA Medical Center | 20

Figure 2 shows the MCR in its current operating condition. The current power usage in the room is 22.4kW, the airflow demand is 3,532CFM, and the airflow supply is 11,300CFM (75.2kW) with both CRACs operating. Significant amounts of supply air bypass the IT equipment cabinets and return directly to the CRAC units without performing any useful heat rejection work. This represents an opportunity for significant energy efficiency improvement.

Figure 2. Phase 0 Main Computer Room (MCR) – As-Is Conditions

Engineering Assessment & Recommendation – Chillicothe VA Medical Center | 21

Figure 3 is another view of the MCR in its current operating condition, showing a heat fog indicating general in-room conditions. Even with significantly more cooling air being supplied than the room’s heat load requires, two racks are above the ASHRAE recommended maximum air intake temperature of 80.6°F, and one rack is above the ASHRAE allowable maximum air intake of 89.6°F. Racks with higher inlet temperatures than recommended or allowable are shown with red alarms in Figure 3. Recommended changes will change the airflow in such a manner as to ensure that it is only used to reject heat from the IT equipment, maximizing the usefulness of the energy used in cooling the room.

Figure 3. Phase 0 Main Computer Room (MCR) – As-Is Conditions

Engineering Assessment & Recommendation – Chillicothe VA Medical Center | 22

Figure 4 shows the MCR following simple, no-cost changes that can be made to the computer room today to improve overall conditions and reduce operating costs. After relocating existing air supply grates to maximize airflow at the IT equipment use locations and turning off one of the two redundant CRAC units, the power usage and airflow demand in the room remain the same, but the supply airflow is reduced to 5,650CFM (37.6kW). There is sufficient airflow and heat rejection capacity from a single CRAC unit given the existing IT equipment load.

In Phase 1 work, the air supply grate tiles are rearranged as shown to optimize airflow. One CRAC unit is running, with the other in reserve, at all times. The lead-lag program for these units will initially need to be manually switched and, in the future, automatically. Following these changes, one cabinet (top left) remains above the ASHRAE recommended intake temperature of 80.6°F, but still within the ASHRAE allowable of 89.6°F.

Figure 4. Phase 1 Main Computer Room (MCR) – No-Cost Changes

Engineering Assessment & Recommendation – Chillicothe VA Medical Center | 23

In Figure 5, the MCR operating condition is shown following initial low-cost recommended changes, and modeled with an increased IT equipment loading up from 1.6 to 3.5kW per rack (total load 56kW) which will allow campus consolidation of other IT equipment from around campus.

An investment in replacing the non-standard air supply grates with 25% open perforated floor tiles improves efficiencies, allowing CRAC return temperature setpoints to be raised from 69°F from 73°F as simulated in this model.

One CRAC unit remains off (in reserve). With the air conditioning in an N+1 configuration, the available airflow is 11,050CFM against an 8,831CFM requirement, and the systems are rejecting up to 71.3kW of heat. Note that this near-future state supplies significantly more cooling than the current power distribution system is capable of supplying.

One cabinet (the 3PARS rack) is above the ASHRAE recommended 80.6°F inlet temperature, but within the ASHRAE allowable 89.6°F. The cabinet has a higher inlet temperature because it is installed with the air intake facing the air exhaust from the adjacent row.

Figure 5. Phase 2A Main Computer Room (MCR) – Low-Cost Changes

Engineering Assessment & Recommendation – Chillicothe VA Medical Center | 24

Figure 6 shows another view of the Phase 2A low-cost changes recommended, showing IT equipment cabinet front heat distributions and airflow vectors. Airflow recirculation around the ends of the racks and from hot aisles directly into the air intakes of adjacent rows are clearly obvious.

Figure 6. Phase 2A Main Computer Room (MCR) – Low-Cost Changes

Engineering Assessment & Recommendation – Chillicothe VA Medical Center | 25

Figure 7 shows the optimal condition in the MCR following additional low-cost changes to the air flows in the room.