S02_Atch_Sioux Falls VA Combined (2019-2020) Test Report (757 Pages) - Reduced Size.pdf

Electrical Maintenance & Testing ProjectElectrical Maintenance & Testing Project

CoverCover

Veterans Affairs Medical Center Sioux Falls, South Dakota

Contract No. 36C26319C0078 Travis Johnson, Contracting Officer, (605)333-6822

Project Test Report September 24 – November 24, 2019 in cooperation with

Slocum & Associates

16901 Blakeway Place Mosely, Virginia 23120

P - (512)484-5594

845 Long Branch Circle Sugar Hill, Georgia 30518

P - (678)546-9393, F – (678)828-5903

Signed by:_________________________________ Jerry M Parks Jr, Project Manager

Maintenance, Inspection, and Testing – Electrical Power Distribution System

Contract No. 36C26319C0078

Veterans Affairs Medical Center

Sioux Falls, South Dakota September 24 – November 24, 2019

CD Sleeve Combined Test Report, CD-ROM

Section 1 Electrical Testing and Maintenance Summary

Section 2 Publications/Exhibits

Section 3 Electrical Test Records

Section 4 Individual Thermographic Test Records

Section 5 Specialized Test Equipment Listing

Section 6 Testing Reference Materials (Digital Version Only)

Section 7 Degrees, Licenses & Certificates (Digital Version Only)

Table of Contents

1 | P a g e

Electrical Testing & Maintenance Report Summary

General

Slocum & Associates, in cooperation with SDV Services Inc., provided electrical testing and maintenance services at the Veterans Affairs Medical Center in Sioux Falls, South Dakota during the period September 24 – November 24, 2019. All services were conducted in accordance with contract specifications, industry standards, NFPA 70B and manufacturer’s recommendations.

This report consists of the following:

Section 1 Electrical Testing and Maintenance Report Summary

Section 2 Publications/Exhibits

Section 3 Electrical Test Records

Section 4 Thermographic Records

Section 5 Specialized Equipment Listing

The types of equipment inspected during this session included:

Medium Voltage Transformers

Medium Voltage Circuit Breakers and Protective Relays

Low Voltage Main and Distribution Switchboards w/ Circuit Breakers

Distribution Disconnect Switches

Motor Control Centers

Low Voltage Automatic Transfer Switches

Low Voltage Dry-Type Transformers

Panelboards

Thermographic Inspection (Infrared)

In addition to regular testing procedures, infrared scan procedures to determine serviceability and reliability of the equipment inspected. The infrared services were conducted using a FLIR T440 thermal imaging camera.

While infrared technology can be an invaluable tool in predictive maintenance, it is not without limitations. Reflectivity, background, material emissivity and ambient temperature are among

Slocum & Associates in cooperation with

2 | P a g e many factors that can influence thermal readings. As a result, reported temperature readings outlined in this report should not be viewed or used in terms of absolute temperature readings.

Instead, the readings should be viewed in terms of trends and relationships with other equipment components or like materials.

Supplemental tests were performed wherever possible and on an as-needed basis. These tests consisted of recording circuit loads and conducting Voltage Drop Tests. Circuit loads were monitored to assist in identifying potential phase imbalances or overload conditions. Voltage Drop tests were conducted to determine potential contact resistance issues. Both tests are extremely valuable in helping to determine potential sources of temperature variances.

The last section of this report contains the Individual Thermographic Report section. This section provides a report on each piece of equipment with a deficiency to include among other things:

Thermal image

Visual image

Thermal readings

Location

Condition and cause

Priority level

Recommended action

Repair Priority Levels

It is important to note the “temperature rise” documented on each page of the report is stated for example, “At least 17°C over reference.” This is stated this way as we know for certain the temperature is at least that value. What we don’t know is the true temperature at the point of the problem as other pieces of equipment or hardware shield us and the camera from that view.

We also know at some point in time the equipment item will fail based on the reported condition but cannot with any certainty predict when that will happen. Therefore, every documented issue is important and should be carefully considered.

Many factors or questions need to be addressed when attempting to establish a repair priority level to resolve documented issues. In random order, some general guidelines or questions you might consider in assigning a repair priority level are:

1. How critical is the piece of equipment?

2. What are the ramifications if it fails? What areas will be affected?

3. Is there a personnel hazard or danger?

4. Can the issue lead to collateral damage of related equipment either upstream, downstream or within the same area?

5. What is the cost of resolving the issue using a controlled, scheduled shutdown versus an emergency repair/replacement of the equipment item should the equipment fail unexpectedly?

3 | P a g e

6. Are resources (personnel, contractors, money, etc.) available to make the repairs?

The thermographer cannot readily address such issues or make such determinations. As a result, repair priority levels need to be determined by the VA personnel with the ability to address such questions or those pertinent to their facilities. Note – answers to questions 1-5 should help to overcome or support obstacles encountered with question 6.

Findings & Recommendations

Outlined below are findings and related recommendations applicable to all buildings tested and inspected during this testing session.

1. Equipment Age

Issue

Efforts have been or are being undertaken to upgrade or replace many segments of the electrical system within this facility. There are a couple of areas where equipment is well over twenty-five (25) years of age, is in poor condition and have surpassed the threshold of a 15-year service life outlined in the VA Electrical Design Manual reference below. It’s time to initiate equipment replacement/upgrades necessary to maintain the proper level of operation and protection of the facilities electrical system in the years to come.

VA Electrical Design Manual Reference

1.15.4 AGE AND PHYSICAL CONDITION

(b) The equipment should be capable of remaining in use for a minimum of 15 years of additional life or having 60 percent of remaining life, if not the equipment shall be replaced.

The older panelboards in buildings 1 and 5 are becoming problematic primarily due to hardware issues. The older style panels were designed to have the outer cover attach to the interior deadfront cover. The issue is the deadfront fasteners are increasingly stripping out to the point where the ability of the deadfront and outer covers to remain in place becomes questionable. Another issue is the older Square D panels where flip latches are broken and replacements are no longer available. In fact, four (4) panels should be considered dangerous due to defective hardware and should be replaced as soon as possible. Those panels are:

Building Location Circuit ID

5 Chapel Hall Panel C

5 Hall next to rm. 139 Panel E1N

5 Hall next to rm. 139 Panel F1

28 Rm. 111 Panel B

Building 5 switchboards N1 (Rm. B66) and N2 (Rm. E125) as well as panels MDP, LA and LPA in building 11 are the oldest in the facility and in need of upgrade. Replacement parts are becoming increasingly more difficult to obtain due to age and technical obsolescence.

Equipment of this age and condition raises the question of reliability in regards to the ability

4 | P a g e of the equipment to properly and safely protect the building. Careful consideration should be given to replacing these switchboards.

As electrical equipment ages, the associated risk of a failure increases significantly. Such failures usually occur without prior warning and can be potentially catastrophic. Repairing or replacing aged equipment is another issue due to obsolescence or non-availability of parts. In the event of a failure, there is a great possibility parts will not be available or will have significant lead times. As an example, Slocum found a defective main switch in a federal building that took 6 months to acquire a replacement part due to technical obsolescence. This can lead to very costly repairs and extended down time for equipment and related circuits.

Included in section 2 to this report is an article presented by three professional engineers (PEs) specializing in electrical distribution systems – “Renovating Electrical Distribution Systems” by Syed M. Peeran, PhD, PE; Mario Vecchiarello, PE; and Jeff Romeo, PE. The first page of this article references the hazards of aging electrical systems and the unpredictability of equipment exceeding its useful service life. Equipment identified above has far exceeded this service life without upgrade and has this element of unpredictability regardless of past performance or testing.

Recommendation

A project needs to be initiated to accomplish the following:

a. Conduct a survey of building 1 and 5 panelboards to identify and prioritize those panelboards in need of replacement.

b. Replace the four “dangerous” panelboards referenced above as soon as possible.

c. Initiate a project to upgrade switchboards N1 (Rm. B66) and N2 (Rm. E125) as well as panels MDP, LA and LPA in building 11. Prioritization should be established based on operational needs of the facility.

2. Trip/Fall Hazard

The relatively newer medium voltage substation (bldg. 27) was built in such a manner where the concrete pad extends out considerably from the base of the substation. This pad is the area where personnel work to operate, remove, test and install the circuit breakers in the substation. At times there is a large cart that is used to remove and install the circuit breakers into and out the substation cubicles. These operations place

5 | P a g e personnel at the very edges of this pad creating the potential for an individual to fall or trip over the edges of the pad.

When the cart is in use to transport the breakers, there is the additional risk of the cart going over the edge of the pad creating a significant hazard to personnel due to the weight of the equipment not to mention the potential for damaging a very expensive and critical component of the facility electrical system.

National Electric Code section 110.26 outlined below provides guidance for protecting personnel from such hazards.

NEC Reference

110.26 Spaces About Electrical Equipment.

Sufficient access and working space shall be provided and maintained about all electrical equipment to permit ready and safe operation and maintenance of such equipment.

Key to understanding 110.26 is the division of requirements for spaces about electrical equipment in two separate and distinct categories: working space and dedicated equipment space. The term working space generally applies to the protection of the worker, and dedicated equipment space applies to the space reserved for future access to electrical equipment and to protection of the equipment from intrusion by nonelectrical equipment. The performance requirements for all spaces about electrical equipment are set forth in this section. Storage of material that blocks access or prevents safe work practices must be avoided at all times.

(A) Working Space. Working space for equipment operating at 600 volts, nominal, or less to ground and likely to require examination, adjustment, servicing, or maintenance while energized shall comply with the dimensions of 110.26(A)(1), (A)(2), and (A)(3) or as required or permitted elsewhere in this Code.

Recommendation

Consideration should be given to raising the floor of the building to match the level of the extended pad to eliminate the fall/trip hazard. This would create a “step hazard’ upon entering or exiting the building where the exterior would be at a lower level. However, this can be cured by placing a “step up” or “step down” warning placard on the exterior and interior sides of the entrance doors.

3. Overhead Sprinklers

The majority of electrical equipment in this facility has sprinkler heads and related water pipes directly over or within the spray pattern of the electrical equipment as represented by the picture to the right. No protective covers have been installed to prevent water from getting into the electrical equipment nor are physical protective devices installed to protect the actual sprinkler heads from physical damage. Both are required by National Electric Code section 110.26(F)(1)(a) through (F)(1)(d) as outlined below.

110.26(F)(1)(a) through 110.26(F)(1)(d)

(F) Dedicated Equipment Space. All switchboards, panelboards, distribution boards, and motor control centers shall be located in dedicated spaces and protected from damage.

Exception: Control equipment that by its very nature or because of other rules of the Code must be adjacent to or within sight of its operating machinery shall be permitted in those locations.

(1) Indoor. Indoor installations shall comply with 110.26(F)(1)(a) through (F)(1)(d).

6 | P a g e

(a) Dedicated Electrical Space. The space equal to the width and depth of the equipment and extending from the floor to a height of 1.8 m (6 ft) above the equipment or to the structural ceiling, whichever is lower, shall be dedicated to the electrical installation. No piping, ducts, leak protection apparatus, or other equipment foreign to the electrical installation shall be located in this zone.

Exception: Suspended ceilings with removable panels shall be permitted within the 1.8-m (6-ft) zone.

(b) Foreign Systems. The area above the dedicated space required by 110.26(F)(1)(a) shall be permitted to contain foreign systems, provided protection is installed to avoid damage to the electrical equipment from condensation, leaks, or breaks in such foreign systems.

(c) Sprinkler Protection. Sprinkler protection shall be permitted for the dedicated space where the piping complies with this section.

(d) Suspended Ceilings. A dropped, suspended, or similar ceiling that does not add strength to the building structure shall not be considered a structural ceiling.

The dedicated electrical space includes the space defined by extending the footprint of the switchboard or panelboard from the floor to a height of 6 ft above the height of the equipment or to the structural ceiling, whichever is lower. This reserved space permits busways, conduits, raceways, and cables to enter the equipment. The dedicated electrical space must be clear of piping, ducts, leak protection apparatus, or equipment foreign to the electrical installation. Plumbing, heating, ventilation, and air-conditioning piping, ducts, and equipment must be installed outside the width and depth zone.

Foreign systems installed directly above the dedicated space reserved for electrical equipment must include protective equipment that ensures that occurrences such as leaks, condensation, and even breaks do not damage the electrical equipment located below.

Sprinkler protection is permitted for the dedicated spaces as long as the sprinkler or other suppression system piping complies with 110.26(F)(1)(c). A dropped, suspended, or similar ceiling is permitted to be located directly in the dedicated space, as are building structural members.

The electrical equipment also must be protected from physical damage. Damage can be caused by activities performed near the equipment, such as material handling by personnel or the operation of a forklift or other mobile equipment. See 110.27(B) for other provisions relating to the protection of electrical equipment.

Exhibits 110.21, 110.22, and 110.23 illustrate the two distinct indoor installation spaces required by 110.26(A) and 110.26(F), that is, the working space and the dedicated electrical space.

In Exhibit 110.21, the dedicated electrical space required by 110.26(F) is the space outlined by the width and the depth of the equipment (the footprint) and extending from the floor to 6 ft above the equipment or to the structural ceiling (whichever is lower). The dedicated electrical space is reserved for the installation of electrical equipment and for the installation of conduits, cable trays, and so on, entering or exiting that equipment. The outlined area in front of the electrical equipment in Exhibit 110.21 is the working space required by 110.26(A). Note that sprinkler protection is afforded the entire dedicated electrical space and working space without actually entering either space. Also note that the exhaust duct is not located in or directly above the dedicated electrical space. Although not specifically required to be located here, this duct location may be a cost-effective solution that avoids the substantial physical protection requirements of 110.26(F)(1)(b).

7 | P a g e

Exhibit 110.21 The two distinct indoor installation spaces required by 110.26(A) and 110.26(F): the working space and the dedicated electrical space.

Perform a facility-wide survey to identify electrical components with water pipes or sprinkler heads placed overhead. A project should be initiated to install protective covers over these components as well as cages over the sprinkler heads.

4. Adverse Conditions

The generators for building 50 have louvers that open when the generators are operating pulling outside air into the building. Pulling the outside air in exposes the MSG-1 switchboard to considerable amounts of dirt and debris. The same condition exists in room B66A of building 5 where switchboard G1 and the UPS are exposed to the same conditions.

This exposes this equipment to considerable amounts of dirt/debris. Pictured to the left below is one of the circuit breakers in switchboard G1 and to the right below is the UPS in the same room.

8 | P a g e

The ATS-3 breaker on the left has dust all around the operating handle and trip unit itself. It is unclear if this dirt has penetrated the breaker. In the event the dirt is penetrating the breaker, it could cause the breaker operating mechanism to fail in response to a fault.

Not only is the UPS on the right extremely dirty but it appears there are water stains on the front of the UPS. It should be noted the picture reflects a drip pan and drain being placed above the UPS.

Of significant importance is the fact that the electrical equipment in these two areas lack conditioned air and are exposed to lower than normal ambient operating temperatures.

Experience has shown low ambient temperatures can result in a hardening of grease used to lubricate the internal operating mechanisms. Hardened grease can cause the circuit breakers and switches to either operate slowly or not at all in the event of a fault. A delayed response in opening (seconds versus cycles) can lead to a sustained arc while failing to open can lead to a fire. Both have the potential for a catastrophic failure if left uncorrected.

Section 5.2.2 Primary Switchgear and section 5.5.1(f) Electrical Rooms and Closets of the VA Electrical Design Manual referenced below outlines the requirement to provide ventilation or cooling systems to maintain indoor temperatures required for proper operation of equipment.

VA Design Manual

5.2.2 PRIMARY SWITCHGEAR

(a) Primary switchgear for Mission Critical facilities shall be located indoors, in a secure, protected location that complies with the Physical Security Design Manual. The area shall have the necessary ventilation or cooling systems to maintain indoor temperature as required for proper operation of the equipment, as well as access control. Equipment shall be located above the Base Flood Elevation.

VA Design Manual

5.5.1 GENERAL

(f) Electrical spaces shall have the necessary mechanical ventilation or cooling system to maintain the indoor temperature range required for proper operation of the equipment.

Additionally, enclosed in the Publications section of this report is a Square D publication concerning placing electrical equipment in adverse environments with guidelines for properly protecting such equipment.

Recommendation

As the generator louvers in buildings 5 and room B66A of building 5 are critical to proper generator operation, the most prudent mean of protecting switchboards MSG-1 (bldg. 50) and G1 (bldg. 5) would be to create a room around these switchboards. These room should have conditioned air to prevent dirt/debris penetration as well as exposure to abnormal ambient operating temperatures that can lead to equipment failure. If left uncorrected, the potential exists of a catastrophic failure and possible fire. Additionally, these measures will help to extend the useful service lives of this equipment.

The UPS in room B66A (Bldg. 5) needs to be relocated as it is not designed to operate in such an environment in addition to having water lines/pipes overhead.

9 | P a g e

5. Outdoor Switchgear

Condensation - Some of the medium voltage outdoor switchgear (transformers and junction boxes) was found with condensation on the inside of the enclosures.

Moisture is one of the contributing factors in the development of corona damage with the potential for a catastrophic failure. Corona damage is evidenced by a white, powdery residue or nitrogen deposit signifying low level arcing due to an insulation breakdown in the cables.

If this develops, the residue will continue to build up and track until it reaches a non-insulated component or ground with the potential for a catastrophic failure. The impact to the facility would be significant due to extended utilization of the emergency power system and emergency repairs at a high cost.

None of the outdoor equipment has heater strips installed to minimize the potential for moisture accumulation. Pictured to the right is condensation in the Junction Box 5 enclosure.

Transformer Bases – Several transformers were found with concrete pads at grade level without caulk around the base of the enclosures. Pictured to the right is a transformer reflective of this condition. This lack of a seal enables water, dirt and debris to freely flow through the bottom of the enclosure that will lead to corrosion and a reduced service life.

Those transformers with pads at grade level are:

▪ TR-5-1

▪ TR-5-2

▪ TR-5-4

▪ TR-5-5

▪ TR-5-6

▪ TR-11

▪ TR-16

Rusted Enclosure – Pictured to the right is Junction Box 3 reflecting considerable amounts of corrosion. The corrosion has penetrated all segments of the enclosure to include the door hinges. Operating these doors under such conditions could create a safety hazard in the event the hinges separate from the enclosure.

10 | P a g e

Heater strips should be installed in all transformer and junction boxes.

Transformers with pads at grade level need to have roughly 6” deep troughs created around the concrete pad and filled with crusher run to help with drainage around the bases.

Additionally, all enclosures should have their bases caulked. Both measures will help to minimize the water flowing through the enclosures and serve to extend the service life of the equipment.

Junction Box 3 needs to be replaced due to its condition.

A program needs to be implemented to perform periodic inspections of the outdoor electrical equipment and address corrosion issues proactively or as they appear. A common method to minimize or eliminate corrosion on this equipment is to remove loose rust/corrosion with a wire brush then apply readily available paints that chemically convert the remaining corrosion to become part of the paint. This has proven to be highly effective in keeping such conditions in check and enabling the equipment to reach or exceed its useful service life.

6. Junction Box 5 Corona Issue

During the infrared scan of Junction Box 5, a sizzling noise was detected indicative of arcing.

Upon further inspection, it was noticed one of the bracket bolts for the “A” phase segment of conductors was up against the left side of the enclosure as pictured below.

There is typically a minimum 1” clearance between these two components. The arcing and the presence of moisture (mentioned above) had led to corona damage and subsequent tracking along the enclosure. The potential for catastrophic failure and impact to the facility was mentioned in note 5 above.

11 | P a g e

Action Taken

During the scheduled facility power outage, Slocum personnel performed the following steps to at least provide a temporary repair:

a. The junction box conductors were completely de-energized.

b. The bracket bolt was moved to provide a minimum 1” working clearance from the side of the enclosure.

c. A wire brush was used to remove larger corona deposits and rust.

d. A Scotch-Brite pad and denatured alcohol were used to remove any additional tracking residue.

e. A micro-fiber cloth and denatured alcohol were used to wipe the surface clean.

f. Once dry, an insulating mastic pad was placed over the area where the tracking had occurred.

Pictured below is the result of the post repair process.

Monitor the repair performed on junction box during future inspections for evidence of the return of corona damage. Any further action would be dependent upon findings.

12 | P a g e

7. Transformer Conductor Routing

Numerous medium voltage transformers were found with conductors looped or routed at excessive angles between the conduit and load break elbows. Conductors in the junction boxes were found to be routed up against the concrete well of the junction box. These two conditions can lead to a breakdown of the insulation of the conductors with the potential for corona damage or a cable failure. Examples of the first condition are picture below:

The picture to the left above shows the “B” side loop conductors routed up and around to terminate in the load break elbows. Routing of cables in this manner introduces a “physical” stress on the cable terminations in addition to the already present “voltage” stresses. The picture to the right above shows the one conductor (highlighted by the red arrow) entering the load break elbow at almost a 90 angle placing a tremendous amount of physical stress on the termination. Both conditions can ultimately lead to a breakdown of the termination with the potential for a catastrophic cable failure.

The second condition stems from conductors in the junction boxes being routed against the concrete wells in the enclosure as highlighted by the arrow in the picture of Junction Box 5 below:

13 | P a g e

Conductors under load will vibrate even though not perceptible to the naked eye. Such movement against the rough concrete edge will lead to chafing and a breakdown of the conductor insulation with an eventual catastrophic failure.

Recommendation

For existing terminations where the conductors have been looped or formed at a severe angle, the conductors should be monitored for signs of corona damage. Any further action would be dependent upon findings.

Short of replacing the conductors in Junction Box 5, a temporary repair process similar to that for the arcing mentioned above should be used:

a. De-energize the conductors.

b. Remove the cable ties from each conductor.

c. Clean the conductors with denatured alcohol to remove any tracking or dirt from the conductors with the potential to develop tracking.

d. Inspect the conductor insulation for serviceability.

e. If serviceable, wrap mastic tape/pads or similar insulating material around the edge of the framework and each conductor where it has been in contact with the framework.

f. Cover the mastic tape with Scotch 33+ vinyl electrical tape.

g. Use the Scotch 33+ vinyl electrical tape to secure the conductors to the framework.

h. Monitor for presence of corona deposits.

i. Consider cable replacement if corona damage persists.

The picture below is a transformer repair performed by Slocum personnel at the Iron Mountain, Michigan VAMC using the above procedures. These repairs were performed over 5 years ago and have withstood the test of time.

The success of this repair process depends on the condition of the insulation and the ability to fully work the mastic tape/pads or similar insulating materials around the areas coming in contact with the conductor.

14 | P a g e

Note

Not allowing the use of cable ties for securing medium voltage conductors should be a point of emphasis in future installations. This is a common issue where medium voltage dry-type transformers have replaced previous oil-filled transformers.

Routing and termination of medium voltage conductors is an extremely important facet of any new installation as power is distributed throughout the facility on these very conductors. Proper termination to eliminate the conditions noted above should be a point of emphasis in any future installations. The key is to have as little bend in the conductor between the conduit and load break elbow.

8. Junction Box 2 Conductors

The medium voltage conductors in Junction Box 2 have cable ties used to secure the them to a frame work inside the enclosure as shown in the picture to the right.

Medium voltage conductors have a “soft” exterior insulation prone to damage. The installed cable ties have sharp edges that can lead to chafing of cable insulation resulting in a breakdown of the insulation. As the conductor insulation deteriorates over time, this can lead to corona damage (discussed in note 5 above) with the potential for a catastrophic failure.

The picture below shows what the conductor looks like when the cable ties are removed (picture is from the Iron Mountain, Michigan VAMC). Clearly, you can see where the cable ties cut into and damaged the insulation.

15 | P a g e

A temporary repair process like that proposed for the conductors in Junction Box 5 in note 7 above should be used.

9. Wye-Wye Winding Configuration

All of the medium voltage oil-type transformers have grounded wye-wye configurations of the high and low side coil windings instead of the typical delta-wye configuration required by the section 2.9F of the VA Master Specifications outlined in the VA Electrical Design Manual (excerpt enclosed in the Publications section) outlined below:

VA Electrical Design Manual Reference 2.9F. Primary and Secondary Windings for Three-Phase Transformers:

1. Primary windings shall be delta-connected.

2. Secondary windings shall be wye-connected, except where otherwise indicated on the drawings. Provide isolated neutral bushings for secondary wye-connected transformers.

Wye-wye configured transformers are widely used by power companies but typically not used in other distribution applications. Outlined below is an excerpt from page 44 of ABB’s “Distribution Transformer Guide” describing potential issues with wye-wye configured transformers. An excerpt from pages 42 & 44 of this publication has been included in the Publications section to this report.

ABB Reference

Grounded Wye-Wye Notes

1. Three-phase transformers with this connection may experience stray flux tank heating during certain external system unbalances unless the core configuration utilized provides a return path for the flux.

2. Fundamental and harmonic frequency zero-sequence currents in the secondary lines supplied by the transformer also flow in the primary lines (and primary neutral conductor).

3. Ground relay for the primary system may see load unbalances and ground faults in the secondary system. This must be considered when coordinating overcurrent protective devices.

4. Three-phase transformers with the neutral points of the high voltage and low voltage windings connected together internally and brought out through an HOXO bushing should not be operated with the HOXO bushing ungrounded (floating). To do so can create very high voltages in the secondary systems.

Note 2 describes a scenario where this configuration can generate harmonic currents (created by computer power supplies, VFDs, etc.) that can flow into the primary side of the transformer, primary distribution system and even the utility’s distribution system.

Note 3 describes the potential for load unbalances with this winding configuration. The significance lies in the fact that the unbalance can now be detected by the primary distribution system ground relays. Depending on relay settings and the size of the unbalance, this can lead the relay for the related feeder circuit in the metalclad building to OPEN the feeder breaker resulting in a normal power loss for all circuits on this feeder.

Note – delta-wye configured transformers eliminate such issues.

Below is an excerpt from Eaton Corporation’s white paper, “Harmonics in Your Electrical System,” regarding the potential impact of harmonic loads:

16 | P a g e

Eaton White Paper Excerpt

Most power systems can accommodate a certain level of harmonic currents but will experience problems when harmonics become a significant component of the overall load. As these higher frequency harmonic currents flow through the power system, they can cause communication errors, overheating and hardware damage, such as:

• Overheating of electrical distribution equipment, cables, transformers, standby generators, etc.

• High voltages and circulating currents caused by harmonic resonance

• Equipment malfunctions due to excessive voltage distortion

• Increased internal energy losses in connected equipment, causing component failure and shortened life span

• False tripping of branch circuit breakers

• Metering errors

• Fires in wiring and distribution systems

• Generator failures

• Crest factors and related problems

• Lower system power factor, resulting in penalties on monthly utility bills

A copy of the entire white paper has been provided in section 2 to this report.

A harmonics study should be performed to determine the impact of the utilization of variable frequency drives (VFDs), computer systems, etc. and requirements to mitigate the effects of the related harmonics on the electrical system. Any recommendations would be based on those outlined in the study.

Additionally, the VA Electrical Design Manual requirements for delta-wye configured transformers should be strictly adhered to on all future medium voltage transformer installations.

10. Transformer Oil Test Results

Test Results

Oil samples were taken of all medium-voltage oil-type transformers except for TR-5-1 and TR-5-2 currently scheduled for replacement. Each sample was tested using dissolved gas analysis (DGA). Laboratory test results have been incorporated into the test records for the medium-voltage transformers located in tab 3 to this report. All results were found to be normal with the exception of transformers TR-28 and TR-50. Both transformers showed excessive water content. Water can lead to a degradation of the transformer insulation and reduced service life if left uncorrected. Additionally, TR-50 had a low dielectric strength.

17 | P a g e

Test results from previous testing sessions are unknown. It would seem prudent to resample and test these two transformers in the Spring 2020. Results should be compared to those in this report. If the moisture and low dielectric strength levels remain, consideration should be given to filtering the oil to remove the moisture and allow for a normal transformer service life.

11. Transformer Gauges

A couple of oil-type transformers serviced during this testing session had operating gauge issues as follows:

Building Circuit ID Missing Gauge

Pressure Temp

5 TR-5-4 X

28 TR-28 X X

50 TR-50 X X

Pressure Gauge

Pressure gauges are designed to indicate whether the transformer has negative, neutral or positive pressure in the oil tank. Of great importance is where a transformer has negative pressure as obtaining oil samples on these transformers creates the potential for moisture and air to be pulled into the oil compartment. This can lead to “bubbling” with the potential for a flashover and catastrophic failure. Transformer manufacturers recommend placing a positive pressure on the oil compartment by first pulling a vacuum and then placing a nitrogen blanket on top to prevent such an occurrence. Typical pressures range from +3 – +7.5 psi.

Temperature Gauges

Transformer gauges are designed to identify operating temperature and the highest oil temperature readings since being reset. This is a critical gauge used to identify transformers with oil temperatures either approaching or exceeding the manufacturer’s maximum oil operating temperature. A transformer exceeding its maximum operating temperature can be indicative of an overloaded transformer or potential defect.

Enclosed is a publication by Transformer Testing and Repairs, Inc. discussing the importance of transformer gauges and reasons for maintaining a positive pressure in the oil tank. An ABB publication has also been included describing the process for establishing the vacuum and subsequent nitrogen blanket on the oil compartment.

18 | P a g e

Pressure gauges need to be installed in transformers TR-5-4, TR-28 and TR-50. The oil tanks then need to have a positive pressure applied as outlined in manufacturer’s recommendations. This can be done by pulling a vacuum on the tank and filling the space between the oil and the top of the tank with a transformer-grade nitrogen blanket.

Additionally, transformers TR-28 and TR-50 needs to have a temperature gauge installed.

Pressure, temperature and oil level gauges are required by VA electrical design specifications and need to be a point of emphasis on future installations.

12. Transformer Negative Pressure

Several oil-type transformers were found to have negative or neutral pressure per the enclosed pressure gauge. The implications of negative pressure were detailed in finding 11 above. Those transformers and pressure levels are:

Building Transformer ID Pressure Lvl

1 TR-1 -1

5 TR-5-5 -3

5 TR-5-6 -2

5 TR-5-7 -2

5 TR-5-7 0

11 TR-11 -0.5

16 TR-16 -1

38 TR-38 -1

The above-referenced transformers need to have a positive pressure applied as outlined in manufacturer’s recommendations. This can be done by pulling a vacuum on the tank and filling the space between the oil and the top of the tank with a transformer-grade nitrogen blanket.

13. Circuit Breaker Defects/Issues

A couple of circuit breakers were found to be defective during testing. Those breakers are as follows:

Building Room Swbd/Panel ID Circuit Issue

5 Penthouse P8 Main Failed to trip during overcurrent testing.

11 Generator Rm. MDP MCC Failed to trip during overcurrent testing.

19 | P a g e

Building Room Swbd/Panel ID Circuit Issue

5 Penthouse P8 Main Breaker was replaced with a spare.

11 Generator Rm. MDP MCC Breaker needs to be replaced.

14. Improper Working Clearance

There are numerous instances of electrical closets being used for storage where access to the electrical equipment is blocked. Additionally, there are numerous instances where IT personnel have installed server racks in small electrical closets where, in some cases, access to the panels is completely obstructed as pictured to the right.

This picture is room 228E in building 5 where the panel cannot be worked on due to the position of the server rack.

National Electric Code section 110.26 outlined below provides guidance working spaces around electrical equipment. Specifically referenced the dedication of space fee from intrusion by nonelectrical equipment. Aside from access-related issues, IT equipment is data-related and should not be in the same area as electrical equipment.

110.26 Spaces About Electrical Equipment.

Sufficient access and working space shall be provided and maintained about all electrical equipment to permit ready and safe operation and maintenance of such equipment.

Key to understanding 110.26 is the division of requirements for spaces about electrical equipment in two separate and distinct categories: working space and dedicated equipment space. The term working space generally applies to the protection of the worker, and dedicated equipment space applies to the space reserved for future access to electrical equipment and to protection of the equipment from intrusion by nonelectrical equipment. The performance requirements for all spaces about electrical equipment are set forth in this section. Storage of material that blocks access or prevents safe work practices must be avoided at all times.

(A) Working Space. Working space for equipment operating at 600 volts, nominal, or less to ground and likely to require examination, adjustment, servicing, or maintenance while energized shall comply with the dimensions of 110.26(A)(1), (A)(2), and (A)(3) or as required or permitted elsewhere in this Code.

National Electric Code further requires a minimum 30” working clearance as outlined in code section 110.26(A)(2) below.

110.26(A)(2) Width of Working Space

Width of Working Space. The width of the working space in front of the electrical equipment shall be the width of the equipment or 762 mm (30 in.), whichever is greater. In all cases, the work space shall permit at least a 90 degree opening of equipment doors or hinged panels.

20 | P a g e

Additionally, IT personnel are well within the arc flash approach boundary when working on their equipment and may or may not be properly trained on electrical safety, arc flash hazards as well as possessing required personal protective equipment (PPE) as outlined in NFPA 70E, Electrical Safety in the Workplace.

Compounding this situation is the fact that the room 228E electrical closet door opens into an automatic door as pictured to the right. This is a safety hazard should an arc flash incident occur within this electrical closet where a person’s path to exiting the room is blocked by the automatic door being in the open position.

The first step should be to eliminate electrical closets from being used as a storage area. A survey should be conducted of all electrical closets or rooms containing electrical equipment to ensure access to the equipment is not being blocked.

The IT servers in electrical closets should be relocated to provide dedicated working spaces for electrical equipment as well as for potential safety reasons stated above.

The automatic door issue will take additional thought on how to resolve. A possible solution is to reverse the sequence of the doors so that the door on the side of the electrical closet opens outward instead of inward.

15. ATS Maintenance

At least a couple of automatic transfer switches (ATS’) had the appearance of never having been maintained or serviced. This is extremely important as one of the key inspection elements is to lubricate and exercise the ATS to the extent possible. Many transfer operations fail due to improper lubrication of the operating mechanisms.

Recommendation

Consideration should be given to establishing an annual maintenance program for the ATS’.

This could be coordinated with generator runs to ensure proper operation in transferring loads to and from emergency power.

16. Transformer QH-ADP

Transformer QH-ADP (Rm. 81) was found to be extremely noisy and appears to have loose blocking or bracing of the coils. There is also the possibility of harmonic loads contributing to the noise for which this transformer is not designed to handle.

21 | P a g e

This transformer should be replaced with one capable of handling harmonic loads.

17. Ground Fault Issue

The normal power source for ATS-1 is located in Swgr #1 (Bldg. 5, Rm. B66). This breaker was found with the neutral wire disconnected. It was reconnected for testing but tripped the breaker on a ground fault condition. It is unclear if this trip condition is the reason for the ground fault wire being disconnected.

Recommendation

The source of the ground fault trip needs to be determined so the ground fault wire can be reconnected and provide protection as designed.

18. Server Room A/C Units

The power sources for the two A/C units in the building 5 main server room are apparently fed from the same panel DP QL-ADP. This would be considered a “common point of failure” barring any other backup provisions. Should the main breaker fail, it would power down both UPS’ for that room meaning the servers would have to be shutdown.

The same condition exists for the UPS’ in rooms E23 and B66A being fed out of panel DP-

QH2.

Recommendation

Again, barring any other backup provisions, it would seem prudent to segregate the power sources for the two A/C units in each of these areas. That would provide some redundancy provided the A/C units were of a size to keep the servers/UPS going even if at a marginal level.

19. Improperly Marked Emergency Equipment

None of the emergency equipment in this facility is properly designated as emergency equipment as required by National Electric Code section700.9 referenced below.

700.9 Wiring, Emergency System.

(A) Identification. All boxes and enclosures (including transfer switches, generators, and power panels) for emergency circuits shall be permanently marked so they will be readily identified as a component of an emergency circuit or system.

22 | P a g e

All of the emergency equipment in this facility needs to be properly labeled to clearly identify those components as being part of the emergency system. One methodology is to use red labels with white lettering to satisfy this requirement. Some facilities will incorporate the use of colors to distinguish between emergency, life safety, critical and equipment branches.

20. Circuit Identification

Numerous switchboards and panels throughout the facility were found be missing or have incomplete circuit directories. In some cases, circuits have multiple identifications where the circuits have changed over the years as represented by the picture to the right. This makes it is unclear which label is correct. This condition can be dangerous for personnel attempting to isolate a circuit for maintenance. Circuit markings are required by National Electric Code section 110.22(A) outlined below.

110.22(A) Circuit Identification

Each disconnecting means shall be legibly marked to indicate its purpose unless located and arranged so the purpose is evident. The marking shall be of sufficient durability to withstand the environment involved.

A project needs to be initiated to create or update all circuit directories and remove old circuit labels throughout the facility. If this is done by VA personnel, consideration should be given to purchasing a 42-circuit tracing kit readily available for purchase for around $1,200. Such a kit substantially reduces the manhours required to trace circuits versus the effort required when utilizing a single circuit tracer.

It would be it would be extremely beneficial to place a label on each panel, disconnect, etc.

identifying the power source. This will greatly ease the effort needed to properly and safely isolate a circuit for maintenance or troubleshooting in addition to enhancing lockout/tagout procedures.

It is further recommended an electronic version of each directory be created – Microsoft Excel is a good platform for this purpose. Future circuit changes can be readily updated in the program and new panel directories created on the fly and placed inside the panels ensuring directories are properly maintained.

23 | P a g e

21. Circuit Labeling

With the facility being built over time with several additions, there now exist panelboards in different locations with the same circuit identification (i.e., Panel SL, DP HN4, etc.). This can create confusion over which panel is being referenced at any point in time.

Recommendation

To avoid confusion and maintain consistency in developing new panel circuit identification, consideration should be given to developing a standard methodology for creating new labels. It would be highly beneficial if the methodology incorporated some physical characteristics of the hospital. A possible solution would be to start a panel ID with the building number, circuit type, floor number, wing number followed by a sequential number.

Using building 5 as an example, the above suggestion would translate to the following identification system:

Building Floor Wing Circuit Type Sequential No.

5 - G N L 001

Building No. - G - Ground N - North Wing N - Normal 3-Digit No.

1 - 1st Floor S - South Wing E - Emergency

2 - 2nd Floor W - West Wing L - Life Safety

3- 3rd Floor

C - Critical

4 - 4th Floor Q - Equipment

5 - 5th Floor

6 - 6th Floor

P - Penthouse

This example creates a panel ID of 5-GAL001. Breaking this down would mean the panel is in building 5, ground floor, north wing, life safety panel and its panel number 001 (sequentially numbered). This or similar type of system could be applied to any of the buildings within this facility.

This process would have to be managed to ensure new panels were accounted for to avoid duplication. Numbers could be developed ahead of time on new projects. The key is this or a system like it provides consistency going forward and if created properly can put a person in the ballpark of the panel’s physical location without having anything more than the panel

ID.

It would be highly beneficial to at some point re-label all of the existing panels and could be done in conjunction with the circuit tracing recommendation made above. New panel labels could also include the voltage and power source to enhance lockout/tagout procedures.

24 | P a g e

22. Electrical Drawing Updates

The last set of electrical drawings for this facility were produced in July 2016. Since that time, there have been a fair amount of changes to the electrical system that were not documented in a readily available manner. This can create issues for personnel attempting to troubleshoot or resolve issues. Furthermore, it has a direct safety impact in relation to lockout/tagout procedures.

Recommendation

Consideration should be given to enlisting the services of an engineering technician to take “as built” drawings from each project and maintain a current set of electrical drawings for maintenance personnel. Additionally, the updated drawings should be used to trigger an evaluation of lockout/tagout procedures to ensure the highest level of electrical safety.

23. 4-Hour ESS

The 4-hour ESS Test was performed on November 24, 2019 in accordance with guidelines established by NFPA 110 and VHA Directive 1028. Below is the timeline for completion of this phase of the project:

a. Outage was initiated at 8:00 AM as scheduled.

b. All generators were reported as starting…