S06-657-25-104-2022 JC VA Combined Test Report.pdf

Project Test Report October 16-26, 2022

Maintenance, Inspection and Testing –

Electrical Power Distribution System Project

Maintenance, Inspection and Testing –

Electrical Power Distribution System Project

CoverCover

Veterans Affairs Medical Center

John Cochran Division St. Louis, Missouri

Contract No. 36C25222P0290

Laura Ferguson, Contracting Officer, (913)946-1962

CoverCover

Signed by:_________________________________

Jerry M Parks Jr, Project Manager

305 Wendy Hill Road

Savannah, Georgia 31410

(803)760-3138

Maintenance, Inspection, and Testing – Electrical Power Distribution System

Contract No. 36C25222P0290

Veterans Affairs Medical Center - John Cochran Division

St. Louis, Missouri October 16-26, 2022

CD Sleeve Top Combined Test Report, CD-ROM

CD Sleeve Bottom Digital Images (Before & After), DVD-ROM

Section 1 Electrical Testing and Maintenance Summary

Section 2 Repair/Upgrade Cost Projections

Section 3 Publications/Exhibits

Section 4 Electrical Test Records

Section 5 Individual Thermographic Test Records

Section 6 Specialized Test Equipment Listing

Section 7 Testing Reference Materials (Digital Version Only)

Section 8 Degrees, Licenses & Certificates (Digital Version Only)

Table of Contents

1 | P a g e

Electrical Testing & Maintenance Report Summary

General

SDV Services Inc. provided electrical testing and maintenance services at the Veterans Affairs Medical Center in St. Louis – John Cochran division during the period October 16-26, 2022. All services were conducted in accordance with contract specifications, industry standards, NETA/ANSI MTS 2015 guidelines and manufacturer’s recommendations. This report consists primarily of the following components:

Section 1 Electrical Testing and Maintenance Report Summary

Section 2 Repair/Upgrade Cost Projections

Section 2 Publications/Exhibits

Section 3 Electrical Test Records

Section 4 Thermographic Records

Section 5 Individual Thermographic Test Records

Section 6 Specialized Equipment Listing

The types of equipment inspected during this session included:

Medium Voltage Transformers and Switches

Medium Voltage Circuit Breakers and Protective Relays

Low Voltage Main and Distribution Switchboards w/ Circuit Breakers

Motor Control Centers & Motor Starters

Low Voltage Automatic Transfer Switches & Generators

Low Voltage Dry-Type Transformers

Uninterruptible Power Supplies

Panelboards

Thermographic Inspection (Infrared)

In addition to regular testing procedures, infrared scan procedures were used to determine serviceability and reliability of the equipment inspected. The infrared services were conducted using a FLIR T440 thermal imaging camera.

2 | P a g e

While infrared technology can be an invaluable tool in predictive maintenance, it is not without limitations. Reflectivity, background, material emissivity and ambient temperature are among many factors that can influence thermal readings. As a result, reported temperature readings outlined in this report should not be viewed or used in terms of absolute temperature readings.

Instead, the readings should be viewed in terms of trends and relationships with other equipment components or like materials.

Supplemental tests were performed wherever possible and on an as-needed basis. These tests consisted of recording circuit loads and conducting Voltage Drop Tests. Circuit loads were monitored to assist in identifying potential phase imbalances or overload conditions. Voltage Drop tests were conducted to determine potential contact resistance issues. Both tests are extremely valuable in helping to determine potential sources of temperature variances.

The last section of this report contains the Individual Thermographic Report section. This section provides a report on each piece of equipment with a deficiency to include among other things:

Thermal image

Visual image

Thermal readings

Location

Condition and cause

Priority level

Recommended action

Repair Priority Levels

It is important to note the “temperature rise” documented on each page of the report is stated for example, “At least 17°C over reference.” This is stated this way as we know for certain the temperature is at least that value. What we don’t know is the true temperature at the point of the problem as other pieces of equipment or hardware shield us and the camera from that view.

We also know at some point in time the equipment item will fail based on the reported condition but cannot with any certainty predict when that will happen. Therefore, every documented issue is important and should be carefully considered.

Many factors or questions need to be addressed when attempting to establish a repair priority level to resolve documented issues. In random order, some general guidelines or questions you might consider in assigning a repair priority level are:

1. How critical is the piece of equipment?

2. What are the ramifications if it fails? What areas will be affected?

3. Is there a personnel hazard or danger?

4. Can the issue lead to collateral damage of related equipment either upstream, downstream or within the same area?

3 | P a g e

5. What is the cost of resolving the issue using a controlled, scheduled shutdown versus an emergency repair/replacement of the equipment item should the equipment fail unexpectedly?

6. Are resources (personnel, contractors, money, etc.) available to make the repairs?

The thermographer cannot readily address such issues or make such determinations. As a result, repair priority levels need to be determined by the VA personnel with the ability to address such questions or those pertinent to their facilities. Note – answers to questions 1-5 should help to overcome or support obstacles encountered with question 6.

Findings & Recommendations

Outlined below are findings and related recommendations applicable to all buildings tested and inspected during this testing session.

1. Outdated Arc Flash Risk Assessment

Issue

The equipment in this facility contains required arc flash warning labels. However, the equipment was evaluated in 2016. There have been many panel upgrades in building 1 since the last testing session. National Fire Protection Association (NFPA) standard 70E, Electrical Safety in the Workplace, section 130.5 (2) outlined below requires arc flash risk assessments be updated under the following conditions:

a. Major modification or renovation takes place, or

b. At intervals not to exceed 5 years.

NFPA 70E Reference

130.5 (2) Arc Flash Risk Assessment Be updated when a major modification or renovation takes place. It shall be reviewed periodically, at intervals not to exceed 5 years, to account for changes in the electrical distribution system that could affect the results of the arc flash risk assessment.

Recommendation

Conduct an updated, comprehensive shock and arc flash hazard risk assessment to determine potential hazard and incident energy levels. Upon completion, ensure arc flash NFPA-compliant warning labels are posted on all required electrical equipment. Personnel should be familiarized on potentially new hazards to ensure adequate levels of personal protective equipment (PPE) are utilized when servicing such equipment.

All electrical single-line drawings should be updated to incorporate all upgrades and additions to the electrical system. This is necessary to provide maintenance personnel with up-to-date information to ensure proper safeguards are in place while performing maintenance operations. This should be incorporated into the short circuit and selective coordination study project.

4 | P a g e

Lockout/tagout procedures should be revised upon completion of the drawing updates.

Both are necessary to provide maintenance personnel with up-to-date information to ensure proper safeguards are in place while performing maintenance operations.

2. Protective Device Selective Coordination

The circuit breakers in switchboard GDB-3 (outside of building 14) as well as the recently replaced panelboards in building 1 were found with protective device setting at the factory default minimum setting. Additionally, it is unclear if these settings have been selectively coordinated as a part of an electrical study. The lack of coordination and minimum settings can lead to nuisance trips on these circuits due to non-fault related issues. Both phase imbalances and inrush currents during equipment startup are common causes of breakers opening under what appears to be a ground fault condition.

The requirement for coordination of these devices is outlined in National Electric Code (NEC) sections 240, 700.27 and 701.18 outlined below:

NEC Reference.

240.12 Electrical System Coordination.

Where an orderly shutdown is required to minimize the hazard(s) to personnel and equipment, a system of coordination based on the following two conditions shall be permitted:

(1) Coordinated short-circuit protection

(2) Overload indication based on monitoring systems or devices

240.100 Feeders and Branch Circuits.

(A) Location and Type of Protection. Feeder and branch-circuit conductors shall have overcurrent protection in each ungrounded conductor located at the point where the conductor receives its supply or at an alternative location in the circuit when designed under engineering supervision that includes but is not limited to considering the appropriate fault studies and time–current coordination analysis of the protective devices and the conductor damage curves. The overcurrent protection shall be permitted to be provided by either 240.100(A)(1) or (A)(2).

700.27 Coordination.

Emergency system(s) overcurrent devices shall be selectively coordinated with all supply side overcurrent protective devices.

Exception: Selective coordination shall not be required in (1) or (2):

(1) Between transformer primary and secondary overcurrent protective devices, where only one overcurrent protective device or set of overcurrent protective devices exists on the transformer secondary,

(2) Between overcurrent protective devices of the same size (ampere rating) in series.

701.18 Coordination.

Legally required standby system(s) overcurrent devices shall be selectively coordinated with all supply side overcurrent protective devices.

Exception: Selective coordination shall not be required in (1) or (2):

(1) Between transformer primary and secondary overcurrent protective devices, where only one overcurrent protective device or set of overcurrent protective devices exists on the transformer secondary,

(2) Between overcurrent protective devices of the same size (ampere rating) in series.

Upon completion of the study or if already completed, protective device settings should be checked against study specifications and changed as needed. Note – such a study is an inseparable component of the above-mentioned arc flash study.

5 | P a g e

A coordination study to determine necessary protective device settings is an integral part of an arc flash study. A separate study should not be performed separately unless the above-referenced arc flash study is not performed. Upon completion, all protective device settings should be adjusted to reflect values derived in the study. This is important to ensure the facility is safely and properly protected in the event of a fault.

3. Electrical System Condition

Issues

There are or have been recent electrical system upgrades in this facility designed to provide a safer and more reliable environment for the buildings and their occupants. However, there are a few areas containing equipment that is in poor overall condition and has far exceeded its useful industry service life 25 years of age and the 15-year service life outlined in the VA Electrical Design Manual reference below.

VA Electrical Design Manual Reference

1.15.4 AGE AND PHYSICAL CONDITION

(b) The equipment should be capable of remaining in use for a minimum of 15 years of additional life or having 60 percent of remaining life, if not the equipment shall be replaced.

Primary Condition Issues

Those areas and equipment considered to be the primary target areas for replacement are as follows:

a. Boiler Plant – the Boiler Plant equipment is in very poor condition. It is approximately

40 years of age with any improvements coming via repairs. The main switchboard (8EP1), panels 8EL3, 8EL2 and 8EL1 have been penetrated by water or some other liquid source evidenced by the stains as pictured below.

The enclosures are corroding and it appears this water source has penetrated at least some of the circuit breakers in these enclosures.

6 | P a g e

Additionally, panels 8EL1 and 8EL2 have oil or similar substance coating the circuit breakers. It is unclear if this substance is flammable and needs to be addressed immediately. The picture to the right attempts to capture this condition.

b. Medium Voltage Metalclad Substation – numerous issues were noted in this outdoor switchgear of which most issues are related to the condition of the enclosure and lack of conditioned air. Outlined below are documented issues:

a. Exterior is corroding with the potential for water penetration into the enclosure.

b. Enclosure lacks controlled air subjecting the equipment to abnormal ambient temperatures and dirt/debris penetration. Exhaust fans were found to be inoperative. Mold was found under the rubber matting inside the enclosure.

c. The automatic transfer scheme failed to trip the main breaker necessary to allow the alternate main breaker to close in the event of a loss of power for the primary feed to the facility. Tie breaker failed to close in response to either Main breaker opening as called for in the transfer scheme.

d. Both main breakers failed to open on undervoltage conditions. Had to be manually opened. Cause is unclear but most likely due to a control issue.

e. Breakers FA1 and Tie were found in a frozen condition and would not open either manually or electrically. Both breakers were repaired on the spot by SDV personnel.

f. Attempted to open breaker FB3 using the open/close switch. Breaker failed to open and the switch caught fire. Breaker was manually opened and the 30 amp control fuses were replaced by SDV personnel.

It should also be noted the circuit breakers are a 40+ year old technology air circuit breakers in contrast to the newer technology vacuum-based breakers found in rm.

A020. Vacuum style circuit breakers are more reliable and require less maintenance.

Additionally, the protective relays are electro-mechanical and are less reliable than the newer solid-state relays.

c. Building 3 Main Electrical Vault – the main switchboard and ATS-16 were installed directly on the floor of the main electrical vault. This has allowed water to penetrate the enclosures leading to corrosion of the enclosures as pictured below.

7 | P a g e

Supplemental Notes on Equipment Aging

As electrical equipment ages, the associated risk of a failure increases significantly. Such failures usually occur without prior warning and can be potentially catastrophic. Repairing or replacing aged equipment is another issue due to obsolescence or non-availability of parts. In the event of a failure, there is a great possibility parts will not be available or will have significant lead times. As an example, SDV found a defective main switch in a federal building that took 6 months to acquire a replacement part due to technical obsolescence.

This can lead to very costly repairs and extended down time for equipment and related circuits.

Included in section 2 to this report is an article presented by three professional engineers (PEs) specializing in electrical distribution systems – “Renovating Electrical Distribution Systems” by Syed M. Peeran, PhD, PE; Mario Vecchiarello, PE; and Jeff Romeo, PE. The first page of this article references the hazards of aging electrical systems and the unpredictability of equipment exceeding its useful service life. Equipment identified above has far exceeded this service life without upgrade and has this element of unpredictability regardless of past performance or testing.

Additionally, the Publications section of this report contains a Square D publication concerning placing electrical equipment in adverse environments with guidelines for properly protecting such equipment.

Recommendation

Due to the condition, age and obsolescence of the above-referenced systems, a project should be initiated to replace the following electrical system components:

a. Boiler Plant - NEMA standard for “Evaluating Water-Damaged Equipment” indicates this switchboard may be reconditioned but the manufacturer needs to be contacted for further guidance. Due to the age of the switchboard, replacement is the most cost-effective solution. Panels 8EL1, 8EL2 and 8EL3 should be replaced due to liquid penetration and potential oil accumulations on the enclosed circuit breakers.

Additionally, the automatic transfer switch and motor control center should be replaced due to age, condition and technical obsolescence.

b. Medium Voltage Metalclad Substation – replace in its entirety incorporating vacuum-style circuit breakers. The new substation should have an additional shell placed over the metalclad substation. This will provide an additional barrier to potential water and dirt penetration and help to ensure the equipment meets or exceeds its expected service life. Conditioned air should be an essential part of replacing this substation.

c. Building 3 Main Electrical Vault – the main switchboard and ATS-16 need to have the corrosion removed and a rust inhibiting paint applied. The base of the main switchboard should be caulked to prevent water from flowing through the switchboard.

8 | P a g e

Additional Equipment Condition Issues

Other condition-related issues that need to be addressed but are not as critical as those mentioned above are as follows:

a. Building 1, Substation US 8, US-9, US-10, US-11 and US-12 – these substations are over

40 years of age and are obsolete. Any updates have come in response to failures. Even though breakers performed satisfactorily, their age make them highly unpredictable in future performed that can lead to a failure at any point in time.

It should be noted the 4160 volt transformers in these substations are showing their age and insulation resistance values are well below expected results in comparison with measurements in comparable equipment. A contributing factor to insulation degradation is the lack of conditioned air in room CB02. This room is considerably warmer than that of room A020 where conditioned air is in use.

Section 5.2.2 Primary Switchgear and section 5.5.1(f) Electrical Rooms and Closets of the VA Electrical Design Manual referenced below outlines the requirement to provide ventilation or cooling systems to maintain indoor temperatures required for proper operation of equipment.

VA Design Manual

5.2.2 PRIMARY SWITCHGEAR

(a) Primary switchgear for Mission Critical facilities shall be located indoors, in a secure, protected location that complies with the Physical Security Design Manual. The area shall have the necessary ventilation or cooling systems to maintain indoor temperature as required for proper operation of the equipment, as well as access control.

Equipment shall be located above the Base Flood Elevation.

VA Design Manual

5.5.1 GENERAL

(f) Electrical spaces shall have the necessary mechanical ventilation or cooling system to maintain the indoor temperature range required for proper operation of the equipment.

b. Building 6, Substations C & D – these two (2) substations carry the same age and condition issues as the older substations in building 1 noted above. Additionally, several breakers were found to have frozen operating mechanisms or were stuck in their cubicles. Frozen breakers mean they cannot open in response to a fault that can lead to a potentially catastrophic failure and/or fire. A contributing factor is this room has louvers to the outside allowing considerable dirt/debris to freely penetrate the room and the enclosed equipment.

c. Outdoor Sectionalizing Switches – three (3) sectionalizing switches (SW-1, SW-2, SW-3) are used to distribute power to several buildings and substations within the facility.

These enclosures are in fair condition and most issues relate to corrosion due to improper pad sizing and weather seals in and around the enclosure.

9 | P a g e

Sectionalizing switches have troughs along the top of the frame where the doors come to rest. This area is prone to dirt/debris buildups that retain water.

As seen in the picture to the right, the pad is at grade level without any form of a base seal to prevent water, dirt, debris from freely penetrating the enclosure.

This has led to corrosion of the base. If left uncorrected, this will lead to premature failure of the enclosure.

d. Building 7 Transformer – similar conditions to those noted in the sectionalizing switches exists in this transformer. The pad is at grade level, no base seal exists and the door weather stripping is deteriorated. Water freely penetrates this enclosure leaving deposits of mud in the base and has led to corrosion as pictured below:

e. Building 7 Electrical System – the electrical system for this building contains FPE and other obsolete panelboards and circuit breakers well in excess of 40 years of age.

Additional Recommendations

a. Building 1, Substation US 8, US-9, US-10, US-11 and US-12 – a project should be initiated to begin replacing these older substations.

b. Building 6 – same recommendations as those in note a above but should include sealing off the room to outdoor air and installing a conditioned air system. Additionally, consideration should be given to performing annual lubrication, cleaning and exercising.

c. A project needs to be initiated to treat the existing corrosion on the outdoor sectionalizing switches and transformers and minimize its potential going forward as follows:

1) Leaves, dirt and debris need to be removed from the troughs along the top of the door enclosures.

2) Heavy concentrations of corrosion should be removed using a wire brush.

3) Each enclosure should be painted with a rust-inhibiting paint that coverts corrosion and bonds it with the paint to provide a weather proof surface.

4) A layer of silicone caulk should be placed around the base of each enclosure to prevent water from penetrating the enclosure.

10 | P a g e

5) A routine maintenance program should be initiated to incorporate the above steps on an as-needed basis.

d. Building 7 – a project should be initiated to begin replace the entire electrical system in this building.

4. Substation 1A Bus Termination

Testing on October 23, 2022 included distribution panels GDB1-1 and GDB1-2 in building 1, room A020A. While working in this area, a bare bus termination was noted coming through a wall in room A020A. Such bus terminations typically have a cap over the end to prevent accidental contact. This bus is physically connected to a circuit breaker labeled “2000 Amp Buss” located in substation 1A in room A020. Were this circuit breaker to be energized, the exposed bus termination would present a dangerous shock hazard to personnel. Pictured below is the circuit breaker in substation 1A as well as the exposed bus termination.

It should be noted the 2000 Amp Buss circuit breaker is presently locked out preventing the circuit breaker from being energized. Were someone to inadvertently remove the lockout and close the breaker, this termination would become energized exposing personnel to potential contact with the energized termination.

Recommendation

Purchase and install a bus termination section to eliminate a potential safety hazard. This would not require a power outage as the related circuit breaker is not energized but should still be verified that power is not present. Additionally, this could be done by VA electricians to reduce the overall cost.

11 | P a g e

5. Substation US-11 Power Issue

Testing on October 23, 2022 included substation US-11 in building 1 room CB02. Testing started by turning off the 4160-volt load interrupter switch to remove normal power source for this substation. SDV personnel opened the various sections of the switchboard enclosure to perform voltage checks used to verify power had been removed from the substation. While performing these checks, a condition was encountered where voltage was found to still be present in the distribution sections of the substation.

Findings

The process of determining this source of this voltage began by tracing the various electrical connections in this substation. It was determined that multiple power sources were present

– normal power and emergency power via ATS-EQ3 and ATS-CR6.

One normal power source comes from the load interrupter switch mentioned above. This source supplies power to the line side of the ATS-EQ3 and ATS-CR6 circuit breakers within the substation. It does not directly supply power to the two distribution sections located in the substation. Instead, new sets of conductors run via conduit from the secondary side of these two circuit breakers to the normal power side of the ATS-EQ3 and ATS-CR6 automatic transfer switches. New sets of conductors then run via conduit from the secondary or load side of these ATS’ to what is now labeled as the ATS-EQ3 DB (distribution board) and ATS- CR6 DB. This configuration means either normal or emergency power is always present on these two distribution boards.

Concerns

Of primary concern is the two distribution boards are physically located in the substation US-11 enclosure giving the appearance they are connected to the same power source of the substation. There are no warning signs advising maintenance personnel that two voltage sources are present within this substation. The physical layout of the substation makes it difficult to see these additional conductors and how they are connected. Typical safety protocols would lead personnel to perform voltage checks on the secondary side of the substation transformer. There is nothing that would lead them to check for the presence of voltage on the two distribution boards exposing them to dangerous shock and arc flash hazards.

Recommendations

Create warning labels advising maintenance personnel that multiple voltage sources are present – normal power via the US-11 load interrupter switch and the load side of each of the two ATS’. Labels should contain white lettering on a red background used to identify it as part of the emergency power scheme. Affix the warning labels to the front of each of the two distribution boards – ATS-EQ3 DB and ATS-CR6 DB. Develop specific lockout/tagout procedures identifying each of the voltage sources present within the substation. This should include the requirement to isolate the normal power source from the 4160-volt load

12 | P a g e interrupter switch and normal and emergency power sources from both of the ATS’. Brief maintenance personnel on the hazards and necessary procedures to properly and safely work on this substation. Due to the nature of this issue, immediate action should be taken.

Note

Attachment B following this summary contains a drawing depicting the power sources for this substation. Pictures and a brief description of these connections are also included in Attachment B.

6. Overhead Sprinklers

The majority of electrical equipment in this facility has sprinkler heads, related water pipes and drain pipes directly over or within the spray pattern of the electrical equipment as represented by the picture to the right. No protective covers have been installed to prevent water from getting into the electrical equipment nor are physical protective devices installed to protect the actual sprinkler heads from physical damage. Both are required by National Electric Code section 110.26(F)(1)(a) through (F)(1)(d) as outlined below.

NEC Reference 110.26(F)(1)(a) through 110.26(F)(1)(d)

(F) Dedicated Equipment Space. All switchboards, panelboards, distribution boards, and motor control centers shall be located in dedicated spaces and protected from damage.

Exception: Control equipment that by its very nature or because of other rules of the Code must be adjacent to or within sight of its operating machinery shall be permitted in those locations.

(1) Indoor. Indoor installations shall comply with 110.26(F)(1)(a) through (F)(1)(d).

(a) Dedicated Electrical Space. The space equal to the width and depth of the equipment and extending from the floor to a height of 1.8 m (6 ft) above the equipment or to the structural ceiling, whichever is lower, shall be dedicated to the electrical installation. No piping, ducts, leak protection apparatus, or other equipment foreign to the electrical installation shall be located in this zone.

Exception: Suspended ceilings with removable panels shall be permitted within the 1.8-m (6-ft) zone.

(b) Foreign Systems. The area above the dedicated space required by 110.26(F)(1)(a) shall be permitted to contain foreign systems, provided protection is installed to avoid damage to the electrical equipment from condensation, leaks, or breaks in such foreign systems.

(c) Sprinkler Protection. Sprinkler protection shall be permitted for the dedicated space where the piping complies with this section.

(d) Suspended Ceilings. A dropped, suspended, or similar ceiling that does not add strength to the building structure shall not be considered a structural ceiling.

The dedicated electrical space includes the space defined by extending the footprint of the switchboard or panelboard from the floor to a height of 6 ft above the height of the equipment or to the structural ceiling, whichever is lower. This reserved space permits busways, conduits, raceways, and cables to enter the equipment. The dedicated electrical space must be clear of piping, ducts, leak protection apparatus, or equipment foreign to the electrical installation. Plumbing, heating, ventilation, and air-conditioning piping, ducts, and equipment must be installed outside the width and depth zone.

Foreign systems installed directly above the dedicated space reserved for electrical equipment must include protective equipment that ensures that occurrences such as leaks, condensation, and even breaks do not damage the electrical equipment located below.

Sprinkler protection is permitted for the dedicated spaces as long as the sprinkler or other suppression system piping complies with 110.26(F)(1)(c). A dropped, suspended, or similar ceiling is permitted to be located directly in the dedicated space, as are building structural members.

13 | P a g e

The electrical equipment also must be protected from physical damage. Damage can be caused by activities performed near the equipment, such as material handling by personnel or the operation of a forklift or other mobile equipment. See 110.27(B) for other provisions relating to the protection of electrical equipment.

Exhibits 110.21, 110.22, and 110.23 illustrate the two distinct indoor installation spaces required by 110.26(A) and 110.26(F), that is, the working space and the dedicated electrical space.

In Exhibit 110.21, the dedicated electrical space required by 110.26(F) is the space outlined by the width and the depth of the equipment (the footprint) and extending from the floor to 6 ft above the equipment or to the structural ceiling (whichever is lower). The dedicated electrical space is reserved for the installation of electrical equipment and for the installation of conduits, cable trays, and so on, entering or exiting that equipment. The outlined area in front of the electrical equipment in Exhibit 110.21 is the working space required by 110.26(A). Note that sprinkler protection is afforded the entire dedicated electrical space and working space without actually entering either space. Also note that the exhaust duct is not located in or directly above the dedicated electrical space. Although not specifically required to be located here, this duct location may be a cost-effective solution that avoids the substantial physical protection requirements of 110.26(F)(1)(b).

Exhibit 110.21 The two distinct indoor installation spaces required by 110.26(A) and 110.26(F): the working space and the dedicated electrical space.

Perform a facility-wide survey to identify electrical components with water pipes or sprinkler heads placed overhead. Shields need to be installed in all areas where electrical components are within the spray pattern or have overhead water lines to prevent damage to prevent overhead water sources (sprinklers and water lines) from penetrating the enclosures. Such penetration could lead to a potentially catastrophic failure. Protective covers or cages need to be installed on unprotected sprinkler heads to prevent accidental activation with the same potential for catastrophic failure as mentioned above.

14 | P a g e

7. Switchboard Water Leak

Switchboard GSWB1A in the generator switchgear room was found with water leaking out of a conduit junction box overhead the switchboard as pictured to the right. It does not appear the water has penetrated the switchboard and enclosed circuit breakers.

The source of the water leak needs to be determined and eliminated before water penetrates the enclosure requiring evaluation under NEMA Water Damaged Equipment guidelines. Consideration should be given to placing a shield above the switchboard to protect it from any further water leaks.

8. Adverse Environmental Conditions

The outdoor transformer-based circuit breaker for building 3 and the main circuit breaker for US-14 do not have heater strips installed. Abnormal ambient temperatures can lead to hardening of the grease used to lubricate the breaker internal operating mechanisms. This can cause the breaker to fail to open or be delayed in opening. Both can lead to potentially catastrophic failure of the breaker, significant arc flash incident energy levels and fire. The enclosure is being penetrated by dirt/debris due to a lack of filter on the louvers.

Additionally, the enclosure for the US-14 main breaker is a new installation but already starting to corrode. Both issues are pictured below:

The circuit breakers in the transformers and US-14 need to be exercised on at least an annual basis to keep the internal grease softened enough to allow the breakers to open properly in response to a fault. The frequency can be adjusted based upon findings.

Additionally, consideration should be given to installing heater strips in each enclosure.

15 | P a g e

The enclosures need to have filters placed over the louvers or openings in the enclosure.

Additionally, heater strips need to be installed to prevent moisture accumulation and further corrosion of the enclosure.

The above conditions can cause breakers to fail to open or be delayed in opening. This can lead to a fire or a significant arc flash incident.

9. Improper Working Clearance

Switchboards GSWB2, EMDP1A, GDB1A and GDBA1A-2 have been placed directly against the wall represented by the picture to the right.

Additionally, switchboards NDB1A, ATS-3 and distribution panel ECDB1 (main generator room) have also been placed back-to-back. These switchboard placements prevent access to the rear-based components discussed below. National Electric Code section 110.26(A)(1) requires a minimum 30” working clearance outlined below.

110.26(A)(1)(a) Dead-Front Assemblies

(a) Dead-Front Assemblies. Working space shall not be required in the back or sides of assemblies, such as dead-front switchboards or motor control centers, where all connections and all renewable or adjustable parts, such as fuses or switches, are accessible from locations other than the back or sides. Where rear access is required to work on nonelectrical parts on the back of enclosed equipment, a minimum horizontal working space of 762 mm (30 in.) shall be provided.

Exhibit 110.9

Distances measured from the live parts if the live parts are exposed or from the enclosure front if the live parts are enclosed. If any assemblies, such as switchboards or motor-control centers, are accessible from the back and expose live parts, the working clearance dimensions would be required at the rear of the equipment, as illustrated. Note that for Condition 3, where there is an enclosure on opposite sides of the working space, the clearance for only one working space is required.

There are bus connections located in the rear of the enclosure that cannot be serviced from the front as they are blocked by other installed equipment (circuit breakers, support structures, secondary conductors, etc.). Any attempts to service these connections as well as clean the rear portion of the enclosure from the front would require extensive disassembly of circuit breakers and secondary conductors that clearly is not the intent of the manufacturer or the NEC.

There appears to be confusion and misapplication related to installation guidelines of similar “dead-front switchboards” throughout the country based on our findings. The National Electric Code (NEC) defines “dead front” as “without live parts exposed to a person on the operating side of the equipment.” The recently-installed main low voltage switchboard located in the basement of the building is a “dead-front switchboard” based on this NEC definition. Operating personnel are protected from the energized components when covers are in place. However, this definition makes no reference to working space requirements.

NEC section 110.26(A)(1)(a), Dead-Front Assemblies (outlined below) uses this definition

16 | P a g e and applies other qualifying factors or elements in determining working spaces around such assemblies. Those factors or elements are:

1) Connections,

2) Renewable or adjustable parts, such as fuses or switches, and

3) Accessibility from locations other than the back or sides.

Additional clarification regarding work space requirements is provided in Exhibit 110.9 further down in NEC Section 110.26(A)(1)(a). It states, “If any assemblies, such as switchboards or motor-control centers, are accessible from the back and expose live parts, the working clearance dimensions would be required at the rear of the equipment.” The switchboard in question does have covers providing rear access that expose the switchboard bus components. This installation clearly meets these conditions requiring a minimum 30” horizontal working space.

Switchboards GSWB2, EMDP1A, GDB1A and GDBA1A-2 and NDB1A need to be relocated to provide the minimum 30” horizontal working distance as required by NEC Section 110.26(A)(1)(a). This is necessary to facilitate proper maintenance and servicing of the bus and other components of the switchboard to include cleaning. Until this is accomplished, consideration should be given to performing annual infrared inspections of these switchboards.

Ensuring proper working clearances are maintained needs to be a point of emphasis for future switchboard installations.

10. Building 3 Egress Lighting

There are presently no egress lights in the basement of building 3. Should the emergency lighting fail, this area goes totally dark. As this is a safety issue for personnel operating in this space in the event of a lighting failure as outlined in National Electric Code section

700.16 below.

700.16 Emergency Illumination.

Emergency illumination shall include all required means of egress lighting, illuminated exit signs, and all other lights specified as necessary to provide required illumination.

Emergency lighting systems shall be designed and installed so that the failure of any individual lighting element, such as the burning out of a lamp, cannot leave in total darkness any space that requires emergency illumination.

Battery-operated emergency lighting needs to be installed in the basement of building 3 to provide, at a minimum, egress lighting for personnel in the event of a normal power loss.

17 | P a g e

11. Building 3 Panic Bars Missing

The main electrical vault of building 3 does not have panic bars, pressure plates, etc. as required by National Electric Code section 110.26(C)(3) outlined below. Such items are necessary to provide safe egress from these areas under emergency conditions - especially an arc fault condition.

110.26(C)(3). Personnel Doors

Where equipment rated 800 A or more that contains overcurrent devices, switching devices, or control devices is installed and there is a personnel door(s) intended for entrance to and egress from the working space less than 7.6 m (25

ft) from the nearest edge of the working space, the door(s) shall open in the direction of egress and be equipped with listed panic hardware.

The doors of the main electrical vault of building 3 needs to have the standard door handles replaced with panic bars or similar devices that allow these doors to be opened with simple pressure in the event of an emergency as required by National Electric Code.

12. Chiller Plant Emergency Power Scheme

Presently, all chillers in the Chiller Plant are solely on normal power creating potential facility operational issues in the event of a normal power loss.

Consideration should be given to putting at least one chiller on emergency power. This should include related control circuits, pumps, etc.

13. Emergency Equipment Markings

Panel CBDP in the main generator switchgear room is not properly labeled to readily identify them as being part of the emergency power system as required by National Electric Code section 700.9 referenced below. It presently has white lettering on a black background typically associated with the normal power system.

700.9 Wiring, Emergency System.

(A) Identification. All boxes and enclosures (including transfer switches, generators, and power panels) for emergency circuits shall be permanently marked so they will be readily identified as a component of an emergency circuit or system.

18 | P a g e

The panel CBDP needs to be changed to white lettering on a red background to be consistent with other markings in the facility identifying emergency system components.

14. Circuit Identification

Numerous switchboards and panels throughout the facility were found be missing or have incomplete circuit directories. In some cases, circuits have multiple identifications where the circuits have changed over the years as represented by the picture to the right. This makes it is unclear which label is correct. This condition can be dangerous for personnel attempting to isolate a circuit for maintenance. Circuit markings are required by National Electric Code section 110.22(A) outlined below.

110.22(A) Circuit Identification

Each disconnecting means shall be legibly marked to indicate its purpose unless located and arranged so the purpose is evident. The marking shall be of sufficient durability to withstand the environment involved.

A project needs to be initiated to create or update all circuit directories and remove old circuit labels throughout the facility. If this is done by VA personnel, consideration should be given to purchasing a 42-circuit tracing kit readily available for purchase for around $1,200. Such a kit substantially reduces the manhours required to trace circuits versus the effort required when utilizing a single circuit tracer.

It would be it would be extremely beneficial to place a label on each panel, disconnect, etc.

identifying the power source. This will greatly ease the effort needed to properly and safely isolate a circuit for maintenance or troubleshooting.

It is further recommended an electronic version of each directory be created – Microsoft Excel is a good platform for this purpose. Future circuit changes can be readily updated in the program and new panel directories created on the fly and placed inside the panels ensuring directories are properly maintained.

15. Defective Circuit Breakers

The Main breaker in switchboard EMDP1A recognized an overload condition during overcurrent testing. However, the breaker took 1-2 minutes before actually opening.

Afterwards, the breaker was found to trip almost instantaneously on the long-time function.

This condition can lead to nuisance trips.

19 | P a g e

Circuit 10 in panel ELSB1 in building 1 would not reset following exercising of the circuit breakers.

The 52-3L (1B Transformer) circuit breaker in the SG-1 substation was found with a defective open/discharge interlock assembly. This assembly would not allow the breaker to be racked back into its cubicle.

Recommendation

Replace the Main breaker in switchboard EMDP1A in the event the breaker develops nuisance tripping.

The circuit 10 breaker in panel ELSB1 was replaced on the spot. No further action required.

The 52-3L (1B Transformer) circuit breaker open/discharge interlock assembly was adjusted by SDV personnel. No further action required.

16. Lack of Physical Protection

The outdoor sectionalizing switch for buildings 6 and 8 as well as the transformer for building 3 do not have any bollards or similar form of physical protection to protect them from being struck by a vehicle, grounds crew vehicle, etc. It appears the building 6 switch has been hit in the past and has been shifted on its pad as pictured to the right.

Failure to correct this condition can lead to a catastrophic failure of these equipment items.

Recommendation

Bollards or similar form of protection need to be installed to properly protect these equipment items.

17. Transformer Pressure Gauge

The oil-type transformer for building 3 does not have a pressure gauge installed. Pressure gauges are designed to indicate whether the transformer has negative, neutral or positive pressure in the oil tank. Of great importance is where a transformer has negative pressure as obtaining oil samples on these transformers creates the potential for moisture and air to be pulled into the oil compartment. This can lead to “bubbling” with the potential for a flashover and catastrophic failure. Transformer manufacturers recommend placing a positive pressure on the oil compartment by first pulling a vacuum and then placing a

20 | P a g e nitrogen blanket on top to prevent such an occurrence. Typical pressures range from +3 – +7.5 psi.

Enclosed is a publication by Transformer Testing and Repairs, Inc. discussing the importance of transformer gauges and reasons for maintaining a positive pressure in the oil tank. An excerpt from an ABB publication has also been included describing the process for establishing the vacuum and subsequent nitrogen blanket on the oil compartment.

Recommendation

As a part of installing pressure gauges in the T3 transformer (building 3), the oil tanks need to have a positive pressure applied as outlined in manufacturer’s recommendations. This can be done by pulling a vacuum on the tank and filling the space between the oil and the top of the tank with a nitrogen blanket.

18. Transformer Oil Test Results

Test Results

Oil samples were taken of all medium-voltage oil-type transformers. Each sample was tested using dissolved gas analysis (DGA). Laboratory test results have been incorporated into the test records for the medium-voltage transformers located in tab 3 to this report. All results were found to be normal.

No further action required. However, trending current results against future test results can provide early warning signs of transformer deterioration or internal issues.

19. Medium Voltage Dry-Type Transformer Fan System

The medium voltage dry-type transformer in the Chiller Plant area of building 8 has a temperature control panel installed. However, there are no fans installed and power available to the control panel in order to control the operating temperature of the transformer. Excessive heat can lead to a requirement to de-rate the transformer or premature failure.

Recommendation

Consideration should be given to installing an operable fan cooling system in the Chiller Plant transformer to help ensure it meets or exceeds its useful service life.

21 | P a g e

20. 4-Hour ESS

The 4-hour ESS Test was performed on October 22, 2022 in accordance with guidelines established by NFPA 110 and VHA Directive 1028. Below is the timeline for completion of this phase of the project:

a. Outage was initiated at 8:00 AM as scheduled.

b. All generators were reported as starting properly and all transfer switches transferred to the emergency position.

c. No reports of equipment problems were reported by VAMC personnel. However, there were issues related to the automatic transfer scheme of the medium voltage substation documented earlier in this summary.

d. SDV had 4 technicians standing by to help with any equipment problems. No problems were reported by VA personnel.

e. After one hour of the facility being on emergency power only, VA personnel had not requested assistance nor were any equipment problems reported by VA personnel.

f. After two hours of the facility being on emergency power only, VA personnel had not

g. After three hours of the facility being on emergency power only, VA personnel had not

h. After four hours of the facility being on emergency power only, VA personnel had not

i. At this time, 12:00 PM, utility power was restored to the entire facility.

j. All transfer switches transferred back to normal power and all generators went through normal cool down and shutdown.

k. During the entire utility outage, no problems were reported by VA personnel and no requests for assistance were made of SDV personnel.

21. Thermographic Issues

Eleven (11) thermal issues were noted during the infrared scan of this facility. Issues were due to high internal resistance, loose/defective connections or overload conditions. Each issue and related recommendation is summarized by location in Attachments A following this summary and is fully reported on in section 4 to this report. A recap of the types of issues detailed in the summary is as follows:

22 | P a g e

Eight (8) issues were due to high internal resistance. This is common in electrical panels where breakers don’t get exercised. The breaker contacts begin to weld shut increasing resistance while drawing more current that can lead to overheating with the potential for nuisance trips.

Three (3) issues were due to loose connections. Two (2) were considered relatively minor in severity of which one (1) was corrected on the spot by SDV personnel. However, one (1) issue has led to overheating of a secondary conductor.