S06-657-25-104-2022 JB VA Combined Test Report.pdf

Project Test Report June 21 – July 27, 2022

Maintenance, Inspection and Testing –

Electrical Power Distribution System Project

Maintenance, Inspection and Testing –

Electrical Power Distribution System Project

CoverCover

Veterans Affairs Medical Center

Jefferson Barracks Division St. Louis, Missouri

Contract No. 36C25222P0290

Laura Ferguson, Contracting Officer, (913)946-1962

305 Wendy Hill Road

Savannah, Georgia 31410

(803)760-3138

Signed by:_________________________________

Jerry M Parks Jr, Project Manager

Maintenance, Inspection, and Testing – Electrical Power Distribution System

Contract No. 36C255222P0290

Veterans Affairs Medical Center - Jefferson Barracks Division

St. Louis, Missouri June 21 – July 27, 2022

CD Sleeve Front Combined Test Report, CD-ROM

CD Sleeve Back Digital Images (Before & After), DVD-ROM

Section 1 Electrical Testing and Maintenance Summary

Section 3 Repair/Upgrade Cost Projections

Section 3 Publications/Exhibits

Section 4 Electrical Test Records

Section 5 Individual Thermographic Test Records

Section 6 Specialized Equipment Listing

Section 7 Testing Reference Materials – Digital Version Only

Section 8 Degrees, Licenses and Certificates – Digital Version Only

Table of Contents

1 | P a g e

Electrical Testing & Maintenance Report Summary

General

SDV Services Inc. provided electrical testing and maintenance services at the Jefferson Barracks Division of the Veterans Affairs Medical Center in St. Louis, Missouri during the period June 21- July 27, 2022. All services were conducted in accordance with contract specifications, industry standards, NETA/ANSI MTS 2015 guidelines and manufacturer’s recommendations. This report consists of the following:

Section 1 Electrical Testing and Maintenance Report Summary

Section 2 Repair/Upgrade Cost Projections

Section 3 Publications/Exhibits

Section 4 Electrical Test Records

Section 5 Thermographic Records

Section 6 Specialized Equipment Listing

Section 7 Manufacturer’s Reference Material (Digital Version Only)

Section 8 Licenses, Degrees, Certifications and Training (Digital Version Only)

The types of equipment inspected during this session included:

Medium Voltage Transformers and Switches

Medium Voltage Circuit Breakers and Protective Relays

Low Voltage Main and Distribution Switchboards w/ Circuit Breakers

Motor Control Centers & Motor Starters

Low Voltage Automatic Transfer Switches & Generators

Low Voltage Dry-Type Transformers

Uninterruptible Power Supplies

Panelboards

2 | P a g e

Thermographic Inspection (Infrared)

In addition to regular testing procedures, infrared scan procedures were used to determine serviceability and reliability of the equipment inspected. The infrared services were conducted using a FLIR T440 thermal imaging camera.

While infrared technology can be an invaluable tool in predictive maintenance, it is not without limitations. Reflectivity, background, material emissivity and ambient temperature are among many factors that can influence thermal readings. As a result, reported temperature readings outlined in this report should not be viewed or used in terms of absolute temperature readings.

Instead, the readings should be viewed in terms of trends and relationships with other equipment components or like materials.

Supplemental tests were performed wherever possible and on an as-needed basis. These tests consisted of recording circuit loads and conducting Voltage Drop Tests. Circuit loads were monitored to assist in identifying potential phase imbalances or overload conditions. Voltage Drop tests were conducted to determine potential contact resistance issues. Both tests are extremely valuable in helping to determine potential sources of temperature variances.

The last section of this report contains the Individual Thermographic Report section. This section provides a report on each piece of equipment with a deficiency to include among other things:

Thermal image

Visual image

Thermal readings

Location

Condition and cause

Priority level

Recommended action

Repair Priority Levels

It is important to note the “temperature rise” documented on each page of the report is stated for example, “At least 17°C over reference.” This is stated this way as we know for certain the temperature is at least that value. What we don’t know is the true temperature at the point of the problem as other pieces of equipment or hardware shield us and the camera from that view.

We also know at some point in time the equipment item will fail based on the reported condition but cannot with any certainty predict when that will happen. Therefore, every documented issue is important and should be carefully considered.

Many factors or questions need to be addressed when attempting to establish a repair priority level to resolve documented issues. In random order, some general guidelines or questions you might consider in assigning a repair priority level are:

3 | P a g e

1. How critical is the piece of equipment?

2. What are the ramifications if it fails? What areas will be affected?

3. Is there a personnel hazard or danger?

4. Can the issue lead to collateral damage of related equipment either upstream, downstream or within the same area?

5. What is the cost of resolving the issue using a controlled, scheduled shutdown versus an emergency repair/replacement of the equipment item should the equipment fail unexpectedly?

6. Are resources (personnel, contractors, money, etc.) available to make the repairs?

The thermographer cannot readily address such issues or make such determinations. As a result, repair priority levels need to be determined by the VA personnel with the ability to address such questions or those pertinent to their facilities. Note – answers to questions 1-5 should help to overcome or support obstacles encountered with question 6.

Findings & Recommendations

Outlined below are findings and related recommendations applicable to all buildings tested and inspected during this testing session.

1. Outdated Arc Flash Risk Assessment

Issue

The equipment in buildings 1, 51A, 54, 55 and 56 contains required arc flash warning labels.

However, the equipment was evaluated over three (3) different periods of time dating back to 2015, 2016 and 2017. These studies are required to be updated at a minimum of 5 years at referenced below. It should be noted the studies for buildings 54, 55 and 57 were performed prior to them being built. That would indicate the studies were based on design drawings and not “as built” conditions. This creates the potential for errors in the related calculations. Building 57 does not contain any arc flash warning labels.

Additionally, National Fire Protection Association (NFPA) standard 70E, Electrical Safety in the Workplace, section 130.5 (2) outlined below requires arc flash risk assessments be updated under the following conditions:

a. Major modification or renovation takes place, or

b. At intervals not to exceed 5 years.

NFPA 70E Reference

130.5 (2) Arc Flash Risk Assessment Be updated when a major modification or renovation takes place. It shall be reviewed periodically, at intervals not to exceed 5 years, to account for changes in the electrical distribution system that could affect the results of the arc flash risk assessment.

Having multiple completion dates indicates these arc flash hazard analyses were performed in a piecemeal manner. Such limited studies fail to ignore contributing fault current levels at points beyond the localized point. A comprehensive arc flash study is necessary to ensure the system is evaluated as a whole as required by National Fire Protection Association (NFPA) 70E sections 130.3(B)(1) and 130.5, National Electric Code (NEC) section 110.16

4 | P a g e outlined below in addition to OSHA regulation §1910.303. This begins with the lowest point(s) in the electrical system up to the incoming service from the utility company.

Determining contributing faults currents at successively higher levels is critical in ensuring upstream incident energy levels are evaluated and protective devices are properly set to minimize incident energy levels. Typically, a complete and comprehensive study provides more accurate results and can often produce more favorable arc flash boundary levels.

NFPA Reference 130(B)(1) Electrical Hazard Analysis

Appropriate safety-related work practices shall be determined before any person is exposed to the electrical hazards involved by using both shock hazard analysis and arc flash hazard analysis.

NFPA 70E Reference

130.5 Arc Flash Hazard Analysis

Employers must conduct an arc flash hazard analysis and place a label on an external surface of equipment that indicates the degree of the thermal hazard or recommends PPE for protection from the degree of existing thermal hazard. The NEC requires a label that warns employees of a potential thermal hazard. Employers must provide sufficient information for an employee to conduct the necessary hazard analysis. Labels are one element of the hazard analysis.

NEC Reference

110.16 Flash Protection.

Electrical equipment, such as switchboards, panel boards, industrial control panels, meter socket enclosures, and motor control centers, that are in other than dwelling occupancies, and are likely to require examination, adjustment, servicing, or maintenance while energized shall be field marked to warn qualified persons of potential electric arc flash hazards. The marking shall be located so as to be clearly visible to qualified persons before examination, adjustment, servicing, or maintenance of the equipment.

FPN No. 1: NFPA 70E-2004, Standard for Electrical Safety in the Workplace, provides assistance in determining severity of potential exposure, planning safe work practices, and selecting personal protective equipment.

FPN No. 2: ANSI Z535.4-1998, Product Safety Signs and Labels, provides guidelines for the design of safety signs and labels for application to products.

Regardless, an assessment for the majority of equipment in this facility has not been updated to meet the 5-year requirement established by NFPA 70E.

Recommendation

Conduct an updated, comprehensive shock and arc flash hazard risk assessment to determine potential hazard and incident energy levels. Upon completion, ensure arc flash NFPA-compliant warning labels are posted on all required electrical equipment. Personnel should be familiarized on potentially new hazards to ensure adequate levels of personal protective equipment (PPE) are utilized when servicing such equipment.

All electrical single-line drawings should be updated to incorporate all upgrades and additions to the electrical system. This is necessary to provide maintenance personnel with up-to-date information to ensure proper safeguards are in place while performing maintenance operations. This should be incorporated into the short circuit and selective coordination study project.

Lockout/tagout procedures should be revised upon completion of the drawing updates.

Both are necessary to provide maintenance personnel with up-to-date information to ensure proper safeguards are in place while performing maintenance operations.

5 | P a g e

2. Electrical System Condition

Issues

There are or have been recent electrical system upgrades in this facility designed to provide a safer and more reliable environment for the buildings and their occupants. However, there are a few areas containing equipment that is in poor overall condition and has far exceeded its useful industry service life 25 years of age and the 15 year service life outlined in the VA Electrical Design Manual reference below.

VA Electrical Design Manual Reference

1.15.4 AGE AND PHYSICAL CONDITION

(b) The equipment should be capable of remaining in use for a minimum of 15 years of additional life or having 60 percent of remaining life, if not the equipment shall be replaced.

Primary Condition Issues

Those areas and equipment considered to be the primary target areas for replacement are as follows:

a. Building 1 – the primary distribution equipment (medium voltage oil-type transformer, main switchboard, emergency distribution panels and automatic transfer switches) combine to provide both normal and emergency power to this building.

b. Building 3 – in addition to age and obsolescence, the panels in the basement main electrical closet are in poor overall condition due to steam pipes detailed in the next section. These panels supply power to the building without an emergency system in the event of a failure.

c. Building 18 – The 480V and 208V transformers. There is presently an issue with an oil leak in the 480V H3 fuse. Again, these transformers supply primary power to this building.

d. Building 25, Panel EDP – This panel supplying emergency power to the building (via building 1) was manufactured by FPE over 40 years ago and is in poor condition.

e. Building 51A - The main circuit breaker for switchboard MSB-51A switchboard as well as the Zenith and Caterpillar transfer switches (CDP and LSDP). The main circuit breaker for switchboard MSB-51A contains the original Square D MicroLogic Series trip unit.

f. Building 53 – The transfer switches in the main electrical vault supplying normal and emergency power to the majority of circuits in this building.

The equipment listed above was primarily manufactured by Federal Pacific Electric (FPE) and Westinghouse well over 40 years ago. The manufacturers are no longer in business and replacement parts are unavailable outside of those salvaged from other installations. The equipment is in poor overall condition when combined with age presents a high degree of

6 | P a g e unpredictable performance. In most cases, these components supply primary and/or emergency power to these buildings. Their critical nature, lack of readily available manufacturer-approved replacement parts and state of condition suggest this equipment should be assessed a General Condition Code of “F” using the Facility Condition Assessment guidelines.

Using the characteristics in developing the code levels, the basis for application of the “F” code for this equipment is as follows:

1. Being the primary and/or emergency source of power to the related building, failure is critical to the operation of this facility.

2. The equipment is in poor condition.

3. It has exceeded its useful service life by both industry and VA standards.

4. Requires immediate attention due to the unpredictability of future performance and reliability.

Upgrading or refurbishing this equipment does not appear to be feasible for several reasons:

1. Replacement parts have not been manufactured for some time causing a reliance on surplus or so-called “reconditioned” parts. In today’s market, such parts are offered at a premium with no warranties providing the VA with the same level of unpredictability as today.

2. Technical obsolescence creates issues in relation to arc flash hazards and the ability to minimize associated hazards as discussed previously.

3. Replacement is inevitable. Delaying replacement increases the risk of failure in addition to higher replacement costs in the future.

For these reasons it would seem prudent to assign a Cost Ratio of greater than 1.0 meaning it would be more costly to repair than replace.

Additional Equipment Condition Issues

Other condition-related issues that need to be addressed but are not as critical as those mentioned above:

Except for the recently-installed transformers and sectionalizing switches for buildings 51/51A,55, 57, 59 and 75, the enclosures for the outdoor sectionalizing switches and transformers have mild to moderate levels of corrosion with deteriorating paint as represented by the picture to the right. Leading causes of corrosion in these equipment types are:

a. The sectionalizing switches have troughs along the top of the frame where the doors come to rest. This area is prone to dirt/debris buildups that retain water.

b. Depressions in the tops of the enclosures enable puddling of water.

c. Lack of seals around the base of the enclosure enables water to freely penetrate the enclosure.

7 | P a g e

The sectionalizing switch for building 53 is in poor condition and needs to be replaced.

Supplemental Notes on Equipment Aging

As electrical equipment ages, the associated risk of a failure increases significantly. Such failures usually occur without prior warning and can be potentially catastrophic. Repairing or replacing aged equipment is another issue due to obsolescence or non-availability of parts. In the event of a failure, there is a great possibility parts will not be available or will have significant lead times. As an example, SDV found a defective main switch in a federal building that took 6 months to acquire a replacement part due to technical obsolescence.

This can lead to very costly repairs and extended down time for equipment and related circuits.

Included in section 2 to this report is an article presented by three professional engineers (PEs) specializing in electrical distribution systems – “Renovating Electrical Distribution Systems” by Syed M. Peeran, PhD, PE; Mario Vecchiarello, PE; and Jeff Romeo, PE. The first page of this article references the hazards of aging electrical systems and the unpredictability of equipment exceeding its useful service life. Equipment identified above has far exceeded this service life without upgrade and has this element of unpredictability regardless of past performance or testing.

Recommendations

Due to the condition, age and obsolescence of the above-referenced systems, a project should be initiated to replace the following electrical system components:

a. Building 1 – the primary distribution equipment (medium voltage oil-type transformer, main switchboard, emergency distribution panels and automatic transfer switches).

b. Building 3 – Panels in the basement main electrical closet.

c. Building 18 – The 480V and 208V transformers

d. Building 25, Panel EDP.

e. Building 51A - The main circuit breaker for switchboard MSB-51A switchboard as well as the Zenith and Caterpillar transfer switches (CDP and LSDP).

f. Building 53 – The transfer switches in the main electrical vault and the sectionalizing switch outside.

Additionally, a project needs to be initiated to treat the existing corrosion on the outdoor sectionalizing switches and transformers and minimize its potential going forward as follows:

8 | P a g e

a. Leaves, dirt and debris need to be removed from the troughs along the top of the door enclosures.

b. Heavy concentrations of corrosion should be removed using a wire brush.

c. Each enclosure should be painted with a rust-inhibiting paint that coverts corrosion and bonds it with the paint to provide a weather proof surface.

d. A layer of silicone caulk should be placed around the base of each enclosure to prevent water from penetrating the enclosure.

e. A routine maintenance program should be initiated to incorporate the above steps on an as-needed basis.

3. Adverse Environmental Conditions

The outdoor transformer-based circuit breakers do not have heater strips installed.

Abnormal ambient temperatures can lead to hardening of the grease used to lubricate the breaker internal operating mechanisms. This can cause the breaker to fail to open or be delayed in opening. Such conditions can lead to a fire or a significant arc flash incident.

The main switchboard for building 23 has open louvers allowing unconditioned outside air to penetrate these enclosures as pictured to the right. This enables dirt/debris to readily penetrate the switchboard and components with the potential for enclosed circuit breakers to fail to open in response to a fault.

Equipment located in the basement main electrical closet for building 3 has active steam pipes periodically releasing steam into the room. Steam penetration and related moisture has led to corrosion of the panels and components.

The generator switchboard for building 56 already has corrosion on the bus and related components as pictured to the right. This is due to a lack of heaters strips designed to minimize the effects of moisture penetration. Note – moisture penetration and exposure to abnormal ambient temperatures can lead to the same opening or delayed opening issues mentioned above.

The Generator Quick Disconnect enclosure for building 55 is not sealed around the base.

Additionally, the enclosure has louvers to the outside that contain filters. However, the lack of a base seal and these filters are ineffective in preventing water as well as dirt/debris from penetrating this enclosure. There are no heaters strips in the enclosure creating the same potential for catastrophic failure as mentioned for the outdoor circuit breaker above.

Section 5.2.2 Primary Switchgear and section 5.5.1(f) Electrical Rooms and Closets of the VA Electrical Design Manual referenced below outlines the requirement to provide ventilation

9 | P a g e or cooling systems to maintain indoor temperatures required for proper operation of equipment.

VA Design Manual

5.2.2 PRIMARY SWITCHGEAR

(a) Primary switchgear for Mission Critical facilities shall be located indoors, in a secure, protected location that complies with the Physical Security Design Manual. The area shall have the necessary ventilation or cooling systems to maintain indoor temperature as required for proper operation of the equipment, as well as access control. Equipment shall be located above the Base Flood Elevation.

VA Design Manual

5.5.1 GENERAL

(f) Electrical spaces shall have the necessary mechanical ventilation or cooling system to maintain the indoor temperature range required for proper operation of the equipment.

The circuit breakers in the transformers need to be exercised on at least an annual basis to keep the internal grease softened enough to allow the breakers to open properly in response to a fault. The frequency can be adjusted based upon findings. Additionally, consideration should be given to installing heater strips in the low voltage compartment of each enclosure.

The main switchboard for building 23 needs to have the louvers sealed off to protect the breakers from dirt/debris penetration. Consideration should be given to installing a conditioned air system to maintain a constant temperature based on VA Electrical Design Manual requirements. For additional guidance, enclosed in the publications section is a Square D publication outlining guidelines for protecting equipment from adverse conditions.

The generator switchboard for building 56 needs to be taken out of service and all bus components cleaned to remove the corrosion. Heater strips need to be installed to minimize the impact of moisture penetration.

The Generator Quick Disconnect enclosure for building 55 needs to have its base sealed and heater strips installed to minimize the potential for moisture buildup. The enclosure needs to be periodically cleaned and the breaker exercised.

The electrical equipment in the main electrical closet for building 3 needs to be replaced due to the corrosion created by the venting steam pipes in that room. The replacement panels should be relocated to another room absent of steam to ensure they will serve their useful service life.

The above conditions can cause breakers to fail to open or be delayed in opening. This can lead to a fire or a significant arc flash incident.

10 | P a g e

4. Overhead Sprinklers

The majority of electrical equipment in this facility has sprinkler heads, related water pipes and drain pipes directly over or within the spray pattern of the electrical equipment as represented by the picture to the right. No protective covers have been installed to prevent water from getting into the electrical equipment nor are physical protective devices installed to protect the actual sprinkler heads from physical damage. Both are required by National Electric Code section 110.26(F)(1)(a) through (F)(1)(d) as outlined below.

110.26(F)(1)(a) through 110.26(F)(1)(d)

(F) Dedicated Equipment Space. All switchboards, panelboards, distribution boards, and motor control centers shall be located in dedicated spaces and protected from damage.

Exception: Control equipment that by its very nature or because of other rules of the Code must be adjacent to or within sight of its operating machinery shall be permitted in those locations.

(1) Indoor. Indoor installations shall comply with 110.26(F)(1)(a) through (F)(1)(d).

(a) Dedicated Electrical Space. The space equal to the width and depth of the equipment and extending from the floor to a height of 1.8 m (6 ft) above the equipment or to the structural ceiling, whichever is lower, shall be dedicated to the electrical installation. No piping, ducts, leak protection apparatus, or other equipment foreign to the electrical installation shall be located in this zone.

Exception: Suspended ceilings with removable panels shall be permitted within the 1.8-m (6-ft) zone.

(b) Foreign Systems. The area above the dedicated space required by 110.26(F)(1)(a) shall be permitted to contain foreign systems, provided protection is installed to avoid damage to the electrical equipment from condensation, leaks, or breaks in such foreign systems.

(c) Sprinkler Protection. Sprinkler protection shall be permitted for the dedicated space where the piping complies with this section.

(d) Suspended Ceilings. A dropped, suspended, or similar ceiling that does not add strength to the building structure shall not be considered a structural ceiling.

The dedicated electrical space includes the space defined by extending the footprint of the switchboard or panelboard from the floor to a height of 6 ft above the height of the equipment or to the structural ceiling, whichever is lower. This reserved space permits busways, conduits, raceways, and cables to enter the equipment. The dedicated electrical space must be clear of piping, ducts, leak protection apparatus, or equipment foreign to the electrical installation. Plumbing, heating, ventilation, and air-conditioning piping, ducts, and equipment must be installed outside the width and depth zone.

Foreign systems installed directly above the dedicated space reserved for electrical equipment must include protective equipment that ensures that occurrences such as leaks, condensation, and even breaks do not damage the electrical equipment located below.

Sprinkler protection is permitted for the dedicated spaces as long as the sprinkler or other suppression system piping complies with 110.26(F)(1)(c). A dropped, suspended, or similar ceiling is permitted to be located directly in the dedicated space, as are building structural members.

The electrical equipment also must be protected from physical damage. Damage can be caused by activities performed near the equipment, such as material handling by personnel or the operation of a forklift or other mobile equipment. See 110.27(B) for other provisions relating to the protection of electrical equipment.

Exhibits 110.21, 110.22, and 110.23 illustrate the two distinct indoor installation spaces required by 110.26(A) and 110.26(F), that is, the working space and the dedicated electrical space.

11 | P a g e

In Exhibit 110.21, the dedicated electrical space required by 110.26(F) is the space outlined by the width and the depth of the equipment (the footprint) and extending from the floor to 6 ft above the equipment or to the structural ceiling (whichever is lower). The dedicated electrical space is reserved for the installation of electrical equipment and for the installation of conduits, cable trays, and so on, entering or exiting that equipment. The outlined area in front of the electrical equipment in Exhibit 110.21 is the working space required by 110.26(A). Note that sprinkler protection is afforded the entire dedicated electrical space and working space without actually entering either space. Also note that the exhaust duct is not located in or directly above the dedicated electrical space. Although not specifically required to be located here, this duct location may be a cost-effective solution that avoids the substantial physical protection requirements of 110.26(F)(1)(b).

Exhibit 110.21 The two distinct indoor installation spaces required by 110.26(A) and 110.26(F): the working space and the dedicated electrical space.

Perform a facility-wide survey to identify electrical components with water pipes or sprinkler heads placed overhead. Shields need to be installed in all areas where electrical components are within the spray pattern or have overhead water lines to prevent damage to prevent overhead water sources (sprinklers and water lines) from penetrating the enclosures. Such penetration could lead to a potentially catastrophic failure. Protective covers or cages need to be installed on unprotected sprinkler heads to prevent accidental activation with the same potential for catastrophic failure as mentioned above.

5. Improper Working Clearance

The MSB switchboards for buildings 23, 51A and 91 (building 60) have been placed directly against the wall represented by the picture to the right.

National Electric Code section 110.26(A)(1) requires a minimum 30” working clearance outlined below.

12 | P a g e

110.26(A)(1)(a) Dead-Front Assemblies

(a) Dead-Front Assemblies. Working space shall not be required in the back or sides of assemblies, such as dead-front switchboards or motor control centers, where all connections and all renewable or adjustable parts, such as fuses or switches, are accessible from locations other than the back or sides. Where rear access is required to work on nonelectrical parts on the back of enclosed equipment, a minimum horizontal working space of 762 mm (30 in.) shall be provided.

Exhibit 110.9

Distances measured from the live parts if the live parts are exposed or from the enclosure front if the live parts are enclosed. If any assemblies, such as switchboards or motor-control centers, are accessible from the back and expose live parts, the working clearance dimensions would be required at the rear of the equipment, as illustrated. Note that for Condition 3, where there is an enclosure on opposite sides of the working space, the clearance for only one working space is required.

There are bus connections located in the rear of the enclosure that cannot be serviced from the front as they are blocked by other installed equipment (circuit breakers, support structures, secondary conductors, etc.). Any attempts to service these connections as well as clean the rear portion of the enclosure from the front would require extensive disassembly of circuit breakers and secondary conductors that clearly is not the intent of the manufacturer or the NEC.

There appears to be confusion and misapplication related to installation guidelines of similar “dead-front switchboards” throughout the country based on our findings. The National Electric Code (NEC) defines “dead front” as “without live parts exposed to a person on the operating side of the equipment.” The recently-installed main low voltage switchboard located in the basement of the building is a “dead-front switchboard” based on this NEC definition. Operating personnel are protected from the energized components when covers are in place. However, this definition makes no reference to working space requirements.

NEC section 110.26(A)(1)(a), Dead-Front Assemblies (outlined below) uses this definition and applies other qualifying factors or elements in determining working spaces around such assemblies. Those factors or elements are:

1) Connections,

2) Renewable or adjustable parts, such as fuses or switches, and

3) Accessibility from locations other than the back or sides.

Additional clarification regarding work space requirements is provided in Exhibit 110.9 further down in NEC Section 110.26(A)(1)(a). It states, “If any assemblies, such as switchboards or motor-control centers, are accessible from the back and expose live parts, the working clearance dimensions would be required at the rear of the equipment.” The switchboard in question does have covers providing rear access that expose the switchboard bus components. This installation clearly meets these conditions requiring a minimum 30” horizontal working space.

The main switchboards in buildings 23, 51A and 91 need to be relocated to provide the minimum 30” horizontal working distance as required by NEC Section 110.26(A)(1)(a). This is necessary to facilitate proper maintenance and servicing of the bus and other components

13 | P a g e of the switchboard to include cleaning. Until this is accomplished, consideration should be given to performing annual infrared inspections of these switchboards.

Ensuring proper working clearances are maintained needs to be a point of emphasis for future switchboard installations.

6. Transformer Oil Test Results

Test Results

Oil samples were taken of all medium-voltage oil-type transformers. Each sample was tested using dissolved gas analysis (DGA). Laboratory test results have been incorporated into the test records for the medium-voltage transformers located in tab 3 to this report. All results were found to be normal with the exception of transformers for buildings 18 (480V), 24, 25, 51A and 59. These transformers showed abnormal levels of moisture content.

▪ Water leads to dielectric breakdown strength of both the solid and oil insulation properties with the potential for a failure.

Additionally, a publication related to analyzing transformer oil test results has been included in the Publications section to this report:

▪ “Transformer Oil Testing: An Early Warning System That Works,” article by Robert

Turcotte, Electrical Loss Control, June 2, 2008. (re-print)

The transformer oil for the above-referenced building should be re-sampled with a DGA analysis performed in the near future. Results should be compared to those levels noted in the oil test records in tab 3 to this report. If moisture levels continue to increase, consideration should be given to having the oil filtered to remove the moisture enabling the transformers to continue in service.

7. Transformer Gauges

Three (3) oil-type transformers serviced during this testing session were found to be missing operating gauge as follows:

Building Circuit ID Missing Gauge

Pressure Temp

3 T3 X

18 T18 (208V) X X

59 T59 X X

14 | P a g e

Pressure Gauge

Pressure gauges are designed to indicate whether the transformer has negative, neutral or positive pressure in the oil tank. Of great importance is where a transformer has negative pressure as obtaining oil samples on these transformers creates the potential for moisture and air to be pulled into the oil compartment. This can lead to “bubbling” with the potential for a flashover and catastrophic failure. Transformer manufacturers recommend placing a positive pressure on the oil compartment by first pulling a vacuum and then placing a nitrogen blanket on top to prevent such an occurrence. Typical pressures range from +3 – +7.5 psi.

Enclosed is a publication by Transformer Testing and Repairs, Inc. discussing the importance of transformer gauges and reasons for maintaining a positive pressure in the oil tank. An ABB publication has also been included describing the process for establishing the vacuum and subsequent nitrogen blanket on the oil compartment.

Temperature Gauges

Transformer gauges are designed to identify operating temperature and the highest oil temperature readings since being reset. This is a critical gauge used to identify transformers with oil temperatures either approaching or exceeding the manufacturer’s maximum oil operating temperature. A transformer exceeding its maximum operating temperature can be indicative of an overloaded transformer or potential defect.

Temperature gauges need to be installed in transformer T59. Pressure gauges need to be installed in transformers T3 and T59. In the event the 208V and 480V transformers for building 18 are not replaced as recommended above, a temperature and pressure gauge should be installed in T-18 (208V).

As a part of installing pressure gauges in the above-referenced transformers, the oil tanks need to have a positive pressure applied as outlined in manufacturer’s recommendations.

This can be done by pulling a vacuum on the tank and filling the space between the oil and the top of the tank with a nitrogen blanket.

8. Transformers with Neutral or Negative Pressure

Five (5) medium voltage oil-type transformers have pressure gauges installed but reflect either a neutral or negative pressure condition. This condition creates the potential for moisture and air to be pulled into the oil compartment when obtaining oil samples. This can lead to “bubbling” with the potential for a flashover and catastrophic failure.

15 | P a g e

Those transformers with pressure issues are:

Building Transformer ID Pressure Lvl

1 T1 0

2 T2 -2

25 T25 -0.5

52 T52 0

58 T58 -0.5

As mentioned previously, transformer manufacturers recommend placing a positive pressure on the oil compartment by first pulling a vacuum and then placing a nitrogen blanket on top to prevent such an occurrence. Typical pressures range from 3 – 7.5 psi. This process should be followed for the above-referenced transformers.

9. Lack of Cooling Fans

The medium voltage dry-type transformers in the MDP-75 and rm. 2C100 switchboards in building 75 do not have cooling fans installed. Both transformers have temperature gauges and controls but not the fans to provide airflow within the transformer coils to reduce operating temperatures. Installing fans can serve to reduce internal operating temperatures and extend the useful service life of the transformers.

Recommendation

Cooling fans need to be installed in the MDP-75 and rm. 2C100 transformers to avoid a premature failure of the windings. Additionally, the transformers need to be evaluated to determine if the need to be de-rated in response to the excessive temperature levels.

10. Leaking Transformer Fuse

The F3 fuse holder in transformer T18 (480V) was found with a leaking F3 fuse assembly.

The oil level gauge indicates the oil level is still in a good operating range.

Recommendation

Contact a local transformer specialist to determine a course of action to repair this transformer.

16 | P a g e

11. Medium Voltage Dry-Type Transformer Operating Set Points

The recently-installed medium voltage dry-type transformers in buildings 54, 55, 57 and 75 are designed to operate with cooling fans but have not been installed in the transformers in the previous note. These transformers have temperature control panels with a temperature probe attached to each of the three (3) transformer windings. The control panel is designed to monitor for potentially damaging high winding temperatures. The control panel initiates the following actions depending on the following set points:

▪ Set point 1 - control panel turns on the cooling fans in an effort to reduce operating temperatures below the maximum transformer temperature rating.

▪ Set Point 2 – control panel initiates an alarm indicating the transformer has risen beyond a certain level.

▪ Set Point 3 – control panel initiates a “trip” signal to “open” the low voltage breaker on the secondary side of the transformer to shed the load of the transformer and reduce the transformer operating temperature to avoid damaging the transformer.

The transformers for MDP/75 and MDP/55 in buildings 75 and 55, respectively, had the set points checked reflecting the following settings:

MDP/75 Set Point 1 – 145C

Set Point 2 – 165C

Set Point 3 - 220C

MDP/55 Set Point 1 – 110C

Set Point 2 – 125C

Set Point 3 - 220C

The maximum ambient operating temperature for these transformers is stated as a part of the data plate information such as “XXXC above ambient.” Each transformer has its own rating depending on the construction and installed features. The MDP/75 transformer is rated at 115C rise above ambient or roughly 135C. The first set point is 10C above the stated maximum. The MDP/55 transformer is rated at 80C rise above ambient or roughly

100C. Again, the set points are above the maximum temperature.

The set points were not verified for buildings 54 and 57.

It would appear the temperature control set points are not being set at installation nor based on individual transformer characteristics.

Recommendations

Realistic set points need to be established for each transformer to ensure proper operation of the transformers and avoidance of damaging excessive temperatures. A local Square D

17 | P a g e representative should be contacted for guidance in establishing the above-mentioned set points.

12. High Temperature Reading

Transformer T25 was found with the high temperature indicator off-scale. This is usually indicative of an internal defect. However, it is quite possible the point was accidentally contacted causing it to go off-scale. To further support the notion this was placed there accidentally is the oil analysis test results did not reflect any conditions typically leading to high temperature conditions.

Recommendation

The high temperature pointer was reset by SDV personnel. The gauge should be periodically monitored to see if the high temperature condition reoccurs. If so, a local transformer specialist firm should be contacted to determine any further course of action.

13. Medium Voltage Dry-Type Transformer Cable Issue

The recently-installed medium voltage dry-type transformers for building 54, 55 and 57 were found with cable ties used to route and secure the conductors to an internal support frame. Additional issues were created as conductors were secured across sharp edges of the support frame and barriers without any additional protection for the soft insulation of the conductors. Pictured below are images showing the support frame and cable tie implementation.

Cable ties and some of the barriers have sharp edges that can produce “chafing” on the conductor insulation as the conductors move while the transformer is energized. Pictured below are examples of conditions found when the cable ties were removed from these conductors. There is clear evidence of damage to the conductors. Damage of this nature creates the potential for “corona damage” to develop. Corona damage is a breakdown of conductor insulation combined with the presence of moisture potentially leading to catastrophic failure of the transformer.

18 | P a g e

However, there was no evidence of corona damage detected. This is most likely attributable to the use of conditioned air in each location and that corona damage does not typically appear with operating voltage under 5000 volts.

Corrective Action

The process outlined below was used to correct these conductor issues:

a. Conductors were removed from the barriers, cable ties removed from points used to secure conductors to frame components or other points with the potential to damage the conductor insulation.

b. Conductors and barriers were cleaned using denatured alcohol to remove existing corona damage and other dirt or deposits with the potential to develop tracking.

c. Mastic tape or similar insulating material was wrapped around each conductor where it passes through the barrier or has been damaged by cable ties, etc.

d. Mastic tape was covered with Scotch 33+ vinyl electrical tape.

e. Conductors were reinstalled through the barriers and re-routed across the support frame. Efforts were made to avoid edges of the support frame or other points where potential for future damage will be eliminated or minimized. Supplemental protection was used in those areas where edges could not be avoided. In all cases, the conductors were secured to the support frame and barriers using the Scotch 33+ tape.

Pictured below are examples of the repairs performed by SDV using the above process.

The red arrow above points to an area where the conductor was run across an edge of the support frame. Behind the conductor is an additional mastic pad providing additional protection of the conductor from the edge of the frame.

19 | P a g e

Monitor the repairs to ensure there is no further damage to the conductors for these transformers. Additionally, use of cable ties without additional protection of the conductors should be a point of emphasis during the acceptance process. Rubber grommets or similar protective items should be a requirement wherever a conductor passes through a barrier or is secured to support structures.

14. Defective Relay

The N7 protective relay in switchboard PS-75-1 (building 75) has a defective display rendering it inoperable. This is a GE Multilin 350.

Replace the N7 protective relay to include programming current settings.

15. S&C Switch Identification

Several S&C switches have been installed that don’t have any circuit markings identifying the downstream equipment being powered from each cubicle.

Recommendation

Identify each of the circuits being fed by these S&C switches. Each cubicle should be marked with the appropriate circuit identification or as a spare.

16. Emergency Equipment Markings

The emergency equipment located throughout the facility is not properly labeled to readily identify them as being part of the emergency power system as required by National Electric Code section 700.9 referenced below.

700.9 Wiring, Emergency System.

(A) Identification. All boxes and enclosures (including transfer switches, generators, and power panels) for emergency circuits shall be permanently marked so they will be readily identified as a component of an emergency circuit or system.

A project needs to be initiated to update the emergency equipment labeling. Consideration should be given to changing the existing labels to a white lettering on a red background.

20 | P a g e

Incorporating designations such as LS (life safety), EQ (equipment) and CB (critical branch) into the labeling scheme will further identify these emergency system components.

17. Dangerous Switchboard Configurations

Switchboard MSB-51A (building 51) has an unusual connection configuration with the potential for a personnel safety hazard. The connections are configured as follows:

a. Transformer T51A feeds the main circuit breaker in Swbd MSB-51A.

b. The secondary side of the main circuit breaker does not power the circuit breakers located in the secondary side of this switchboard. Instead, the secondary side of the main circuit breaker supplies power to the MDP-51 switchboard located in the next room.

c. Switchboard MDP-51A has a 1000 amp circuit breaker supplying normal power to ATS-51A. The generator supplies emergency power to ATS-51A.

d. The output or load side of ATS-51A is connected and supplies power to the secondary side circuit breakers of switchboard MSB-51A.

Typical switchboard configurations use the main circuit breaker to isolate circuits on the secondary side of the circuit breaker. In this configuration, the main circuit breaker does not isolate the secondary side breakers but instead removes normal power to ATS-51A that in turns starts the generator to supply emergency power to the ATS and subsequently the secondary side of switchboard MSB-51A. This is commonly known as a backfeed and means the secondary side of the switchboard is always energized unless both normal and emergency power sources to ATS-51A are removed.

The same configuration is also present in switchboard MSB-53A in building 53.

Below is a drawing of the configuration of switchboard MSB-51A also representative of that for switchboard 53A:

21 | P a g e

The hazard to personnel stems from the fact that this configuration is highly unusual and not documented at the switchboard level. If an individual were to check just the secondary side of the main circuit breaker and not the secondary bus in the switchboard, they would fail to detect the presence of voltage placing them in danger of electrocution.

A sign or placard of some sort should be affixed to the front of switchboards MSB-51A and MSB-53A to inform personnel that the secondary side of the switchboard remains energized when the main circuit breaker is turned off. Isolation of both normal and emergency power sources to ATS-51 or ATS-53 is required before performing maintenance on the switchboard.

18. Defective Circuit Breakers

Two (2) circuit breakers were found to be defective. The defects are noted as follows:

Building Room Swbd/Panel ID Circuit Issue

51A BA100 CDP Main Breaker has excessive internal resistance.

60 Generator Rm - Generator Main Tripped prematurely on LT and ST functions.

Corrective action for these breakers is:

Building Room Swbd/Panel ID Circuit Suggested Action

51A BA100 CDP Main Replace breaker.

60 Generator Rm - Generator Main Replace trip unit if nuisance trips develop.

19. Defective ATS Operating Mechanism

ATS-55-EQ1 has a defective operating mechanism in need of repair. This has been problematic for some time now.

Recommendation

A factory representative should be contacted for a quotation to perform necessary repairs.

Note the switch can be repaired without a power outage by placing the unit in bypass to perform repairs.

22 | P a g e

20. Overheated Connection

The MCC circuit breaker in the Main switchboard for building 53 was found with an overheated secondary connection as pictured to the right. This is most likely attributable to a loose or defective connection.

Recommendation

De-energize the switchboard. Remove the MCC breaker. Inspected the breaker and bus components for serviceability. Repair or replace defective components. This may require replacing the circuit breaker. Install the breaker and tighten all connections per manufacturer’s recommendations.

21. Building 75 Defective Surge Modules

The “B” and “C” phase surge modules in MDP/75 (building 75) are defective.

Replace the “B” and “C” phase surge modules.

22. Building 75 Emergency Power Outlets

It was noted there are no emergency power outlets in and around the office area in building 75.

Recommendation

Consideration should be given to installing some emergency power outlets in building 75 to power computers and other related equipment in the office area.

23. Pixis Server Power

During testing of building 57, it was noted the Pixis servers had no power as there is no emergency power in building 57.

23 | P a g e

Consideration should be given to installing some emergency power in building 75 to power up these Pixis servers in the event of a normal power loss.