CJD SS01 Failure Report Dec 2020_Redacted.pdf

For Reference Only

Not Intended for Bidding Purposes

Chief Joseph Dam SS01 Failure 2020

EVALUATION REPORT

NIELSON NWS HT&E 2020

Contents

1. Executive Summary

2. Background

3. SS01 Specs

4. Failure Timeline

5. Disassembly, inspection, and damage assessment

6. Discussion of Options and considerations to return unit to service

7. Station Power while SS01 is out of service-

Pertinent Facts

Evaluations

As Found

Testing Results

Post Cleaning Tests maintenance suggestions

A possible Culprit

Fault 1 (19 May 2020)

Fault 2 (July 1 2020)

SS01 Unstacked, Inspection and Damage Assessment 14 OCT 2020

Overall Assessment:

Rotor Poles

Bearing Inspection

Station Service Power Supplies

1. Executive Summary Chief Joseph Dam Station Service Generator SS01 tripped offline on 19 May 2020 due to a winding fault.

Tests and analysis were performed after the trip event, and evidence was found of an A phase generator differential, phase imbalance, and overcurrent trip, though no damage was observable. Basic insulation testing was performed after the initial trip, phase to ground resistance readings were sufficient for return to service, in the 4-7Gohm range. With consultation from the Hydroelectric Design Center, the unit was started on 1 July 2020 but quickly tripped offline again and smoke was observed. Disassembly and inspection revealed an apparent phase to ground and phase to phase fault in the lower end turns. There is severe damage to the coils and minor damage to the core. The unit cannot be returned to service without extensive rehabilitation to the stator windings. Rewinding under contract during the capital project is recommended.

2. Background SS01 is one of two house power units at Chief Joseph Dam. SS01 is operated independently or in tandem with SS02, an identical station service sister unit, depending on house power requirements.

3. SS01 Specs SS01 is a 3MVA 1955 Pelton made, francis turbine hydrogenator, producing 4160V 3 phase power. It is a multi-turn asphalt mica insulated 14 pole, 7 parallel, 5 coil per parallel, 105 slot machine, with “Z” winding configuration.

4. Failure Timeline

a. Event 1 – trip from service on 19 May 2020, relay records indicate unbalanced phase currents, high A phase current, and generator differential trip.

b. Post trip evaluation was performed, including evaluation of XP breakers which connect SS01 to station power grid, A-B-C phase insulation resistance and polarization index, A-B-C phase winding resistance, and A-B-C phase CT ratio and excitation current testing. Testing results appeared normal with the exception that A phase had a lower insulation resistance to ground compared to B and C phases, but the IR value for A phase was still a sufficient value to return to service. Relay records were also obtained and shared with HDC.

c. After correspondence between CJD and HDC, there was no clear indication of why the generator had tripped offline, given than the insulation tests were acceptable. HDC recommended a controlled start up of the generator including spinning the unit up to speed no load and slowly building voltage while monitoring generator parameters. The generator was started on 1 July 2020 and faulted immediately following the application of excitation field, and the generator tripped offline again.

d. Event 2 – trip on July 1 after voltage buildup. Shortly after the application of the excitation field, smoke was observed coming from the generator along with some debris, CO2 was discharged.

e. Following the second trip event, it was clear that although the insulation test results were sufficient for return to service, there was a persistent failure mode which was exacerbated during the second start, and disassembly of the machine was necessary to assess the damage.

5. Disassembly, inspection, and damage assessment

a. Findings: Evidence of a phase to ground and phase to phase winding fault in slots 96 and 97 from A phase to B phase coils and ground. Slots 94-101 show evidence of insulation damage such as burning, embrittlement, copper deposits and debris. Damage to the core clamping fingers between slot 96 and 97.

b. Probable fault initiation and progression: Investigation into the root cause of the failure revealed that water contamination may have played a role. There was a water leak in the thrust bearing oil cooling system which resulted in water leaking downwards from the top of the generator about 1 month prior to the failure. SS01 is constructed such that the thrust bearing and upper guide bearing are above the rotor and stator. There was also severe oil and carbon dust contamination forming a conductive slurry which was coating much of the generator end windings, due to a damaged thrust bearing vent tube combined with carbon dust from slip ring brushes. Given that the initial post trip testing indicated acceptable and normal insulation resistance and winding resistance results, it is likely that the failure began as a turn insulation failure. The turn insulation failure would result in phase imbalance, high current, and differential. Ground wall insulation system resistance was diminished in A phase when compared to B and C phases, by roughly 25%. Winding resistance measurements were nearly identical among all three phases.

6. Discussion of Options and considerations to return unit to service a.

7.

CJD SS01 FOOS 20 MAY 2020 9:35AM

PERTINENT FACTS • SS01 was the only SS generator running at the time unit was FOOS.

• The unit was loaded to approximately 1MW.

• Event log indicates: Overcurrent, Phase Imbalance, Generator Differential

• CO2 was released

• Bus tie breakers XP19 and XP20 both tripped resulting in loss of power to the critical buses North/South with SS01 lockout.

• A phase is the phase apparently affected by differential lockout.

• There were multiple loose wires found in the S0 board which connected SS01 metering circuits.

• An instrumentation fuse was burnt up in the S0 board

• SS01 experienced a water leak in the thrust bearing cooler water that apparently washed down into the winding area and resulted in a field ground in late April 2020. This was not widely known information.

• Megger of stator windings found A Phase 6Gohm, B phase 6Gohm, C phase 4.5Gohm.

• The rotor megger results at 55kohm. UNSATISFACTORY

• Inspection revealed heavy oil contamination within the generator slip ring housing, around the brushes, along the shaft, and between the slip rings. Heavy oil contamination appears to have been generated from a damaged thrust bearing vent pipe which appears to have been leaking oil onto the top of the thrust bearing and being drafted up into the slip ring area due to shaft rotation.

Oil was discovered in significant quantities around the stator cooler, all around the stator frame and cover and even accumulations outside in the 38 device wiring cabinet.

EVALUATIONS INSPECTION & TESTING: It seems that this fault is very likely the direct result of contamination in the machine. There are several factors which play into the contamination based fault, such as water, oil, and carbon dust from slip ring brushes. Cleaning has been ordered and retesting of the stator and rotor windings with megger device is planned. Furthermore, disassembly and testing of the current transformers on the neutral and line end are partly completed and testing for misconfiguration or significant differences between phases.

Stator winding and rotor meggering

CT and PT testing for turns ratio and excitation current

PF testing for capacitor and arresters

AS FOUND

TESTING RESULTS

Stator Megger: A 6G ohm / B 6G ohm / C 4.5G ohm

Rotor Megger: 55k ohm

Neutral CTs TTR: (A 119.8/ 25.5mA) (B 119.7 / 15.5mA) (C 119.8/ 16mA)

Line CTs TTR: TBD

A phase CT excitation current is higher than the other phases and indicates a likely residual magnetism from the recent high current event on A phase.

POST CLEANING

TESTS

Rotor Megger: 550M ohm

10,000 x improvement from as found test.

MAINTENANCE

SUGGESTIONS

• Consider running stator heaters during outage in conjunction with dehumidifiers to pull moisture that may be latent in the stator windings from the water leak. Retest stator megger after 1 week of heat/dryout.

• Fix the (apparent) oil leak at the thrust bearing vent tube.

• Plan routine scans and inspections of control boards to find similar problems prior to failure.

A POSSIBLE CULPRIT

This thrust bearing oil vent is severely damaged at the 90.

Heavy oil/carbon contamination

Heavy oil/carbon contamination

Oil leak

Oil vapor condensed inside a device cabinet, internally vented to the stator frame area below the thrust bearing.

Oil mist contamination

13 | P a g e

FAULT 1 (19 MAY 2020)

DATE: 5/19/2020 Time: 8:34:30.784

87R1 A phase differential picked up. Noticed that the PT signals are stable and the Neutral current does not change for the A phase fault condition. Found out neutral breaker was open IN103. IN101 & IN102 are the breakers connecting the unit to the main busses.

14 | P a g e

FAULT 2 (JULY 1 2020)

Analysis on the initial fault appeared inconclusive, so the generator was spun at speed no load and faulted a second time. The differential trip occurred at speed no load with the main XP breakers open.

Relaying details here:

Event Data on 7/1/2020:

IN101 main breakers are open, A-B phase fault.

IN101 main breakers open, IN103 neutral breaker open, A and B phase differential trip (87R1/87R2).

17 | P a g e

SS01 UNSTACKED, INSPECTION AND DAMAGE ASSESSMENT 14 OCT 2020

Maintenance Dept. attempted to spin the unit and see if the unit would trip again, during the start up smoke came from the machine and it was shutdown.

Rotor has been pulled and there is a clear failure in the lower end winding apparently A phase to B phase fault in the end winding area. Slots 95-98 are affected with the suspected fault originating in slot 96 front coil (A phase 3rd position of 5 coils per parallel at a voltage of ~1000V-1500V in the A3 coil. )

From inspection, it appears that Coil A3 in slot 96 experienced a turn to turn failure near the center of the coil, the majority of the copper has been ejected as molten and vaporized material, deflagration appears to have occurred in the CW direction of rotation and downward and toward the center axis of the machine. It appears to have incurred collateral damage to the A4 coil in front of slot 97 as well as B5 coil in the back of slot 96 and C2 in the back of 97. Multiple nearby coils have less apparent damage with embrittled insulation and migrated asphalt binder material from local overheating.

Deposits of copper vapor appear on the outside of the lower end windings beginning on slot 96 and extending to slot 102 in a diminishing pattern.

The steel core clamping fingers between slot 96 and 97 appear to have burned and become molten, likely in the initial failure arcing event. The core seems to have decompressed slightly in the area of the damaged clamping fingers.

OVERALL ASSESSMENT:

It seems that during the initial fault event, fault current was isolated to A phase at below 600A. This appears to have been a fault internal to the winding. Megger test results indicate there was not an active A phase to ground continuity as the resistance to ground was 6Gohm. During the attempt to energize the unit after the initial trip, a normal start up procedure was initiated and an A phase to B phase fault occurred soon after excitation was applied. During this second fault event, significant smoke and debris was noted and fault currents are seen in A phase and B phase in excess of 1500A. During this event it is likely that the initial internal turn-turn A phase fault was quickly exacerbated resulting in high temperature arcing, vaporized copper, melting insulation, and an explosion which may have occurred from A phase to the steel core clamping fingers which resulted in a sustained phase to phase fault in slot 96 between A3 and B5 coils.

High heat in the local area resulted in migration of asphalt binder material, insulation embrittlement, cracking and contamination within the area between slots 95 and 101.

ROTOR POLES

Rotor Poles show contamination from the fault.

BEARING INSPECTION

Not Intended for Bidding Purposes38 | P a g e

Post Failure Electrical Measurements

Stator A phase megger 497mega ohms 10 min 3300 mega ohms.

B/C phase wouldn’t build voltage. Stopped at 6V

Winding resistance. A 38.82mohm T1-T4 increased resistance due to destruction of 1/7 parallels.

B 34.50mohm T2-T5 increased resistance due to damage of 1/7 parallels.

C 33.00mohm T3-T6 relatively unaffected reading.