21. Naples UPS Survey Report.pdf

ISEC Survey Notes From Naples ES/MHS

24 April 2018

DEREK SWEINHART

A0CTAJSB

Highlight

1. Location Naples ES and Naples MHS, for the purposes of this report considered one campus.

2. Issues This school suffers frequent, short power outages and spikes due to poor quality utility power. The issues are said to derive from nearby industrial complexes consuming bursts of power, thus causing local brownouts.

This school also suffers from poor internal wiring, circuits with too many branches, undersized circuits, and failing electrical components.

When the power fails, any network equipment – including servers and distribution switches – not on protected power also temporarily fails. This is exacerbated by the fact that undersized distribution circuits (breakers and wiring) exceed protection levels and breakers trip when all the devices simultaneously come back to life and initial startup current draw exceeds that which the associated breakers allow.

Also problematic is electronic equipment such as projectors, which are plugged into receptacles around the school. They contain sensitive circuitry and have a greater failure rate due to the poor utility power at this school.

3. Solution The solution is a three part solution:

a. Provide server-room quality backup power to all the core IT equipment including distribution switches. This should be in the form of modular, dual-conversion, online Uninterruptible Power Supply units with an n+1 configuration in the power supply modules.

This is the quickest, easiest, and most urgent part to fix. Extra power modules should be kept on-site and accessible so that in the event that any modules fail, they can quickly be replaced without loss of service.

b. Fix internal wiring issues that are compounding the problems. Many of these issues have been identified in the details section, below. Other issues must be reported in detail by the locals at the school, using the guidance below. Many of these problems can be fixed quickly, but others will be discovered over time.

c. Install a facility-wide solution to the poor utility power. This will be the most complicated and expensive part of the solution, but will solve the remainder of the problems. Because the utility power’s integrity cannot be ensured, a good mitigating solution is key. ISEC and the Facilities Project Manager are working toward this solution, and will make it available as soon as possible.

For those readers who wish to skip the details sections, please see Section 8, The Way Forward, below. But note there is important information in the details section.

4. Complicating factors Reportedly, the owner of these buildings has rules that their own electricians have first right of refusal to perform any electrical work to be done on site. From my own brief interaction with the local electrician, it appears that they would much rather blame all issues on utility power rather than focus on what they can control, which is the internal issues that are compounding the problem. While there is clearly a conflict of interest here, it will likely have to be endured and mitigated with clear direction and patience.

5. UPS for IT Solution Details There are two working 10kva and one working 15kva UPS units in the facility already (locations below), with a third on order.

The two legacy units appear to be working fine, although their age should be identified and their batteries replaced if necessary per manufacturer’s recommended replacement intervals. Because a single failure of these UPS units is not critical, the batteries could be replaced only when failure results, but there may be a delay in procurement of the necessary parts. If batteries are ordered early, they must be put into service to ensure their longevity.

Modular UPS units would be preferred for the following reasons:

a. Modular units can be set up in an n+1 or n+2 configuration, so that if one unit (or 2, respectively) fails, it (they) can be replaced without service interruption.

b. Modular units can adapt to the load, and be much, much more efficient than a 15kva unit with a load under 20%, which is the case now. Even good UPS units rarely have a power factor above .9, and this power is wasted whether the load capacity is used or not.

c. If all units take the same modules, than only one spare part must be kept in inventory.

d. Modular units are generally easier to service by the user.

e. Modular units are expected to endure frequent outages longer because a failed module can be replaced locally and quickly without professional intervention.

Unlike Sigonella, the Naples schools have Telecommunications Rooms (TRs) in more sporadic locations, and there is not an abundance of free cabinet or floor space in all of them. Some distribution points are actually in small wall-mounted cabinets throughout the building. For these reasons, it may be appropriate to wire circuits from a UPS output panel in another room, and therefore have fewer UPS units than TRs.

6. Fix Internal Wiring Issues Details – IT Specific.

Below is a list of all TRs and their current power setup, with solutions where applicable.

a. Room 225, MHS

Internal wiring problem solved, but the current setup can be improved.

This room had a 15kva UPS unit in it with a 5% load, yet this room – containing the majority of the IT equipment servicing the top floor of the MHS – has had issues with its IT equipment tripping breakers every time power would come back on. This is due to the high start-up current experienced when a switch is first powered on: every switch was drawing maximum power and tripped the two 10-amp circuits they were running on. These 10-amp breakers are normally reserved for lights, and for some reason in this case are hardwired to power strips in the IT equipment cabinets (which have their own 16-amp breakers).

The UPS in this room is hardwired to a 240V panel with three circuits (one spare). The two usable circuits are both in a single-gang (or European equivalent) box wall-mounted behind the IT equipment cabinets, and each terminates in one power outlet. These outlets had a monitor plugged into one, and a very small disposable UPS unit in the other, which had a computer plugged into it.

I used a bunch of power strips to methodically move all the distribution switches to UPS power, but more importantly, was careful to ensure that all breakers and fuses in their power chain are 16amp or greater. This will fix the problem with the associated breakers tripping from start-up current, and it will also keep these switches from ever going down due to power outages as long as this particular UPS unit is in good shape.

There are plenty of ways to further improve this situation:

i. The UPS was set up as 240v input and 240v output. Because three-phase power is available, it would be more efficient to use a three-phase setup. This unit is capable of three-phase.

ii. The UPS input and output breakers are both 32 amp. This effectively halves the capability of this unit. These wires and breakers need to be upsized to the manufacturer’s recommended sizes to ensure that the input can stand full battery-charging current draw and that the output can handle a full 15kva load + recommended headroom.

iii. Instead of two outlets mounted behind the equipment cabinets, each equipment cabinet should be hardwired from the UPS output panel with two un-switched Power Distribution Units (PDUs). Each network switch with two power supplies should be plugged into two different PDUs so that a local accident will not bring down the network equipment.

a. These cabinets are very tight and will become tighter with appropriate PDUs. If possible, replace them with 80cm wide cabinets and add Cable Management Panels (CMPs) to increase flexibility for moves, adds, and changes.

b. Further improvements can be made by running the network distribution cables directly into the top of the cabinets via overhead cable tray rather than down the wall and across the floor in surface mounted conduit.

c. If one were doing the improvements in iiia and iiib, above, then one might consider moving the cabinets forward so that there are at least 80cm behind the cabinets to fully open the doors. Space is available.

iv. With power inside the equipment cabinets, the doors can be closed. Avoid plugging external devices into IT PDUs.

v. Some network switches do not have two power supplies. Adding a redundant power supply to those network switches will ensure availability.

vi. Currently, power circuits are run in plastic corrugated conduit with the category copper network distribution cables. Per both code and standard, these should maintain 6 inches of separation or be divided by metal conduit. The code portion is a safety issue to keep current off of network cables, but the standard portion is because having power with copper network cabling creates Electromagnetic Interference (EMI). Remove these (now unused) power circuits and adhere to this rule of separation in the future.

vii. Power to and from the UPS unit is in unconnected plastic conduit. This should be re-run in Liquidtight flexible metal conduit (LFMC) with appropriate connectors to the panel and unit to ensure wiring protection in addition to the health and safety of users.

b. Room 118A, Fitness Room, MHS

There is a 5-Rack Unit (RU) wall-mounted cabinet in this room with a power strip. The power cord to this strip exits the top of the cabinet and runs down to a wall outlet in surface-mounted conduit. This network switch has reportedly been losing power and needs to be reset via its power breaker. There is a power panel in this room that contains a breaker for this switch. Fix this area with the following steps:

i. This power panel has some breakers being held closed (on) via electrical lockouts.

These electrical lockouts are meant to hold breakers open (off) when electricians are working on a circuit. This is a fire hazard and these lockouts should be removed immediately. Ironically, they are being used to hold on the emergency lights, which likely produce almost no load unless there is actually an emergency (or the power is out).

ii. The reason for the issue above is because there are no light switches in the gym, and those electrical lockouts are being used to ensure users do not turn off the wrong lights when turning the lights out in the gym at the end of the day. Install proper light switches to control the gym lights, and close this power panel to the public.

iii. The breaker containing the network switch frequently trips because it is on a power strip with other gym equipment. Run a new, dedicated 16-amp circuit with continental European receptacle to just above the equipment cabinet. Plug the switch in directly with the existing opening on top of the cabinet. There appears to be space for one more breaker in this power panel without running a new busbar. If necessary, run a new busbar.

iv. Because this network switch is holding only five users and it is far away from other places where UPS units may be deployed, you may consider rack-mounting a shallow (40cm or less), 2-RU, “disposable” UPS unit in this cabinet, re-mounting the other equipment as necessary. There is room in this small cabinet.

v. Other: there are controls for the basketball nets beside the power panel in this room. They should be re-located to the actual gym so that the user can see when the basketball nets have reached the intended height.

c. Room 112A

There are no problems reported in this room, but this survey has revealed some looming issues that should be fixed:

i. There is a 10kva UPS in this room, but is limited by a 25 amp breaker (which will only allow 9500va). Change this circuit to allow full charging current + headroom, as recommended by the UPS manufacturer (APC).

ii. The output breaker on this UPS is 32A. While not incorrect by code, this should not be larger than the input.

iii. Alarm threshold on UPS set at 10kva, which it cannot reach with this input breaker, and does not allow for mitigation of load were it to approach full capacity.

iv. The breaker subpanel, which contains the UPS and air conditioning equipment, is 25A. This is extremely undersized, and limits the UPS further than 6.c.i, above.

Increase the capacity of this subpanel.

v. Two of the racks/cabinets in this room are on a 16-amp, self-breakered PDU, but the other has its equipment on a cheap PDU and two other plugs on a mini-UPS power.

These mini-UPS units are almost worthless and are completely worthless in a TR.

Hardwire 2 vertically mounted PDUs in the back of each cabinet facing each other and distribute all loads across two different PDUs as outlined in 6.a, above.

vi. I am skeptical that a 16-amp breaker is large enough per manufacturer for the air conditioner in this room. Verify this or up the size of the circuit.

d. Room 124, MHS

No issues were reported here, but a site survey has shown some things to fix before power becomes a problem:

i. There is a hardwired UPS here that was not really meant to be hardwired. It is no longer in service, and has input, bypass, and output breakers. Completely remove this unit and its wiring.

ii. There is also a small UPS unit in its place. Remove this.

iii. Main breaker in local subpanel is 25amp. Sub-breakers include 25amp for UPS and 16amp for air conditioner. This is not enough head room on the main breaker, especially because…

iv. ….as above, I am skeptical that a 16 amp breaker is enough for this air conditioning unit per the manufacturer. Verify or up the size of this circuit.

v. IT equipment is on a 16amp power strip and one small UPS unit plugged into the wall. As before, remove the small UPS and provide 2 PDUs for rack power, as outlined in 6.a, above.

vi. Provide appropriate UPS power if desired.

e. Curriculum Room near Cafeteria, ES This room contains a small wall-mounted cabinet with a network distribution switch. This room has three reported problems, the first two of which have been verified during the survey:

i. There is a 20amp circuit breaker on the first row of this panel labeled “AC” that constantly trips. This breaker is likely too small for the air conditioner, as most two-ton units require a minimum of a 25amp circuit to provide headroom for startup current. Replace this circuit with the size recommended by the manufacturer. A 32amp circuit will likely suffice if the manufacturer’s recommendation cannot be verified. Dispose of the existing 20amp breaker, as it is likely weak after so many faults.

ii. This entire power sub-panel loses power, taking out lights to the cafeteria, network distribution co-located in this room, and the air conditioner in this room, among other items. The problem is that in the main electric room, in the middle power panel, 2nd row of breakers, 3rd breaker from the left is a 25amp breaker for this panel, which is clearly too small. Re-run this circuit to this power panel with an appropriately sized cable and breaker. 63amp should suffice.

iii. In this room’s power panel, there is a Ground Fault Circuit Interrupting (GFCI)-style (Disclaimer: For the purposes of this document, I refer to devices that measure the delta of current return on neutral vs. other conductors as GFCI, even though I am not sure what these specifically Italian devices are called here. I realize that it is reading current on the neutral and not the ground.), 25amp breaker for the adjoining 20amp, 3-phase breaker that reportedly trips often and cuts power to 220-volt outlets in the cafeteria. There are no floor outlets in sight, but this GFCI protection may be required due to Italian law and proximity to the kitchen. Remove any load not absolutely required to be GFCI protected, and put it on its own circuit.

Replace this GFCI protection mechanism to help prevent false positives.

f. Library TR, ES There are no reported issues in this TR, but a survey has uncovered some issues worth noting:

i. The UPS has a fan fault. Replace or fix the fan as applicable. Reset the fault.

ii. This 10kva UPS has a 16amp, 3-phase breaker, which limits it to approximately 6kva.

This should be replaced with an appropriately sized circuit, which also accounts for charging and headroom. This is particularly important if this UPS will be used to power other circuits in the future.

iii. The air conditioner is on a 16amp breaker. As before, I doubt this is appropriately sized per manufacturer’s recommendations.

iv. Power to the IT equipment would be greatly improved were it to have two hardwired PDUs directly in the cabinet on different circuits with the power supplies distributed evenly as outlined in 6.a, above.

g. 132A, Server Room, ES

There are no problems reported with this room, except for the fact that the decommissioned UPS does not hold IT equipment through facility-wide power outages.

There is plenty of room for improvement:

i. The UPS input breaker is 20a, 380v. This will limit the new UPS to 7600va. This is extremely important to recognize and fix if the new UPS ordered for this area is intended to serve equipment in multiple TRs.

ii. There are plenty of 16amp output (from UPS) breakers here, but they are not used in any purposeful fashion. Recommend all cabinets with active equipment receive two to four hardwired PDUs as outlined in 6.a, above. For cabinets with more load, such as those housing servers, more hardwired PDUs would be appropriate. It is best to ensure no individual breaker will ever trip when dealing with IT equipment.

Distribute the power supplies across different circuits as outlined in 6.a, above.

iii. This is an awkwardly shaped room for a server room, but it is made much worse by having an awkward cabinet layout and also by having unused cabinets and equipment in in the way. If improvement of this space is desired, then it is highly recommended to plan and implement a new design before the new UPS is installed.

It will be much easier to fix this room during the power upgrade than after.

h. Room 142, ES

Reportedly, the UPS bypass breaker sometimes trips in this room. The survey indicated no reason for this. The full load on the UPS bypass breaker showed a steady 2.74 amps, with the sub-loads adding up exactly.

This room has a hardwired UPS in it that is long dead and unused. The UPS was never specifically meant to be hardwired anyway. Remove this and all unused wiring. Remove all non-IT equipment from this room and clean it up.

Although there is no obvious reason this breaker trips, it may help to remove the unused UPS and provide straight power to the IT equipment (the best way being outlined in 6.a, above). It makes sense to provide backup power to this equipment, either with a local UPS or circuit from another UPS, as discussed in section 5.

i. TR inside Staff Restroom, ES Reportedly, the UPS bypass breaker trips occasionally in this room. The survey revealed no obvious reason for this. The full load on the 32amp bypass breaker pulled a constant 5.95 amps, with all the sub-breakers sum equal to this.

This room has a hardwired UPS in it that is long dead and unused. The UPS was never specifically meant to be hardwired anyway. Remove this and all unused wiring. Remove all non-IT equipment from this room.

Although there is no obvious reason this breaker trips, it may help to remove the unused UPS and provide straight power to the IT equipment (the best way being outlined in 6.a, above). It may make sense to provide backup power to this equipment, either with a local UPS or circuit from another UPS, as discussed in section 5.

j. TR in new addition, upstairs, labeled “Telephone”, ES There are no reported issues in this room. It contains just two switches on a PDU inside a cabinet, and no UPS. Issues:

i. The PDU is a safety hazard. It is modular and not sealed. Someone could be electrocuted if they touch inside this unit. Replace this PDU, preferably with power distributed as outlined in 6.a, above.

ii. Electric is run with telecommunications copper cable, which is both a code and standards violation. Replace with new circuitry, properly segregated, and remove the old, unused circuit.

iii. Provide backup power as desired.

Power is provided to this cabinet from the electrical room next door. There is one 16amp circuit labeled “Data Rack Receptacles”.

7. Fix Internal Wiring Issues Details - Other

a. There are multiple Compact Fluorescent (CFL) light fixtures around the building that have bad ballasts, probably due to poor utility power. They are making horrendous, high-pitched noises. These should be replaced or removed. It may be worth considering LED bulbs or fixtures (new fixtures likely preferred) before or after 3c, above.

b. Room 117, ES There is a power panel in this room that contains a circuit that reportedly trips often and causes problems for the library and surrounding areas. Upon investigation, a fire hazard was also discovered:

i. During the survey, a loud buzzing noise was coming from a breaker in the second row. This is often an indication that a breaker wants to trip, but has failed to do so.

The 5th breaker from the right, which contains half of the light load for the library, was verified to be pulling a constant ~11amps while on a 10amp breaker. This could theoretically start a fire due to the wires getting too hot; it will cause a fire if more load is applied.

Because the lights in the library are likely wired in a daisy-chain, the easiest way to mitigate this problem is to split the load into two 10-amp circuits and just run one new similar-size cable to the half of the circuit that was disconnected during the split.

This issue could also be mitigated by lowering the load to this circuit, either by removing 1/10th of the bulbs on that circuit or by lowering the wattage of the bulbs (such as replacing them with LED tubes). Either way, the faulty breaker must be replaced.

The faulty breaker has been marked in red and MUST be disposed of completely. It should be destroyed to prevent re-use.

ii. There is a GFCI-style device in this panel which affects multiple sub-circuits. This device is likely required due to floor outlets in the library next door. This device is tripping regularly, possibly due to refrigerators and other random small machine loads on the sub-breakers. This GFCI-style device has been verified as bad.

This GFCI-style device should be replaced as it is no longer functioning properly.

To prevent future breaker trips, segregate the load which requires GFCI protection from the load which does not. Provide large circuits with adequate headroom for both loads.

c. Room 123, MHS

This is the main electrical room in the MHS. The power cables behind the panels are poorly organized and the backs of the panels are literally hanging on, with no fasteners installed.

There is writing on and around many of the breakers, and one breaker is taped open (off) with no explanation. There are some in-panel surge protectors here with a 15ka rating, but that appears similar to what one might expect to see in a household.

Main breaker is 380 volt, 3-phase, 630-amp breaker, drawing approximately 200, 200, and 180 amps on phases 1, 2, and 3, respectively.

The breaker labeled “Generale F.M. Segreteria” reportedly trips often. This 3-phase breaker has three sub-breakers, none of which appear to be drawing anywhere close to their peak current rating. This has a GFCI breaker on it, likely due to local codes requiring any area with a sink to have a GFCI breaker. One of the areas that loses power when this breaker trips is the kitchen area in the front office. Startup current and motor loads such as those from refrigerators and microwaves often trip these GFCI breakers.

To fix this issue, separate the load so that only the receptacles requiring a GFCI breaker are protected with a GFCI breaker, and every other receptacle and device is protected only with a standard breaker.

8. The Way Forward

a. Ask personnel from the schools to document future issues. Write down the exact breakers that trip and when. Also document any other circumstances that seem pertinent. While we have found and can fix many of the internal wiring issues, there will likely be more that have not yet been found. By fixing them one at a time as they are discovered, they will soon be fixed.

b. Government should specify what size circuits to run for each work order to the contractor.

Intuition and past experience lead this surveyor to believe that improper circuit combinations and undersized circuits are often run as a matter of convenience because the materials necessary for smaller circuits are on-hand and easier to implement than ordering appropriate materials and doing a job correctly. The good news is that most circuits seem to follow code and are therefore mostly safe, but this does not make them appropriate.

c. Government should provide oversight to contractor installations to ensure no further shortcuts are taken.

d. Upon implementation of providing server-room quality backup power as mentioned in 3a, above, serious consideration should be given to upgrading TR quality in terms of downstream power distribution, room layout, and cabinet organization.

e. From talking to personnel on site, it seems that whole-facility power outages are no longer as common as they were last summer; they could be seasonal. This situation should be monitored closely as the whole-facility power solution mentioned in 3c, above, is considered.