25-RFP-26_SCADA_Support_Maintenance_Upgrade_and_Repair_Services_-_Attachment_B_SCADA_Master_Plan.pdf

City of Palm Bay, Florida

February 2015

SCADA

Master Plan

ATTACHMENT B

SCADA MASTER PLAN

PALM BAY UTILITIES DEPARTMENT

PREPARED FOR:

PALM BAY UTILITIES DEPARTMENT

250 OSMOSIS DRIVE SE

PALM BAY, FLORIDA 32909

PREPARED BY:

3790 DIXIE HIGHWAY NE, SUITE D

PALM BAY , FLORIDA 32905

FEBRUARY 2015

PBU20 73 .01 L

SCADA Master Plan

Palm Bay Utilities Department

PBU2073. 01L i February 2015

TABLE OF CONTENTS

TABLE OF CONTENTS ............................................................................................................................ i

LIST OF FIGURES .................................................................................................................................. iii

LIST OF TABLES .................................................................................................................................... iii

LIST OF APPENDICES ( DVD) ................................................................................................................. v

1.0 INTRODUCTION

1.1 Background

1.2 Supervisory Control and Data Acquisition ( SCADA)

1.3 Facilities

1.4 Goals and Objectives

1.5 SCADA Master Plan Sections

2.0 EXISTING SCADA SYSTEM

2.1 SCADA System Overview

2.2 SCADA Software

2.2.1 Introduction

2.2.2 Human -Machine Interface

2.2.3 HMI Application Screens

2.2.4 Historian and Data Backup

2.2.5 Software Versions and Security Keys

2.2.6 Software Disaster Recovery Plan

2.2.7 Tag Name Convention

2.2.8 Trending

2.2.9 Automatic and Demand -Driven Reporting

2.2.10 Regulatory Reporting

2.3 SCADA Workstations, Servers and Hardware Standards

2.3.1 Workstations

2.3.2 Servers

2.3.3 Redundancy

2.3.4 Replacement Plan

2.3.5 Requirements and Standardization

2.3.6 Disaster and Recovery Plan

2.4 Communications Network

2.4.1 Overview

2.4.2 Computer and PLC/RTU Network

2.4.3 Communication Protocols

2.5 Field Equipment – PLCs, RTUs & Master Radios

2.5.1 Overview

2.5.2 Standard Equipment

2.5.3 PLC Control Configuration

2.5.4 Equipment Description and Assessment

2.5.5 North Regional RO WTP

2.5.6 North Regional Lime Softening WTP

2.5.7 Master Radio at Water Tower

2.5.8 North Regional WWTP

2.5.9 North Regional Deep Injection Well

2.5.10 North Regional WRF

PBU2073. 01L ii February 2015

2.5.11 South Regional Water Treatment Plant

2.5.12 Aquifer Storage & Recovery Well

2.5.13 Raw Water Wells

2.5.14 Nash Potable Water Re -Pump Station

2.5.15 Lift Stations – Standard RTU Configuration

2.5.16 Lift Stations – Data Flow Systems ( DFS)

2.6 General Security of SCADA System

3.0 HARDWARE & SOFTWARE EVALUATION

3.1 Introduction

3.2 Standardization of PLCs and RTUs

3.3 Lift Stations – DFS vs Portable iFIX Panel

3.4 Energy Monitoring & Management

3.4.1 Background

3.4.2 Hardware Requirements

3.4.3 Cost of Implementation

3.4.4 Monitoring Locations

3.4.5 Smart MCCs

3.5 Water and Wastewater Pipe Monitoring

3.6 Wetlands Monitoring Wells

3.7 Security Camera Network

3.8 SCADA Software

3.9 Software Request for Information ( RFI) Package

4.0 CAPITAL IMPROVEMENT PROJECTS

4.1 Projects

4.2 Project Implementation

4.3 Future Evaluations and Documentation of System Upgrades

PBU2073. 01L iii February 2015

LIST OF FIGURES

Figure 1 -1. Facility Location Map

Figure 1 -2. North Regional Treatment Campus

Figure 1 -3. South Regional Treatment Campus

Figure 2 -1. North Regional WWTP and SRWTP SCADA Application Screens

Figure 2.2. Typical Tag Names – ASR Well and Lift Stations

Figure 2.3. Typical Tag Names – Lime Softening WTP

Figure 2.4. Typical Tag Names – SRWTP

Figure 2 -5. Typical Trending Screen – Pressures in Water Main

Figure 2 -6. SCADA Network Configuration – North Regional Treatment Campus

Figure 2 -7. SCADA Network Configuration – South Regional Treatment Campus

Figure 2 -8. North Regional Lime Softening WTP and RO WTP PLC Locations

Figure 2 -9. WTP Network Switches, Computers, and Security Camera Equipment

Figure 2 -10. North Regional WWTP and DIW PLC Locations

Figure 2 -11. North Regional WRF PLC Locations

Figure 2 -12. SRWTP PLC Locations (North)

Figure 2 -13. SRWTP PLC Locations ( South)

Figure 2 -14. ASR Well and Raw Water Well # 15 PLC/ RTU Locations

Figure 2 -15. Nash Water Re -Pump Station PLC/RTU Location

LIST OF TABLES

Table 1 -1. Facility Listing

Table 2 -1. Computer Access to SCADA Software

Table 2 -2 . North Regional RO WTP System Summary

Table 2 -3. North Regional RO WTP PLC

Table 2 -4. North Regional Lime Softening WTP System Summary

Table 2 -5. North Regional Lime Softening WTP PLC

Table 2 -6. North Regional Lime Softening WTP Filter #1 PLC

Table 2 -7. North Regional Lime Softening WTP Filter #2 PLC

Table 2 -8. North Regional Lime Softening WTP Filter #3 PLC

Table 2 -9. North Regional Lime Softening WTP Filter #4 PLC

Table 2 -10. North Regional WTP Master Radio

Table 2 -11. North Regional WWTP System Summary

Table 2 -12. North Regional WWTP Headworks RTU Panel

Table 2 -13. North Regional WWTP BFP PLC

Table 2 -14. North Regional WWTP RAS RTU

Table 2 -15. North Regional WWTP Maintenance Building Chart Room PLC

Table 2 -16. North Regional WWTP Operations Room RTU Panel

Table 2 -17. North Regional WWTP Chart Room Master Radio

Table 2 -18. North Regional Deep Injection Well Summary

Table 2 -19. North Regional WWTP Deep Injection Well RTU Panel

Table 2 -20. North Regional WRF System Summary

Table 2 -21. North Regional WRF Blower Building ( Main) PLC

Table 2 -22. North Regional WRF Disk Filter #1 Panel (Typical for Disk Filter #2 Panel)

PBU2073. 01L iv February 2015

Table 2 -23. North Regional WRF Reclaim HSP PLC Panel

Table 2 -24. North Regional WRF Sodium Hypochlorite System PLC Panel

Table 2 -25. SRWTP System Summary

Table 2 -26. SRWTP I/O -1 PLC

Table 2 -27. SRWTP I/O -2 PLC

Table 2 -28. SRWTP I/O -3 PLC

Table 2 -29. SRWTP I/O -4 PLC

Table 2 -30. SRWTP I/O -5 PLC

Table 2 -31. SRWTP I/O -6 PLC

Table 2 -32. SRWTP I/O -7 PLC

Table 2 -33. ASR Well Summary

Table 2 -34. ASR Well RTU

Table 2 -35. Raw Water Well Summary

Table 2 -36. Raw Water Well #15 RTU

Table 2 -37. Nash Water Re -Pump Station System Summary

Table 2 -38. Nash Re -Pump Station RTU

Table 2 -39. Lift Station Standard RTU System Summary

Table 2 -40. Typical Standard Lift Station PLC Panel

Table 2 -41. Lift Station DFS System Summary

Table 2 -42. Typical DFS Lift Station TCU Panel

Table 3 -1. PLC Comparison

Table 3 -2. PLC Cost Comparison

Table 3 -3. DFS and iFIX Removable Panel Cost

Table 3 -4. Energy Monitoring Cost

Table 3 -5. Summary of Large Motors

Table 3 -6. Force Main Pressure & Water Main Chlorine Residual Monitoring Locations

Table 3 -7. Collections and Distributions Monitoring Installation Costs

Table 3 -8. Wetland Monitoring Well SCADA Cost

Table 3 -9. SCADA Software Package Pricing

Table 3 -10. SCADA Software Comparison

Table 4 -1. Capital Improvement Projects

Table 4 -2. CIPs – Projects without SCADA Programming Effort

PBU2073. 01L v February 2015

LIST OF APPENDICES ( DVD)

Appendix A. Supplemental Information

Listing of SC ADA Software Keys, Spreadsheet Printout, 6/12/ 2012

Listing of SCADA Software Keys, Gray Matter Systems, 6/15/ 2012

Summary of SCADA Software Keys in Use, Santis Engineering, 10/ 25/ 2013

Appendix B . Summary of SCADA System, Prepared by Santis Engineering, Inc

October 25, 2013)

SCADA System Spreadsheet o HMI and Key Listing o PLC Types and I/O o Network Addressing

SCADA System Drawings o HMI o PLC Types and I/O o Network Addressing

USB Thumbdrive o PLC Program Backups ( as pdf) o IFIX HMI Project Backups o Panel Drawings o Netwo rk Drawings

Appendix C . Summary of SCADA System, Supplemental, Prepared by Santis Engineering, Inc

December 17, 2013)

SCADA Network Architecture Plans

PLC Programs

Pictures

Appendix D . Summary of SCADA System, Supplemental 2, Prepared by Santis Engineer ing, Inc

December 30, 2013)

SCADA Databases and PLC Configuration Printouts

Appendix E . SCADA Application Screens

WWTP and WRF

Lime WTP and RO WTP

Lift Stations and Wells

SRWTP

Appendix F . Summary of Control and Monitoring Functions

WWTP and WRF

Lime WT P and RO WTP

Lift Stations and Wells

SRWTP

Appendix G . Cost Estimate Detail

PBU2073. 01L 1 February 2015

1.0 INTRODUCTION

1.1 Background

The City of Palm Bay Utilities Department ( PBU D ) is responsible for the operation and maintenance of multiple water and wastewater facilities, including two

2) wastewater treatment plants, three ( 3) water treatment plants, 49 water wells ( 43 surficial; 6

Floridan), and over 110 sanitary lift stations. The first facilities to service the City were constructed in the

1960s while other facilities were cons tructed as recent as 2006.

The oldest treatment facilities , located in northeast

Palm Bay , are found at the North Regional Treatment

Campus (formerly known as the Troutman Campus ), includes the 4.0 MGD Wastewater Treatment Plant

WWTP), 1.2 MGD Water Reclamation Facility ( WRF), 10 MGD Lime Softening Water Treatment Plant

WTP) and the 1.5 MGD Reverse Osmosis Water

Treatment Plant ( RO WTP). The South Regional

Treatment Campus , located towards the center of the

City, was constructed in 2006 and is home to the 4.0

MGD South Regional Water Treatment Plant

SRWTP). Additional facilities, including the Aquifer

Storage & Recovery Well and Nash Re -pump Station are located in northern Palm Bay.

Three ( 3) figures have been provided at the end of this Section. Figure 1 -1 provides the general location of each facility. Figure 1 -2 provides the layout of the North Regional Treatment Campus and

Figure 1 -3 provides the layout of the South Regional Treatment Campus

1.2 Supervisory Control and Data Acquisition ( SCADA)

Prior to the installation of control and monitoring technologies at the facilities, the PBUD relied upon audible alarms to notify staff of issues throughout the system. As the City increased in population and size, the number of remote facilities (primarily lift stations) began to increase. The PBUD realized there was a growing need to be able to monitor the facilities’ status remotely, diagnose the cause of alarms and warnings prior to going on service calls, and control the facilities remotely during alarm situations and wet weather events.

In response to these system demands, the Supervisory Control and Data Acquisition ( SCADA) software, Intellution iFIX (now known as Proficy iFIX), was installed in 1993 and used to monitor the lift stations via computer screen applications at the North Regional Treatment Campus. The SCADA system was further expanded in 2003 to provide centralized control and monitoring capabilities at the

North Regional WWTP. To date, all of the treatment facilities and remote facilities are integrated with the SCADA system.

PBU2073. 01L 2 February 2015

As each facility was added to the SCADA system, varying degrees of automated control and monitoring were provided. The control of the connected components is achieved by using a combination of

Programming Logic Controllers ( PLCs) and Remote Telemetry Units ( RTUs) that communicate with the

SCADA system via microwave, radio, and/or fiber optic broadband.

Since the initial SCADA installation at the North Regional WWTP , PBUD ’s SCADA system has evolved over the years from multiple projects and acquisitions , which has resulted in a system of mixed hardware, software, applications, protocols, and implementations. Having no concise vision for the overall integration of the system, the PBUD has a greater chance of providin g disjointed or incompatible hardware and software as repairs and upgrades are made. The PBUD has been diligently working to improve the functionality with the goal of having a SCADA system that is consistent, logical and simple to use.

1.3 Facilities

The SCADA system presently connects all of PBUD ’s facilities, including the treatment plants, lift stations, remote pump stations, and water wells. A summary of the facilities and the approximate date

SCADA technology was installed has been provided in Table 1 -1 .

Table 1 -1 . Facility Listing

Facility SCADA Installation Date

North Regional Wastewater Treatment Plant ( WWTP) 2003

North Regional Water Reclamation Facility ( WRF)

2005 – Main Plant

1999 – Reclaim High Service Pumps

2009 – Disk Filters

North Regional Lime Softening Water Treatment Plant ( WTP) 2012

North Regional Reverse Osmosis Water Treatment Plant ( RO WTP) 2006

South Regional Water Treatment Plant ( SRWTP) 2004

Aquifer Storage and Recovery Well (ASR) 2005

Nash Water Re -Pump Facility 2007

Raw Wate r Wells 2007

Sanitary Lift Stations – Standard Install 1993 to Present

Sanitary Lift Stations – Data Flow Systems ( Pilot Program)* 2012

Refer to Section 2.5.16 for more information regarding Data Flow Systems ( DFS)

In addition to the facilities identified in Table 1 -1 , the SCADA system is set to monitor the chlorine residual ( in mg/L) and system pressure ( in PSI) in the water distribution piping near seven lift stations

LS -36, -44, -58, -67, -90, -93, and - 99), two remote locations ( 1 -Pace/Delano; 2 -Raleigh/Sturbridge) and the Nash Water Re -Pump Facility.

1.4 Goals and Objective s

The SCADA system is over 30 years old and many of the hardware and software components of the system are old, outdated and/or not completely functioning as intended. The PBUD is ready to invest in modernizing the SCADA system with the ultimate goal of having a product that is consistent, logical and

PBU2073. 01L 3 February 2015 straight forward to use. Once the system is inherently intuitive to all users, the Utilities staff will have confidence in using the SCADA system as an effective tool in controlling, monitoring and maintaining the facilities.

The purpose of this Master Plan is to provide a complete overview of the existing SCADA system, to document the PBUD’ s vision for upgrading the SCADA system and to develop an implementation plan that meets the needs of the PBUD. The primary objectives of this Master Plan have been summarized as follows:

Identify software and hardware deficiencies and provide recommendations for corrective action;

Evaluate various SCADA software packages to determine if another product is more capable of meeting the PBUD’ s needs;

Evaluate additional hardware and software SCADA tools, such as energy management technologies that can track equipment performance and usage in an effort to reduce energy, maintenance and operational costs; and

Prioritize improvements in an effective manner ( i.e. resolve software issues prior to making optional hardware changes).

The SCADA Master Plan will include recommendations to be used for both renovations to existing systems as well as for new installations to ensure that future hardware, software, and system design will comply with the new SCADA system plan.

1.5 SCADA Master Plan Sections

This SCADA Master Plan has been prepared in four sections. Section 1.0 provides the basic background of PBU’s SCADA system and the goals and objectives of this Master Plan. Section 2 .0 documents the components of the existing SCADA system , determines how City staff utilizes it , and evaluates the existing hardware, software, and design configuration. Section 3.0 evaluates SCADA hardware and software components and provides recommendations such as standardizing on specific equipment, adding energy monitoring & management capabilities , and investigating alternative SCADA software that may more closely meet the PBUD’s operational goals. Finally, Section 4 .0 provides costs and a proposed sequence of implementation of new hardware and software necessary to fulfill the City’s vision.

FACILITY LOCATION MAP

PALM BAY UTILITIES

BA

B C O

C K

S T

MI

N T

O

N

R D

MALABAR RD

D I

X

I

E

H

W

Y

GRANT

RDDE

G

R O

O

D

T

R D

W

E B

E

R

R D

MICCO RD

OSMOSIS DR

C

O

G

A

N

D R

S A

N

FILIP

P O

D R

ST. ANDRE BLVD

S

A

N

F

I

L

I

P

P

O

D

R

O

A

K

S

T

EBER BLVD

B

A

B

C

O

C

K

S

T

L

I

P S C

O

M

B

S T

U

S

H

W

YM

I

N

T

O

N

R D

D

A I

R Y

R

D

E

L

D R O

N

B L V

D

E

M

E R

S

O

N

D

R

PALM BAY RD

JUPITER BLVD

VALKARIA RD

MALABAR RD

MELBOURNE

BEACH

MELBOURNE

MALABAR

GRANT-

VALKARIA

BREVARD COUNTY

INDIAN RIVER COUNTY

Indian

River

CITY OF PALM BAY, FLORIDA

2015 SCADA MASTER PLAN

ºPAGE

4 FIGURE 1-

1 Nash

Water Repump

Station North Regional

Treatment Campus ( Formerly

Troutman) ASR

Well South

Regional Treatment

Campus Source: City of Palm Bay, Florida, GIS Dept., 2012; Wade Trim, Inc., 20140120.

NORTH REGIONAL TREATMENT CAMPUS

CITY OF PALM BAY, FLORIDA

2015 SCADA MASTER PLAN

ºPAGE

5 FIGURE 1-

2LIME

SOFTENING WTP & RO

WTP Source: City of Palm Bay, Florida, GIS Dept., 2012; Wade Trim, Inc., LEGEND Fiber Optic

Cable Facility

Boundary0300

150Feet

WWTP DEEP INJECTION

WELL

WRF ASR

SOUTH REGIONAL TREATMENT CAMPUS

CITY OF PALM BAY, FLORIDA

2015 SCADA MASTER PLAN

ºFIGURE 1-

3 Source: City of Palm Bay, Florida, GIS Dept., 2012; Wade Trim, Inc., O

S

M

O

S I

S

D

R RAW WATER

WELL0150300

75Feet PAGE

6 FUTURE SRWRF

SITE DEEP INJECTION

WELL HOLDING

POND RAW WATER

WELL

SRWTP

ADMINISTRATION

BUILDING RAW WATER

PBU2073. 01L 7 February 2015

2.0 EXISTING SCADA SYSTEM

2.1 SCADA System Overview

SCADA refers to centralized systems which monitor and control processes ( locally and remotely) such as power plants, oil/gas refineries, telecommunications and water/wastewater treatment facilities.

Control functions through SCADA are usually restricted to basic overriding or supervisory level intervention while control actions are typically performed automat ically by Programming Logic

Controllers ( PLCs) or Remote Telemetry Units ( RTUs ) at the unit process ( equipment). For example, a

PLC may control the flow of mixed liquor ( sludge) to an aerobic digester , but the SCADA system may allow operators to change the set points for the flow, and enable alarm conditions, such as loss of flow and high fluid level in the digester , to be displayed and recorded. The feedback control loop passes through the PLC or RTU while the SCADA system monitors the overall performance of the loop.

Data acquisition begins at the PLC and RTU level and includes various readings and equipment status reports that are communicated to SCADA as required. Data is then compiled and formatted in such a way that a control room operator using the Human -Machine Interface ( HMI) can make decisions to adjust or override normal controls at the PLC or RTU . Data is typically sent to a Historian (software database application) to allow trending and other analytical auditing.

SCADA systems typically impleme nt a tag database which contains data elements called tags or points. A point represents a single input or output value monitored or controlled by the system. A “hard point ” represents an actual input or output within the system . A “soft point ” results from logic and math operations applied to other points. Points are normally stored as a value with a timestamp when it was recorded or calculated. A series of value -timestamp pairs gives the history of that point. It is also common to store additional metadata with tags, such as the path to a field device or PLC register, design time comments, and alarm information.

In 1993, the existing SCADA system was introduced to control and monitor the lift stations and has been expanded over the years to the other facilities, resulting in a SCADA system that runs different versions of software, hardware and communication protocols. The following subsections will describe and provide an assessment of the components that make up the SCADA system. The assessment is based on on -site inspections, existing plans, and information provided by the PBUD and the current

SCADA Integrator (Santis Engineering Inc.).

2.2 SCADA Software

2.2.1 Introduction

The SCADA system is comprised of software that is used to control and monitor the system while also maintaining a database of the activities of the facilities. The following will detail the software components and requirements needed to maintain a healthy SCADA system.

2.2.2 Human -Machine Interface

Since the inception of the SCADA system, the PBUD has utilized Proficy iFIX ( formerly Intellution) software to serve as the Human -Machine Interface ( HMI). The SCADA HMI is the input -output ( I/O) device /software through which the operators control the processes and which presents process data to the operators . The HMI is linked to the SCADA system' s databases and software programs to provide trending, diagnostic data, and management information . The HMI is accessible through PCs with the

PBU2073. 01L 8 February 2015 proper access credentials ( USB or software key) or through speci al view screens located at/near the

PLCs ( i.e. RO Unit SCADA Node).

The HMI software presents the information to the facility operators graphically as a mimic diagram. This means that the operator can see a schematic representation of each facility . The mimic diagrams consist of a combination of line graphics and schematic symbols to represent process elements and photographs of the process equipment overlain with symbols and parameters .

2.2.3 HMI Application Screens

Since the SCADA system has been implemented facility by facility over the years, the various HMI applications were created by multiple programmers; inconsistencies exist in screen appearance, navigation, objects, control functions and alarm displays. This is apparent when comparing the older

WWTP screens (left) to the more current SRWTP screens (right), as demonstrated in Figure 2 -1 .

Figure 2 -1 . North Regional WWTP and SRWTP SCADA Application Screens

Though utilizing the same software package, the functionality and screen configurations are very different and have varying degrees of functionality. Additionally, there are multiple screens in which obsolete/decommissioned equipment is depicted on the screens. Several screens, primarily for the

WWTP, are registering errors and not providing output. At this time, there is not a rebuild policy to remove obsolete screens or address corrupted/ inoperable screens. When an issue occurs with the

SCADA application, the contract SCADA integrator is contacted to assist with correcting the problem.

2.2.4 Historian and Data Backup

Information collected by the SCADA system is sent to the Historian. A Historian is a software service which accumulates time -stamped data, events, and alarms in a database which can be queried or used to populate graphic trends and generate reports in the HMI. The Historian compresses the data as it is received in order to store as much as possible on the computer servers’ hard drive. The quantity of data stored is a function of the size of the hard drive, quantity of I/O information being stored, and the frequency of the data sampling.

The PBUD ’s SCADA system uses Proficy iHistorian software for the database. The data collected by the Historian can be used within iFIX or can be exported as a comma -separated value (CSV) file format for use in spreadsheet software such as Microsoft Excel.

PBU2073. 01L 9 February 2015

Routine software maintenance is not performed on the PBUD’ s Historian. The database should be reviewed a minimum of once a year to insure that there are no conflicts with database entrie s and to remove any entries relating to tags which are no longer being used. The database should also be reviewed after any major upgrades to the application have been made.

2.2.5 Software Versions and Security Keys

Software versioning is a standard process of assigning version names or numbers to revisions in a piece of software. Versions that are released can vary in complexity from minor changes that would be unnoticeable to the end user ( software code changes) to major changes that would require additional effort to implement and learn ( functional changes to interface). Depending upon the service agreement with the software vendor, updates are provided for free or for a fee through continuing maintenance contracts. The most current version of the Proficy iFIX software that has been released by the vendor

General Electric) as of December 2014 is iFIX 5.8 (HMI) and iFIX 5.5 (iHistorian).

The standard practice of PBUD has been to purchase the iFIX software as needed and to not upgrade as new versions are made available. This has resulted in the PBUD owning varying versions of the same software. Per documentation provided by Gray Matter Systems ( GE distributer) via e -mail (June

12, 2012), the earliest version that the PBUD pays for is iFIX ( HMI) 2.1. Addit ional versions of iFIX

HMI) that have been identified are iFIX ( HMI) 2.21, 3.0, 3.5, and 4.0. The version of iHistorian that is in use is 3.1. Refer to Section 2.3 .1 for the iFIX locations and version numbers that are actively used.

It is very importan t for the PBUD to consider upgrading to the latest version of iFIX or replacing the software package with a comparable product across all facilities. The benefits are many, including:

Consistent Programming and Licensing – PBUD personnel will only need to learn one version of the software. The system’ s integrator will be able to streamline programming and ensure HMI applications are the same . All licensing will be consistent.

Functionality – Newer software versions feature programming improvements, including simplified coding, more efficient computing, more robust displays ( 3D, HD), and better compatibility with third -party applications and add -ons.

Operating System Compatibility – The Microsoft Windows Operating System ( OS) is the only

OS in use by the PBUD. Earlier iFIX software versions are not compatible with the newer versions of Windows. For instance, iFIX 3.5 is compatible with Windows XP OS; however, it cannot be transferred to a workstation utilizing the Windows 7 OS. It is important to upgrade to newer software as Microsoft officially ended extended support of Windows XP OS in April 2014;

thus making the OS obsolete and susceptible to security risks.

Security – Security flaws, risks, and other issues are typically addressed as versions of software are released. New software is specifically programmed with current threats in mind, including the advent of cyber -attacks on public and private networks. Newer software versions support strong encryption and the ability to run in a sandbox mode ( blocking of potentially dangerous commands).

In order to gain access to the iFIX HMI from a computer, security keys (USB dongles) are required.

There are 12 iFIX security keys in use by the City; however, the City may be paying a license fee for up to 20 security keys as per Gray Matter System’ s database. The security keys are linked to the iFIX versions that are in use.

PBU2073. 01L 10 February 2015

The iFIX keys and the associated serial numbers that are currently in use have been identified in

Appendix A . It is recommended that PBUD provide Gray Matter Systems this list to begin the process of eliminating unused and unneeded iFIX keys.

2.2.6 Software Disaster Recovery Plan

A software disaster recovery plan is a documented process or set of procedures to recover and protect the IT infrastructure in the event of a disaster. The plan is usually documented in written form and specifies procedures to follow in the event of a critical software system’ s failure ( disaster ). It is a comprehensive outline of actions to be taken before, during and after the disaster.

A written document has not been created to outline the required procedures to be taken in the event of a major failure of SCADA system software. As of now, once a failure has been recognized, maintenance personnel troubleshoot the issue to determine if they are able to correct the problem in -house. If the software problem involves a PLC or RTU, maintenance personnel can reinstall a copy of the program back into the processor. If the problem involves iFIX or the iHistorian, the system’s integrator is contacted to troubleshoot. A contractor may be contacted to assist with correcting the problem.

2.2.7 Tag Name Convention

A SCADA tag ( I/O point) is a link to the memory location in a PLC or RTU that is associated with each parameter that is controlled or monitored at the equipment or process. These tags are given names in order to properly identify the parameter.

The tag naming convention throughout the SCADA system is varied and depends upon the facility and the system’ s integrator that programmed the tags. Figure 2.2 provides a listing of the tags utilized at the ASR Well and lift stations. The tags are heavily abbreviated and may not be clear; however, a description column is provided to clarify the purpose of the tag.

Figure 2 .2 . Typical Tag Names – ASR Well and Lift Stations

Figure 2.3 provides a listing of the tags utilized at the Lime Softening Water Treatment Plant. In this case, the tags are less abbreviated but clearly define the item being monitored.

PBU2073. 01L 11 February 2015

Figure 2 .3 . Typical Tag Names – Lime Softening WTP

Figure 2.4 provides a sample of the tags used at the South Regional Water Treatment Plant. The tags are a combination of codes ( i.e. ZCS -711) and codes combined with descriptions ( i.e. RC -1601

Antiscalant # 1 Start / Stop Command). The mix of tags is slightly confusing and clutters the tag listing.

Figure 2.4 . Typical Tag Names – SRWTP

Though the tag names are varied, the descriptors are sufficient to identify the parameters. No issues have been identif ied regarding the tag naming convention; therefore, it is not recommended to replace or change the tag names at this time. It is recommended; however, that the PBUD implements a standard tag naming convention for future PLC and facility installations.

2.2.8 Trending

The tag data that is collected from the system and stored in the iHistorian software database can be queried and displayed within the HMI as a time -lapsed graph. From this graph, trends within the data can be identified, such as fluctuations in pressures, increasing pressures, and decreasing chlorine residuals, that can assist the facility operators in identifying potential issues in the system.

For process data that an operator wishes to view regularly, a trend template can be created and stored.

This template can then be easily accessed in order to view the most recent data as a trend. Temporary trends can also be created in the event that an operator wishes to view seldom used data or to compare several different process trends together in order to establish relationships between data.

Figure 2 -5 demonstrates the trending screen used by the water treatment plant operators to monitor the water main pressures at specific points in the system.

PBU2073. 01L 12 February 2015

Figure 2 -5 . Typical Trending Screen – Pressures in Water Main

2.2.9 Automatic and Demand -Driven Reporting

As demonstrated in Section 2.2 .8 , the Historian can be used to generate graphical trends in the system for a given parameter. The Historian can also be queried manually ( i.e. demand -driven) to generate reports or can be set to automatically generate reports as required by the facility personnel . This process is currently being achieved by use of a program called XLReporter, which is an interface between Proficy iFIX and Microsoft Excel.

At this time the only report presently being generated automatically on the wastewater collection and treatment systems via iFIX is the lift station pump runtime report. The facility staff has stated there is a three -day delay in obtaining pump runtimes. This is suspected to be a report configuration issue and not a limitation of the iFIX software.

2.2.10 Regulatory Reporting

The PBUD is responsible for providing the Florida Department of Environmental Protection ( FDEP) with monthly reports in order to comply with the water and wastewater facilities permits. Drinking Water statistics are provided on Monthly Operating Reports ( MOR) while the Domestic Wastewater statistics are provided on Discharge Monitoring Reports ( DMR). Data is collected from the field or measured through SCADA and each report is populated.

The PBUD has expressed a desire to complete FDEP compliance forms through iFIX . This can be accomplished by integrating third -party software, such as Hach WIMS ( Water Information Management

Solution), into iFIX. The software is designed to collect the required data from the Historian, calculate and convert the data as necessary, and populate the form templates. Data that is not collected through iFIX, such as wastewater effluent fecal coliform testing results, can be manually added to the forms.

2.3 SCADA Workstations, Servers and Hardware Standards

2.3.1 Workstations

The City utilizes workstations ( PCs) running the Microsoft Windows OS . Each computer ( view node) that has access to SCADA is using unique Proficy iFIX graphic user interface software keys and USB

PBU2073. 01L 13 February 2015 dongles . Mobile iFIX licenses have not been provided; therefore, laptops, tablets and other mobile devices do not have access to the SCADA system. The workstations that were identified to have access are described in Table 2 -1 .

Table 2 -1 . Computer Access to SCADA Software

Facility Building Location Function iFIX Name Version

North

Regional

WWTP

A *

F

G

Maintenance

Lab

Electrical Office

View Node

View Node

SCADA Node

TR -V1

TR -V2

TR -P3

3.5

2.21

3.5

North

Regional

WTP

H

H

H

H

H

Lab

Lab

Lab

RO Office

Superintendent’ s Office

Primary SCADA Node

LS & Telemetry SCADA Node

WTP SCADA Node (Touch)

RO Unit SCADA Node

View

TR -P1

TR -P2

TR -P4

TR -S1

None

3.5

3.5

3.5

3.5

3.5

SRWTP

Control Room

Control Room

Control Room

Control Room

Control Room

Primary SCADA Node

Secondary SCADA Node

View Node

Terminal Server

Historian Server

SR -P1

SR -S1

SR -V1

Riga

Belgrade

4.0

4.0

3.5

4.5

3.1

The Dataflow System ( DFS) server and view nodes are located in Building A.

A full assessment of the workstations was not performed since the City’s Communications and

Information Technology Department ( CIT ) has a regular replacement program for computers. They are replaced every three (3) years. The workstations used to run iFIX are using the Windows XP OS. The workstation in the WWTP Operations Room was updated in June 2014 to use Windows XP; however, iFIX at this location ( version 2.21) must be run in Windows Compatibility Mode (as Windows 98). For security purposes, this workstation should be replaced with a modern version of iFIX, a new workstation and a more current operating system.

2.3.2 Servers

South Regional Treatment Campus

There are two types of hardware servers located at the South Regional Treatment Campus: a terminal server and a Historian server.

The terminal server is a hardware device that provides terminals ( PCs, printers, and other devices) with a common connection point to a local or wide area network. The terminal server has the iFIX software running on it. Other City computers that have access to the network and have been granted rights to view the iFIX graphic screens are able to do so.

The Historian Server stores the data that is collected from the iHistorian software that was described in

Section 2.2 .4 . The Historian server has the iHistorian software running on it. The Historian server archives data that is linked to tags. The Historian server archives the data at “snap shot” time intervals;

the desired time interval for the frequency of data collection ( i.e. once every second) is input to the iHistorian software and can be changed at any time. The iHistorian software will archive the data until

PBU2073. 01L 14 February 2015 the server hard drive is full, at such time the oldest data will be overwritten with new data. Therefore, the data stored on the Historian server should be backed up to long term storage, such as backup tapes.

North Regional Treatment Campus

The iFIX system at the North Regional Treatment Campus is an older system and is configured differently. This Campus does not have a dedicated Historian server with the iHistorian software;

rather, the primary SCADA PC (node) located in the Lime Softening Lab Room collects the data.

Assessment

A full assessment of the servers and switches was not performed since CIT has a regular replacement program. Modern computer equipment is more than capable of supporting the functionality of the

SCADA system.

2.3.3 Redundancy

Redundancy within the SCADA system is necessary in order for it to remain operational in the event of a hardware or software failure. The existing SCADA system has varying levels of redundancy as described below:

North Regional Treatment Campus o The PCs at the WWTP and WTPs are all on the same SCADA network. In the event of a PC failure, such as a hard drive failure, another PC can be used to conduct operations.

o The Lime Softening WTP has additional redundancy since it can be controlled from the

PC in the Lab Room, the PC mounted in the door of the Filter Control Panel, the PC in the RO WTP, and the PCs at the SRWTP .

South Regional Treatment Campus o There are no redundant servers for the Terminal Server or the Historian Server. These servers do not run the plant processes. The operator controls are performed from other computers in the Control Room. If the Historian server crashes, then data collection stops, but it does not stop the processes. If the Terminal Server crashes, users with view access will not be able to view the application screens.

o In the event of a failure of the workstations in the control room, the South Regional

Treatment Campus can be operated from the North Regional Treatment Campus.

2.3.4 Replacement Plan

The SCADA workstations and servers are managed by CIT . Workstations are replaced every three ( 3) years. The servers are replaced as needed . Other server communication equipment , such as network switches , is replaced upon failure, old age, or lack of sufficient memory.

2.3.5 Requirements and Standardization

The workstations and servers are typical Dell business -class models. The network switches are Nortel

PBU2073. 01L 15 February 2015

Networks “ Baystack” models. Since most modern computer hardware is compatible with SCADA systems, the standardization of this type of equipment is controlled by CIT .

2.3.6 Disaster and Recovery Plan

Similar to the Software Disaster and Recovery Plan described in Section 2.2 .6 , a written document has not been created to outline the required procedures to be taken in the event of a major failure of a

SCADA system hardware component. As of now, once a failure has been recognized, maintenance personnel troubleshoot the issue to determine if they are able to correct the problem. If not, CIT is contacted to assist with correcting the problem.

2.4 Communication s Network

2.4.1 Overview

The SCADA system is a mixture of communication networks and protocols. Radios are used to communicate between most of the WWTP PLCs; they are also used by remote facilities such as the lift stations and the Nash Re -Pump Station. Fiber optic cabling is used to connect the North Regional facilities to each other. All PLCs located at the South Regional Treatment Campus are connected through fiber optic cabling. A combination of microwave and fiber optic broadband is used for the North and South Regional facilities to communicate to each other.

2.4.2 Computer and PLC/ RTU Network

The SCADA computers and servers as previously shown in Table 2 -6 are interconnected with the various PLCs and RTUs at each facility and share a common network. Figure 2 -6 illustrates the basic

SCADA network configuration for the North Regional Treatment Campus while Figure 2 -7 shows the basic SCADA network configuration for the South Regional Treatment Campus.

Figure 2 -6 . SCADA Network Configuration – North Regional Treatment Campus

PBU2073. 01L 16 February 2015

Figure 2 -7 . SCADA Network Configuration – South Regional Treatment Campus

2.4.3 Communication Protocols

The primary protocols utilized in the system are Ethernet/ IP, RS -485 Modbus Plus, and RS -485

Modbus. Newer PLC processors can use Ethernet/ IP, Modbus TCP/IP, or other protocols such as DNP

3.0. The protocol used must be supported by the PLC type in order for it to be considered as an alternative. Different protocols can offer different benefits such as stability, security, and the ability to integrate with different brands of equipment.

The PLC and RTU communication protocols vary depending on the facility. The following is a brief description of the protocols currently in use.

RS -485 Modbus is used for communication between the PLCs and the PCs running iFIX at the

WWTP.

Ethernet/IP is used for communication between the Lime Softening WTP PLCs and the PC.

RS -485 Modbus Plus is used for communication between the RO WTP PLC and the PC running iFIX at the Lime Softening WTP.

RS -485 Modbus is used for communication between the PLCs at the wells, the PLC at the ASR, the PLC at the Nash Re -Pump Station, the PLCs at the WRF and the PC running iFIX at the

Lime Softening WTP.

Ethernet is used for communication between the PLCs and the PCs running iFIX at the WTP.

SCADA servers and clients communicate via TCP/IP protocol.

PBU2073. 01L 17 February 2015

2.5 Field Equipment – PLCs , RTUs & Master Radios

2.5.1 Overview

The SCADA field equipment is comprised of PLCs and RTUs located at various process locations.

Data is transmitted to PCs by means of fiber optic cable or radio , depending on the location . The process data is displayed on the PCs using Proficy iFIX graphic user interface software. The data is stored on a Historian server using Proficy iHistorian software. Most of the installations consist of a single PLC and RTU . If one of those units fails, the problem can only be corrected by replacing the failed unit.

2.5.2 Standard Equipment

The radios are General Electric MDS and the PLCs are primarily Modicon , though other PLCs can be found throughout the system ( i.e. Allen -Bradley at the WRF Disk Filters). The existing PLCs transmit data when polled – they do not store or buffer any data. In the event of a communication failure between the PLC and SCADA network, any data from the PLC will be lost.

PLCs and radios are typically only “ upgraded” if a new version of the firmware is released by the manufacturer. Otherwise, they are replaced in the event of failure, old age, or lack of sufficient memory.

PLC and radio firmware would be updated only if the manufacturer releases a new version of the firmware, which is essentially the operating system for these devices. When made available, the new firmware would be installed by the staff which installs/ maintains this equipment.

2.5.3 PLC Control Configuration

The PLC systems for the North Regional’ s WWTP, WRF, and Lime Softening WTP are configured as a

Distributed Control System ( DCS) network, as opposed to a single “ Plant PLC”; the DCS network distributes” the control of treatment processes over multiple PLCs. The DCS type of system insulates the entire treatment plant process from a catastrophic failure of a given PLC – only the process equipment controlled by that particular PLC will be affected. The DCS is the proper design configuration for these treatment facilities and should be retained when future upgrades are implemented.

The PLC system for the RO WTP is a single “ Plant PLC”. Since the RO WTP is essentially a packaged system”, the “ Plant PLC” approach is appropriate for this application. Further redundancy was included with the existing control system since the “ Plant PLC” is comprised of two PLC processors wired in a hot -backup configuration.

The SRWTP is configured as a DCS network.

2.5.4 Equipment Description and Assessment

The following subsections will provide the locations and description of each PLC , RTU and Master

Radio that are connected to the SCADA system. Photographs , technical parameters and a description of the condition of the equipment will be provided in tabular form. Prior to the discussion of each facility’s equipment, a location map of each PLC and RTU has been provided.

LIME SOFTENING WTP AND RO WTP PLC

LOCATIONS

PALM BAY UTILITIES

CITY OF PALM BAY, FLORIDA

2015 SCADA MASTER PLAN

ºPAGE

18 Source: City of Palm Bay, Florida, GIS Dept., 2012; Wade Trim, Inc., 2012 1 LIME SOFTENING WTP & 4 FILTER

PLCS FIBER

OPTIC) SECTION 2. 5. 6; PAGE 23 TO

27 RO WTP PLC ( FIBER

OPTIC) SECTION 2. 5. 5; PAGE

20 PLC

Panel

LEGEND07515037.

5Feet MASTER RADIO (

RADIO) SECTION 2. 5. 7, PAGE

Radio FIGURE 2-

PBU2073. 01L 19 February 2015

2.5.5 North Regional RO WTP

The PLC control panel was constructed in 2006 and is located in the RO Building . The existing PLC control panel consists of two Modicon Quantum processors which are wired in a hot -backup configuration; if the primary processor fails, the secondary processor will automatically assume control over the process system. There is, however, only one set of I/O modules. The RO WTP system is displayed on the computer ( labeled TR -S1) in the RO Building office.

The SCADA application for the RO WTP provides for certain levels of control and monitoring of the system. A summary of the primary control and monitoring functions has been provided in Table 2 -2 .

Table 2 -2 . North Regional RO WTP System Summary

SCADA Function Components

Monitored Equipment RO Skid & Pumps, Raw Water Wells, PLC Status

Monitored Parameters

May Overlap with WTP)

Chlorine Residual, System Pressures, Plant Flows, pH Temperature, Turbidity, Fluoride, Conductivity, Salt Passage, Salt Rejection, Salt

Recovery

Trending Flows, Temperature, pH, Free Chlorine, Total Chlorine, Turbidity, Raw Water Wells

Control at Screen

RO Sequencing, Normal Start/ Stop, Emergency Shutdown, System

Alarm Reset, Raw Water Start/ Stop, Well/ Skid assignment, PLC

Alarm Acknowledge

Reporting pH, Flows, Concentrate Chloride, Concentrate pH, Distribution pressure/ flow, Skid parameters, Pumping Summary

The RO WTP PLCs were installed in 2006. Although they not new, the Quantum model is currently still in production and will most likely be supported for several years to come. Although future expansion is not anticipated, the control panel has space to add three additional I/O modules and the PLCs have spare memory capacity available to add additional programming. These PLCs have a life expectancy of 5 -7 more years based on their age and indoor environment.

During the field visits and interviews with staff, it was noted that the SCADA application does not allow for control and monitoring of the chemical feed systems. Additionally, automatic population of the

Monthly Operating Reports ( MORs) is not possible.

Table 2 -3 provides photographs, technical parameters and a brief description of the RO PLC.

PBU2073. 01L 20 February 2015

Table 2 -3 . North Regional RO WTP PLC

PLC Panel PLC Panel - Internal

Technical Parameters

Make/Model Modicon Quantum (2)

Communication Modbus to Ethernet

Memory 512KB RAM / 16KB User

Quantity of I/O Modules 5 DI, 2 DO, 3 AI, 1 AO

Spare Slots Three ( 3)

Year Installed 2006

Description and Condition

This PLC monitors and controls the RO WTP process equipment. The PLC system consists of two processors in a hot backup configuration. If the main processor fails, the standby ( redundant) processor automatically assumes control.

PBU2073. 01L 21 February 2015

2.5.6 North Regional Lime Softening WTP

The Main Control Panel ( MCP) and Filter Control Panels ( FCP) were replaced in 2012 and are located in the Operators Building . The existing MCP dated back to 1974 and had been modified as needed over the years to keep it functioning. By 2010, however, most of the MCP equipment was outdated and the wiring modifications were not well documented. In addition, the outdoor mounted FCPs which dated back to 1984 , were also aging and the hydrogen sulfide from the adjacent sand filters had corroded equipment and wiring terminations. PBU staff decided that the MCP and FCPs were due for replacement and that the new FCPs would be located indoors along with the MCP in the Lab Room.

A new MCP was constructed along with two FCPs; each FCP contains the controls for two filters. Five new Modicon M340 PLCs were installed, one for each of the four filters and one for the remaining Lime

Softening WTP functions. The MCP also houses a touchscreen panel -mounted PC which runs the iFIX software and displays the graphic screens for the Lime Softening WTP. Additionally, the existing

SCADA system network equipment ( Switches, Computers, UPS) as well as the security camera equipment and digital video recorder ( DVR) were moved and consolidated into enclosed 19” network racks.

The Lime Softening WTP SCADA system presently consists of the aforementioned PLCs and touch -screen PC, a primary SCADA computer ( labeled TR -P1), and a computer in the Lime Softening WTP

Superintendent’ s office. All of this equipment is wired via copper Ethernet cabling to a switch located in the 19” network rack. A second switch in this rack incorporates a media convert er which converts the copper Ethernet signals to fiber optic. The Lime Softening WTP system is then connected to the overall

North Regional SCADA network with fiber optic cable.

Figure 2.9 depicts the network switches ( left), SCADA computers ( center) and security network equipment ( right) located in the lab.

Figure 2 -9 . WTP Network Switches, Computers, and Security Camera Equipment

PBU2073. 01L 22 February 2015

The SCADA application for the Lime Softening WTP provides for certain levels of control and monitoring of the system. A summary of the primary control and moni toring functions has been provided in Table 2 -4 .

Table 2 -4 . North Regional Lime Softening WTP System Summary

SCADA Function Components

Monitored Equipment Ground Storage Tank, Filters, Clea rwell, Raw Water Wells & Pumps, Fluoride System, Monitored Para meters

May Overlap with RO WTP )

Ground Storage Tank Level, Generator Status, Flows, Pressures, Chlorine

Residual, Remote Monitor Well Pressure @ Lift Stations ( 4), Temperature, pH, Chlorine Residual, Fluoride, Clearwell Level, Turbidity

Trending Flows, Temperature, pH, Free Chlorine, Total Chlorine, Turbidity, Raw

Water Well Parameters

Control at Screen Fill Valve, Transfer Pumps, Pump Setpoints, PLC Clocks, High Service

Pumps, Filters, Filter Backwash, Filter Effluent Valve

Reporting pH, Flows, Distribution pressure/ flow, Pumping Summary

During the field visits and interviews with staff, it was noted that the SCADA application does not allow for control and monitoring of the chemical feed systems. Additionally, automatic population of the

MORs is not possible. Staff has also expressed frustration with data collection from the filter PLCs.

The Lime Softening WTP MCP and FCPs were replaced in 2012 and included five (5) new Modicon

M340 PLCs, which are a current model. The control panels have space to add additional I/O modules and the PLCs have spare memory capacity available to add additional programming.