FW: WWTP SCADA

From: Jennifer Cline

Sent: Fri Jun 19 16:54:04 2015

To: Phillip Bender

Subject: FW: WWTP SCADA

Importance: Normal

Attachments: image001.jpg; may 11 inf flow.pdf; May 10 inf flow.pdf;

Hi Phil,

See Troy’s last paragraph. I high-lighted the reason that most represents what we have seen problems with across the board. We know that all the other reasons for the headwork’s high-level bypass not calling out have been eliminated and shown to be working consistently. In addition, the headwork’s high-level by-pass alarm was not disabled.

I can verify with Jake as well, but I remember him saying all the alarms came through once he logged in and manually activated the pumps through the SCADA system. This was after he had put the pumps in hand (manual) at the TPS.

Jennifer Cline, P.E.

City of Molalla

Public Works Director

O: 503.759.0218

F: 503.829.3676

"file:///C:

From: Troy Collison [mailto:tbc@industrialsystems-inc.com]

Sent: Wednesday, June 03, 2015 5:38 PM

To: Jennifer Cline

Cc: Dan Huff; Phillip Bender; Jason Clifford; Fasth, Bill

Subject: RE: WWTP SCADA

Jennifer,

The print outs for the 10^th and 11^th are attached.

The level in the flume dropped instantly from “off the scale” to “normal” at about 2 AM on Saturday the 10^th and then rose to be off scale again by about 10AM.

I cannot say what caused this anomaly. It could have been that prior to this time, something was blocking the influent channel and that object broke loose dropping the level in the channel rapidly and then it built up again over Sunday morning. This is pure conjecture though.

I cannot tell from the small chunk of alarm/event history that I downloaded if the HWS by-pass alarm was disabled during this period or not, though that might be able to be verified using the full version of the log on the computer.

Besides being disabled, the other things that would prevent the dialer from calling our when an alarm is present that SHOULD cause it to dial would be if the phone line is not working or unplugged, the dialer is “disarmed” using the keypad on the dialer itself, (this is not traceable) or a loss of communications with the PLC. The dialer does not use Ethernet to communicate to the PLC, it uses a GE proprietary protocol called genius. To my knowledge, no one has looked at or worked on the genius communications between the PLC and dialer in a long time (months or years) and as of yesterday it was functional.

Thanks

Troy B. Collison

5835 NE 122nd Avenue, Suite #110

Portland, Oregon 97230

http://www.industrialsystems-inc.com

Phone: 503.262.0367

Cell: 360.957.5039

Fax: 503.262.0371

CCB # 196597 WA #INDUSSI880K9

AK# 1018436

From: Jennifer Cline [mailto:jcline@cityofmolalla.com]

Sent: Wednesday, June 03, 2015 4:13 PM

To: Troy Collison

Cc: Dan Huff; Phillip Bender; Jason Clifford; Fasth, Bill

Subject: RE: WWTP SCADA

Hi Troy,

Thank you for your prompt response, I have a few additional questions.

The alarms on Friday 5/9 were because the operators cleaned out the wet well of the TPS as a regular maintenance practice. The pumps were in operation and functioning when the crew left on Friday. Also seen to be functioning fine when Jake was at the plant on Saturday morning.

Regarding the diagram, it only appears to be from Saturday 5:53 am to Sunday 5:52 am. The graph also indicates the pumps may have been operating on Sunday around noon when the flows dropped to normal again. Can you provide a graph from Sunday 5:00 am to Monday at 10 am as well? Is it possible to snip copy of the screen and send as a .pdf?

I spoke with Jake today and he confirmed the HWS by-pass alarms were enabled during this event. Is there anything else that would have stopped the alarm from communicating to the dialer?

Regards,

Jennifer Cline, P.E.

City of Molalla

Public Works Director

O: 503.759.0218

F: 503.829.3676

"file:///C:

From: Troy Collison [mailto:tbc@industrialsystems-inc.com]

Sent: Wednesday, June 03, 2015 3:06 PM

To: Jennifer Cline

Cc: Dan Huff; Phillip Bender

Subject: WWTP SCADA

Jennifer

During today’s discussion I was asked to look into several things:

1) Availability of historical alarm data (older than 24 to 36 hours)

a. This is available. Please see attached text file covering the period of May 9^th – 11^th

i. Several Transfer Pump Station alarms occurred at 3:55 PM on Friday the 8^th

1. VFD 2 Fault

2. VFD 3 fault

3. General alarm

ii. At 4:02 PM Transfer Pump 1 failed on low flow and then at 4:05 – 4:08 the VFD on this pump issued several other alarms

iii. At 4:12 PM, the lead pump selection was changed several times

iv. At 4:18 PM, the on/off levels for the pumps were changed

v. The next things logged, other than report generation, is the loss of access to the SQL database at 1 second after midnight on the 10^th. (this is NOT the com fail issue we have been examining and does not affect the operation of the plant)

vi. Then the HWS bypass alarm at 8:39 AM on May 10^th

2) Can we tell from any historical data, when the high level occurred

a. Yes, please see attached plot of the influent flow. This shows that the influent parshal flume began experiencing an abnormal high level at about 3:00 PM on Saturday May 9^th, 2015

b. NOTE - I outlined the flow in pen before scanning this print out to enable a person to read it. The actual print out is difficult to see but is readable. The scan would not be. Viewing the data on the SCADA computer is, of course, very clear

3) When was “Log Me in” installed at the WWTP?

a. Friday May 15^th, 2015

4) How good is the security rating of “Log me in”? It is as described during today’s meeting. This is outlined on the webpage at this link.

a. https://secure.logmein.com/products/pro/security.aspx

b. I did not find a review of this product by any governmental entities

c. I also attached a white paper on the security of this product

i. I believe it is safe to say, the software provides a reasonably secure remote connection:

1. It is protected by two sets of user names and passwords

2. The software CAN be configured to store logon and logoff events

3. Data being transmitted over the internet is encrypted

4. The software is configured to detect when someone tries to intrude on the link

5) What triggers the HWS bypass alarm?

a. This is triggered by a discrete input to the Programmable Logic Controller (PLC). In one place in the PLC logic it is listed a afloat switch and in another it is listed as an alarm output from the HydroRanger level transducer so I am not sure which is accurate

I hope this information is helpful. Please do not hesitate to call with any questions,

Best Regards,