Hi Jason,
We should discuss with IS when they come out if they work is included in the quote is in line with Mark’s recommendations below, see high-lighted below? I see they call out “re-route the cabling” to the traditional Cat 5. I see Mark is recommending Cat 6 (shielded) or fiber. This may be something we want to discuss prior to the re-wiring work indicated in the quote.
Regards,
Jennifer Cline, P.E.
City of Molalla
Public Works Director
O: 503.759.0218
F: 503.829.3676
"From: Maisonville, Marc [mailto:mmaisonville@brwncald.com]
Sent: Thursday, May 21, 2015 11:03 PM
To: Jennifer Cline; Mark Strandberg
Cc: Dan Huff; Jason Clifford; 'J.W. Ring'; 'Christine Hein Outside'; Phillip Bender; Lee, Rob
Subject: RE: Industrial Systems Emails & Quotes
Hi Jennifer,
I’ve had a chance to look over the emails that you’ve forwarded to me as well as the photos and notes I took when I visited the waste water treatment plant (WWTP) last week.
Here’s my comments/observations:
1. I reviewed the forwarded email from Industrial Systems pertaining to upgrades recommended in May 2014 for both the WWTP and the water treatment plant (WTP). The scope provided new PC servers running Windows 7 and new SCADA software (GE Cimplicity) to replace the older versions of Cimplicity running on obsolete Windows XP computers. You questioned why the WTP also got “RSLinx Classic” whereas the WWTP did not. This is not an Ethernet switch, rather it’s another software package that’s necessary for communications with the programmable logic controllers (PLCs) at the WTP. I don’t have any info on what’s at the WTP, but RSLinx Classic is a Rockwell/Allen Bradley product, so I suspect the PLCs at the WTP are Rockwell/Allen Bradley in which case RS Classic would enable the PLCs to communicate with the new GE Cimplicity SCADA software. There was no need for RS Classic at the WWTP because the Cimplicity SCADA software and the WWTP PLCs are all compatible GE products.
2. Since the event at the WWTP discovered on May 11th, Industrial Systems has performed troubleshooting on the network system and made recommendation to do the following:
· Provide a network Block Diagram
· Replace the old Ethernet switch for the control system with an industrial managed switch in an enclosure
· Reroute and reterminate Ethernet cabling and test
· Provide loss of PLC communications alarms
I agree that these measures would improve the reliability of the network for the plant’s control system and should help eliminate random drop outs in communications that were reported. The overall network should be investigated, documented and reconfigured such that the Ethernet switch for the control system is not interconnected with the Ethernet switch for IT/voice communications. Upgrading the old residential-grade Netgear switch to an industrial-grade, managed Ethernet switch will greatly improve reliability and allow monitoring of the health of port status. Ethernet cabling should be cleaned up and protected as far as practical by use of enclosures and conduit.
3. Consideration should be given to upgrading existing Ethernet cables in buried runs from the Admin Building to the PLC panels in the field. I did not physically inspect the cables, but suspect they are residential or commercial-grade unshielded CAT5 or less. These cables are not really suitable for industrial installations where they are subject to harsh, wet environments and electromagnetic interference from large variable frequency drives (VFDs) such as installed at the Transfer Pump Station (TPS). In any case, the existing Ethernet cables should be inspected and tested. If replacements are necessary, consider upgrading to industrial-grade, shielded CAT6 cable. However, also note that copper Ethernet cables such as CAT5 or CAT6, are limited to relatively short runs of 100 meters (328 ft.) or less. For these reasons, fiber-optic cables, which are immune to electromagnetic interference and can be routed over long distances, are much better suited for plant-wide networks.
4. Operations mentioned that a high level float was to be added to the TPS and to be hardwired to the autodialer to provide a dedicated call-out alarm. This is a great measure to provide back-up for failed Ethernet communications or a failed level transmitter in the TPS.
5. It was reported that the SCADA screen for the Transfer Pump Station was found to be in “STOP” mode when the high-level event was discovered. It was also reported that communications drop-outs had occurred on several recent occasions as well as apparently random triggering of “STOP” mode. Moreover, all alarms associated with the TPS are disabled when in “STOP” mode. So, if TPS is stopped while the plant continues to receive flows, a high level event will eventually occur without an alarm. Personally, this is recipe for disaster. The possibility of a software glitch, communications glitch or inadvertent click of a mouse shutting down a critical facility is highly undesirable. Accordingly, it is strongly recommended to remove “STOP” modes from all critical facilities. If an operator needs to actually stop a critical process, then a deliberate manual means is recommended. For instance, the TPS pumps could be shut down by going to each VFD Panel and turning its “Hand/Off/Auto” selector switch to the “Off” position.
It’s unfortunate that this high level event happened. I found that staff are very conscientious and capable in their work. Hopefully, programming and network improvements can be made to avoid future occurrences of this problem.
Please feel free to call if you have any questions or want to follow up on anything.
Thanks.
Marc Maisonville, PE
Brown and Caldwell
T 503.977.6626
From: Jennifer Cline [mailto:jcline@cityofmolalla.com]
Sent: Wednesday, May 13, 2015 3:11 PM
To: Mark Strandberg
Cc: Dan Huff; Jason Clifford; 'J.W. Ring'; 'Christine Hein Outside'; Phillip Bender; Maisonville, Marc
Subject: Industrial Systems Emails & Quotes
Hi Mark,
Dan has some concern regarding the last upgrades Industrial Systems did for the City. As you can see in the May 7, 2014 quote under Scope of Work that Industrial Systems (IS) updated the RSLinx Classic at the WaterTP. I believe this is part of the Ethernet Network Switch (I may be wrong) for the WaterTP, however a switch compatible for the WWTP was not quoted at that time for the WWTP. I feel this shows the City is willing to do the upgrades/preventative maintenance, however due to our limited expertise in these systems our staff does not have the background to determine necessary updates.
Jennifer Cline, P.E.
City of Molalla
Public Works Director
O: 503.759.0218
F: 503.829.3676
