About the author: Cliff Campbell is general manager for Frakes Eng. Campbell can be reached at [email protected].
By now, the vast majority of us have updated our antiquated technology. Cassette tapes gave way to CDs, which gave way to digital music, thanks to iTunes and MP3s. The clunky cell phones of the 1980s have fallen, and smaller, sleeker models have risen in their place. Today’s computers make the models of even a few years ago look like dinosaurs.
The city of Indianapolis’ Public Works Department underwent a similar transition with its wastewater collection network’s supervisory control and data acquisition (SCADA) system. With some help from Frakes Eng., a certified member of the Control System Integrator Assn., the transition has been a smooth one.
The Indianapolis Department of Public Works Wastewater Collection Div. is responsible for maintaining more than 250 sewer lift stations throughout its jurisdiction and ensuring the safety of the public waterways through monitoring the collection process. The original lift station monitoring system was an obsolete, homegrown alarm dialer that communicated through locally leased phone lines, providing only an alarm. Master controllers were notified when something went wrong at one of the monitored sites, but had no means of determining the problem or a way to predict potential future problems. Flow and pump condition data were non-existent, and overflows to streams could not be detected. In short, there was insufficient data within the system to provide good management decisions.
In 2003, the department approached engineering firm Donohue & Associates about the prospect of providing this system with a much-needed update. After two years of learning about the system and determining the correct approach, Donohue began the design phase in 2005 and selected a control and information system supplier (Rockwell Automation). It selected communications methodologies in 2006 and bid the project in 2007. Indianapolis-based Frakes was selected as the control and information systems integrator, under sub-contract to Divane Brothers Electric of Franklin Park, Ill.
The challenge was to implement a complete SCADA system to collect and process the status and data from more than 250 lift stations, spread over a 1,600-sq-mile area. Processing the data included requirements to immediately process alarms and dispatch service personnel, provide a graphical display overview of the entire system, historically store the data, trend data on demand, and provide a remote information window (Portal) for management to make intelligent decisions.
Remote Terminal Unit
At the basic level of a lift station, the system was designed with Allen Bradley Micrologix 1100 Ethernet-based processors with analog and digital inputs and outputs, communications modules (radio or telephone), two-hour-capability battery-backed uninterruptible power supply and surge protection in a stainless, drip-proof enclosure with outer and inner doors with front-mounted AB Panel view for a complete remote terminal unit (RTU).
The RTU was tightly coupled, but in a separate enclosure from the existing lift station pump control package. Interface to the existing pump controllers was a major task, as each had been supplied by different vendors and included various types of programmable logic controllers (PLCs) and relay control schemes. Frakes developed a standard terminal interface for the electrical contractor and for future maintenance and troubleshooting. Therefore, all lift stations throughout the city now have the same control scheme wiring. Future installations must also match the control scheme as developed.
Getting the Data Home
With more than 250 individual lift stations connected on a citywide communications grid, every possible method was contemplated. The city of Indianapolis has an emergency notification system for fire, police and disaster alerts based on microwave towers communicating from various regions of the city—such as fire stations and other significant structures—to the Metropolitan Emergency Communications location on a redundant microwave ring that connects 10 regional communication towers. Each tower has a tail-end, spread-spectrum radio network connecting lift stations to the SCADA system through the microwave ring. Once data has been relayed to this downlink, an AT&T Optiman 100-MBPS link connects the radio system to the Belmont Advanced Treatment Plant. A 5.8-GHz radio link is in place between the downlink and the plant as a hot backup to the Optiman circuit.
Sites that cannot be reached with radio or are too critical for wireless communication are connected to the plant through an OC-3 SONET ring, also provided by AT&T. From there, communications are established to remote sites using everything from always on modem connectivity to 3G cellular to DSL circuits. The most critical remote sites have redundant communication paths to them.
Handling the Data
The communications methodology is a polling master with report-by-exception (RBE) alarm reporting. Communications with the remote sites are handled by four hardware control devices (HCDs) that are redundant Allen Bradley ControlLogix PLCs. In normal operation, each of the four HCDs handles a single quadrant of the city. Should one communication driver fail, the others will load balance to pick up the load.
Handling the number of tags required to manage a 1,600-sq-mile wastewater collection system requires a massive computer network. To isolate and protect critical public utility systems from the security problems that plague corporate networks, two domain controllers manage the process network. The process network includes 19 servers, eight work stations and 10 laptops used for data storage, retrieval, viewing, programming and analysis.
Control Room Software
The project was built around a Rockwell Automation FactoryTalk solution. By selecting a complete solution, the city was able to leverage many of the advantages of the Rockwell federated database, not least of which was the need to create tags only once. This was a major timesaver in a system with more than 25,000 historical tags and more than 30,000 display points.
Thinking communication systems will never fail would be poor engineering; therefore, the RTUs required contingencies for just such occurrences in order to prevent loss of data.
Data is retrieved from the remote sites in several ways. In normal operation, each site is polled and replies with the current status of the site and the flow and level trends for the period of time since the last poll cycle. If communications go down for a period of time, the remote site will store the trended data at a lower rate and, when communications are re-established, the remote site will backfill the data to the plant at a rate that does not interfere with the communications to the other sites. In the event of an alarm condition at any site, that site will use RBE to immediately notify the plant operations staff of an alarm condition, allowing them to respond quickly.
Originally, the specification called for data collection on a 10-minute basis. This was found to be not timely enough to give an accurate picture of flow data, and was later changed to a one-minute basis, which greatly improved the flow data trend charts. Upon loss of communications, the remote sites collect data and work in a store-and-forward mode to make certain critical data is never lost. One of the challenges was to “re-sync” the time stamps on alarms and events that occurred while communications were down. Normally, the FT Historian server software would apply a time stamp to any collected data based on the time received.
Because the data was collected on a one-minute basis and uploaded when polled on a three-hour basis, the collected data would all be stored in the Historian as of the time/date received; an unacceptable trend display would result. The time-date stamp is encoded in the data at the MicroLogix and with the help of an ACE tool, the software strips out the time-date stamp and re-presents it to the FT Historian.
The communication network includes another advantage over the old system—flexibility. Bob Neils, control systems engineer with Donohue, said the openness and adaptability of the communication system is a real asset. “We wanted to keep the system open as much as possible,” Neils said. “Having the network Ethernet-based and having an open network allows for multiple options at each site. For example, if one or two sites can’t use radio because of their location or other circumstances, the open network allows us to have other options, rather than being locked into only one solution.”
This project was completed in early 2012.
Water System Central Control System Upgrade
A water company had developed a central control system (CCS) that consisted of a master distribution control room, connected to remote tanks, pumping stations, bleeder valves, and district pressure sites using tone telemetry communications. These sites consisted of tone transceivers at each end, with I/O hard wired into the CCS PLC-5 before connecting into the SCADA system. As the system grew over the years and decades, an assortment of other control systems had been implemented. Other sites had been upgraded to Allen Bradley PLCs and communicated with CCS via 4-wire lease line.
As the tone equipment got older it proved to be very maintenance intensive, requiring frequent calibration of the frequency scanners in use.
Frakes Eng. was hired to perform several projects in the design planning stages leading up to the upgrade. These included a study to assess the feasibility of replacing the tone telemetry equipment, conducting radio feasibility studies to evaluate the possibility of utilizing radio to replace portions of the existing phone modem network. The work culminated in a system replacement study outlining the best mix of technologies the water company could use to bring the system up to date and provide the reliability required of such a critical operation serving over 1 million people.
Frakes Eng. was awarded a two-year project to provide a turnkey solution based on the company’s recommendations. This new system would utilize frame relay Ethernet communications, spread spectrum radio tail-end networks as well as CDMA cellular communications to connect all of the sites which stretch over a 1,600 sq mile area.
The heart of the system was built around Allen Bradley’s ControlLogix Automation Controller. Redundancy was designed in from the start with redundant PLC racks, processors, power supplies and ControlNet communication channels to the gateway PLC rack, which would contain the Ethernet communication modules. While these modules remain as the only single point of failure, they are hot swappable and do not require the system be halted for repairs. To address disaster preparedness, the entire CCS was duplicated at an alternate command site, providing 100% control and monitoring redundancy and allowing the entire distribution system to be operated from the remote site in case of system wide failure at the primary CCS.
Select sites featuring tall structures, such as elevated tanks, were used as the head end for spread spectrum radio networks that would function as tail-end networks, connecting nearby sites, such as bleeder valves, district pressure recorders, to the network without the monthly overhead of frame relay connectivity.
Sites where connectivity to the frame relay network would be difficult were fitted with CDMA cellular modems manufactured by Airlink Communications. These sites communicate over the cellular data network with the ControlLogix Controller through redundant serial ports at the CCS.
Of primary concern during deployment was prevention of communication loss as each remote system was transferred from the existing system to the new system. Each site was converted on-line with our programmers modifying the existing Fix32 SCADA package to redirect communications through our new data concentrator while our field installation team removed old equipment, installed the new control panels and connected it to the frame relay network. This involved extensive coordination between the phone utility, Frakes Eng. and the customer to bring all resources together at the right time, making each transition transparent to the operators of the system.
This project was completed on schedule and without change orders. When completed, Frakes Eng. had either modified or replaced controls at more than 65 sites, including conversion of three major pump stations. The water quality to the city was never compromised and no loss of control was experienced, a major requirement of an on-line project such as this. The firm continues to provide both engineering control solutions to this customer.
Download: Here