Albert Rooyakkers is founder, CEO and CTO for Bedrock Automation. Rooyakkers can be reached at [email protected].
The PLCs that the East Cherry Creek Valley (ECCV) Water & Sanitation District had been using to control the potable water treatment facilities and pump stations that supplied water to 50,000 people in the Denver suburbs were becoming obsolete. It was time for an automation upgrade and the utility’s operations managers saw this as an opportunity to deepen cybersecurity protection, as well.
“Like most other public utilities, we must adapt to an ever-changing world and that includes cybersecurity,” said Shay Geisler, I&C administrator for ECCV Water & Sanitation District. “We’ve always had robust physical security and required usernames and passwords for access to critical systems and controls, but we saw the world around us changing quickly. Many of today’s automation technologies are not as secure as they could be because they were developed long before security was a major issue in the industry. Most of the security added to them was an afterthought.”
Legacy System
ECCV’s existing plant control architecture was comparable to what many municipal water systems use: a dedicated Windows Desktop or Windows Server OS. The top end SCADA software system is housed on a dedicated Windows desktop or server along with a communications driver, in this case an OPC Server that speaks to the PLCs via legacy Bristol Standard Asynchronous/Synchronous Protocol (BSAP) and to some Ethernet IP devices.
Data concentrators sat above the PLC network to help manage data communications and aggregation across a serial radio network involving about 80 sites running a mixture of RTU and PLC types and generations. These radios functioned like firewalls separating the SCADA Network and the PLCs in the field, but signals were not encrypted.
“We knew security could not be limited to the SCADA software only,” Geisler said. “There were too many downstream systems and assets that, if left untouched, would present a huge vulnerability. We determined that the vast majority of these potential vulnerabilities could be solved by addressing the PLC and SCADA communications system.”
Geisler and his team decided it had to focus on securing three communications paths: SCADA software to PLC; PLC to PLC; and the radio network.
The team explored several strategies to secure those communications, including adding firewalls and network cloaking, but ultimately determined that getting the depth of security needed required upgrading the PLCs, RTUs and network radio. Working with automation solutions supplier Process Control Dynamics and system consultant RSI Company, ECCV chose Bedrock Automation’s Open Secure Automation (OSA) to provide PLC/RTU functionality because of its intrinsic security along with a new ethernet radio solution to provide high data encryption capabilities.
Software Upgrade
Supporting the new capabilities required upgrading the current 32-Bit SCADA Software to a 64-Bit solution, which enabled ECCV to leverage the latest Windows Server and Windows-10 based OS capabilities.
“Just upgrading the software provided a much higher level of confidence within both our IT and OT Departments,” Geisler said. “We also gained valuable operational features and functions, along with many new and powerful security features in the SCADA software itself. This addressed some of the security issues we had with our legacy systems, but it was not enough. We still saw those possible security holes downstream of the SCADA System and we wanted to address those.”
Geisler and his team concluded that the most secure and cost-effective approach would be to connect the SCADA network and control networks with a secure communications channel. However, fully implementing this would have required ripping and replacing the entire system, which would have been costly and would require significant disruption of operations. Instead, ECCV adopted a phased-in approach, which began by deploying Bedrock OSA remote control units as a secure proxy server between the data concentrators and the SCADA server.
“A cyber-secure data concentrator functions as a proxy server that secures communications from the SCADA software and the PLC network,” Geisler said. “Downstream the data concentrator speaks BSAP or Modbus directly to the existing unit in the field, as well as some Ethernet/IP for smart devices.”
Because the OSA remotes support BSAP, the utility could continue communicating with its remaining legacy devices while transitioning to new controls, this avoided any significant interruption of service.
Moving Downstream
The next phase was to secure a direct connection between the SCADA software and the well sites, the pump stations and the water treatment facilities. This would be done with PLCs and controllers that have intrinsic cyber security along with new Ethernet radios. Covering so many input and output (I/O) points — roughly 9,000 — required scaling to a Bedrock OSA platform that scales infinitely through the addition of 5, 10 and 20 control module racks, depending on the number of I/O at each site.
With these Bedrock units installed, the utility can leverage new SCADA features that extended a root of trust from the PLC controllers to the HMI/SCADA System, thereby limiting all communications with the PLC/controllers to certificated programs and users only.
This enabled the district to execute standard IT certificate practices such as time limitation, revocation, etc. to individual users or groups with ease. The result is secure, certificated communications from the SCADA software all the way down to the remote PLCs and RTUs.
Figure 3 above shows the target completed architecture. The 64-bit SCADA software connects directly and securely to a peer-to-peer network of scalable Bedrock OSA control systems connected with an encrypted radio network.
The intention was to complete the final architecture within five years, but COVID-19 related delays may extend this. As the team builds the system out, ECCV has the option to keep the OSA remote concentrator/proxy nodes in place or remove them because the system will be secure all the way to the field level PLC and RTU devices. This is one of the advantages to transitioning in this manner. The district is now evaluating the operational pros and cons of the data concentrator model and will decide later on, but either way, it will not break the security model. Geisler feels that the plant is well-equipped to weather the next round of changes.
“We expect the technology for industrial systems to be ever evolving and improving. With this open architecture and technology, we will be able to continually improve and upgrade as we need to, so we don’t have to face this type of wholesale transition again,” Geisler said, adding that ECCV expects to get more than 30 years of useful life from the new PLC and RTU systems, and because Bedrock manufacturers most of its own chips and controls its secure supply chain, the company offers a non-obsolescence policy if maintenance requires new hardware.
