Like many in the field of Information Technology (IT), I find that technology shifts before our eyes at a nosebleed pace. Technologies come and go - some technologies are replaced quickly, while others seem to hang around for the long haul. As seen in Automation World, I believe it is important to stay relevant and knowledgeable about new technologies for those of us who are in the field of Industrial IT.
At each shift in my career, from Support Specialist on a global banks stock trading network, to Network and Security trainer, and then virtualized datacenter designing engineer, I spent some time thinking about my role and how I could best contribute, often in ways that are not confined to the title of the role. I would like to think that my hard work has led me to where I am today, as part of an Information Solutions team at Avid, a leader in the field of automation and information solutions. At this point with Avid, I’m asking about the next thing that I should be learning, which led me right to the world of Industrial Control Systems or ICS.
I’m new to ICS. But, contrary to popular belief, being new doesn’t mean being unproductive in a new industry. This is the busiest I’ve been in my entire career, and I’m enjoying it in ways that I didn’t expect. I firmly believe that, in this role, I can help our customer’s Business IT (Information Technology) organization to better understand and support the OT (Operations Technology) design and requirements. Customers in the industrial space need a translator between their IT and OT organizations.
Remember, I come from the Business IT side of the technology world. I’ve seen how many of the IT organizations react to individuals or groups trying to run their own networks and servers. IT often feels that they are the defacto experts and should own all aspects of the server’s operation. Sometimes, it’s a lack of trust by the IT group – they worry that a new server that they have not personally built is not secured properly or will become an unmanaged ad-hoc rogue device that will become forgotten about until it becomes a problem.
There are also IT organizations that operate with a bit more hostility towards others. These exist and operate through resource power-grab primarily for reasons of fear or in order to justify their existence. In this situation, IT may resent the firewall team and vice-versa. Private Cloud Virtualization teams don’t trust Public Cloud initiatives and cybersecurity trusts neither.
Where does this leave OT? In my short time here at Avid, it’s become painfully apparent that OT is misunderstood. IT wants to run the OT networks the same way it runs the business networks. People in IT wonder if it’s possible to run the network on behalf of OT. For my part, I think it’s a completely plausible expectation, but I have not seen it work - yet.
An IT network is a relatively available infrastructure. But, there is an argument that the HA (Highly Available) network infrastructure that has been built will failover and the client server stream will continue. I agree, but is it OT quality? IT managed networks don’t conform to the needs of OT in many cases. OT needs the network to be up, all the time. Rebooting a router to fix a problem is often not an option. If there is not a 30ms to 250ms failover between redundant paths, the network is not good enough. In a shared network between IT and OT, an employee’s large file transfer of an ISO file could bottleneck and impede OT telemetry. Of course, all efforts must be made to prevent this.
Similar arguments can be made for virtualization. Could an IT virtualization group run the OT Virtual Machines? Sure, it could. But first, it’s important to understand that the cornerstone of IT virtualization in an enterprise is to put lots and lots of VMs on one or a few pieces of hardware to optimize resource sharing. This doesn’t work for OT without defining guaranteed up-time, reserved memory and CPU, real-time hitless failover, protection for OS changes (including not patching), and being secured from hostile business and internet networks. Lastly, placing OT’s VMs on a distant IT network can lead to major problems if the real-time telemetry traffic from the controls network are dropped.
IT Security compliance is also not directly compatible with OT at the lower levels of manufacturing. These systems are operating in real-time, collecting records, controlling equipment, and informing engineers of status on an HMI. Interrupting this flow of traffic, even by accident, could damage property or worse, it could harm the humans who are operating the system. Applying a patch on a live ICS network could impact data collection, controls, and regulatory compliance. There is a need for IT to understand that it could be a year or even a decade before a plant is offline for maintenance and patching. The focus needs to be one ensuring that only secured, authorized access is permitted from the outside, since the inside OT network needs to remain in a steady state for long periods of time.
In other words, it’s important for OT to explain its needs better to get the other groups to listen.
The National institute of Science and Technology (NIST) has an excellent publication that defines the core differences between IT and OT. Reviewing this publication led me to imagine a person standing between the IT and ICS columns. This was my ‘a-ha’ moment and where I discovered that I can uniquely contribute by helping customers in IT to stop, step back for a moment and review what they are attempting to take on. They often need to decide if it is within their scope to try and be successful at running an OT infrastructure or if a dedicated group would be best suited to do the job. If they are already operating the OT network and it’s not working well, a systems integrator can assist in identifying the changes needed to make the endeavor a success.
Another quick note about the NIST 800-82 publication. If you look at the diagrams 5-1 through 5-5, there is a common Control Server that remains near the controls network. In some of the designs today, this particular server has been moved out to a virtualization infrastructure.
My thought is that its importance is being overlooked since being a server, it’s being moved to the IT Virtualization in the corporate network. This object server is the first in the line of real-time telemetry collection for the manufacturing processes. Consider what happens if this is moved across a best-effort network and multiple routers and switches.
Let’s take a look at how the disconnect formed between IT and OT.
For those unaware or not familiar with Industrial Control System (ICS) requirements, the whole concept of the OT requirements seems backwards when compared to the rapidly shifting Enterprise and Consumer technologies. Someone familiar with IT networking and security best-practices at the enterprise level will rightfully get agitated at how crazy OT sounds, at least until we explain why OT is this way and it’s not easy to change without breaking the manufacturing functionality.
First, they need to get past the gut reaction, ask questions and most importantly listen. Not listening can result in a tough situation where (as in one real customer case) an IT team thought it knew better than those providing the site requirements. They didn’t apply the stated requirements and it resulted in dropped traffic across a shared business/production network. Also, they incorrectly defined VMs on a shared infrastructure. The IT department is an expert at enterprise best-effort and shared resource design, but, it just didn’t work for the ICS and had to be heavily modified.
So, my shared words of wisdom today for all IT and OT staff – don’t assume that you are more knowledgeable than the other and listen. In other words, ‘the quieter you become, the more you will hear.’
For our customers, having a seasoned Information Solutions team as part of the solution integration process can have unexpected value, including clarification of any requirements early on during the design process, and a set of educated eyes to identify any potential concerns that may cause issues in the final product.