The Shifting Landscape of OT Incident Response

I have a strange, unique, and fascinating job at Dragos. For the last 6 years, I have served as a cybersecurity incident responder and digital forensics professional exclusively for our customer industrial networks. What that means in layman’s terms is that when say, a water treatment plant or manufacturing device is infected or potentially hacked into, my team is one of the few groups of people on Earth who respond to it.

Our OT incident response cases vary vastly. We respond to ransomware attacks against massive Fortune 500 companies’ factories. Other days we respond to intrusions or insider threat cases for tiny municipal utilities, where there’s only a single IT resource. Still others we might supplement in-depth forensics efforts of low-level industrial devices. What is consistent is that performing incident response and forensics in these environments is very different than Enterprise cases. Process and vendor integrations often require legacy and manual forensics, as well as custom tooling for unique vendor firmware and interfaces. We frequently see Windows 2003 or older operating systems. There is little ability to safely use modern forensic agents broadly in most environments. Everything centers around life and safety.

However, more recently I have experienced an intriguing shift in incident response cases that I believe reflects a positive growth in OT cybersecurity awareness across various verticals.

Explore the Data

Increasing Trends in Dragos Retainer Activations

Dragos Incident Response saw an increase in incident response retainer activations specific to three general situations.

Triage of Long-Term Compromise and Infections: We have seen an increase in customers with an interest in scoping and creating removal plans for long-term infections (think, 5-10 years) and architectural compromises of their industrial environments. Safe industrial operation requirements make it extremely challenging to do mass clean-up and reimaging efforts in process facilities. Many facilities have maintained a level of infection and compromise for years and deemed it too costly or high risk to conduct mitigation activities. However, these points of compromise can cause eventual operational and technical impact, in an unpredictable way. Interest has increased in understanding the scale of the problem and “projectizing” removal in a safe way.
Investigation of OT due to IT or Supply Chain Compromise: While we respond to many malware and insider threat cases which bridge ineffective DMZ boundaries between Enterprise and OT, we have recently seen an uptick in requests for early involvement in ostensibly Enterprise-only or vendor-only compromises of our customers. Given increasing network and cloud integrations, they want an adequate level of confidence that the intrusions impacting one portion of their network, or a partner have not spread into sensitive process operations environment.
Cybersecurity Forensic Analysis of OT Process Incidents: Finally, we have been more frequently involved early in cases of physical industrial incidents, where a root cause has not reliably been determined. Customers are contacting us as part of the overall effort to analyze both high- and low-level devices, logs, and network behaviors – to identify or rule out digital causes such as device tampering, infection, or misconduct.

What Does This Mean?

I find the increase in incident response calls for these types of cases much more heartening than concerning. They indicate a general increase in cybersecurity maturity and awareness in industrial process environments. They also indicate that cybersecurity is being integrated more effectively into business continuity planning and risk management processes at early stages. As commodity, insider, and state-sponsored cyber attacks against industrial networks increase, it’s encouraging to see our industrial customers willing to involve us early and routinely into their investigative processes to ensure they aren’t missing potential intrusions or misuse of their digital equipment.

Our incident response case load will continue to encapsulate a wide array of verticals, organizations, and incident types. Some of the cases will always be deeply troubling and hugely impactful to peoples’ lives. However, seeing the industry progress towards better cybersecurity maturity and ability to detect and respond to threats gives the Dragos Incident Response team hope.