Threat Hunting Part 1: Improving Through Hunting

This post is a first in series that will describe hunting, discuss best practices and explain our approach and lessons. Because hunting in industrial infrastructure is important to all of us and with focus and effort we can accomplish it. Part 2

Industrial control system (ICS) networks are very defensible, yet it seems increasingly difficult to convert them from defensible to defended environments. There’s a sense, an intuition, that they are ripe for being hacked. For years, the internet device search engine Shodan has shined the light on internet-connected industrial devices. ICS-CERT has annual incident response engagements measured in the hundreds as well as a myriad of news stories articulating even more. And of course, there are a handful of partially understood campaigns such as Dragonfly and Sandworm that targeted industrial sites around the world leveraging HAVEX and BLACKENERGY 2 respectively. We’ve only had four cases of ICS tailored malware deployed against targets: STUXNET, HAVEX, BLACKENERGY 2, and CRASHOVERRIDE. And yet we understand we have an ICS threat landscape that is mostly unknown due to a lack of visibility into the environments and threats we face.

Industrial environments have two fundamental security challenges: Lack of visibility into what devices exist (and their function) and the expertise to safely respond when something is found. Anecdotes of steel mills, power outages, and espionage are powerful but incomplete and unsatisfactory. Our MIMICS project  tried to address this anecdotal problem by trying to measure public sources. Our project continues today, made available to our WorldView customers, and it is constantly evident that adversaries are active in this space.

Visibility is a hard problem. You cannot defend something that you cannot understand, measure, and be knowledgeable on. We keep trying to answer questions that are not fully articulated or are based on copy/pasting best practices from IT security and not necessarily finding the right solutions for ICS. This is combined with the knowledge gap of industrial incident response. There just isn’t enough prior examples, lessons, tools, or practices to generate a community of industrial incident responders. We as a community have made traction in areas such as advocating for network security monitoring (though it’s no magic bullet), focusing on firewalls and perimeter defense (though our perimeters are quickly eroding), and trying to structure active defense mechanisms such as incident response (but, again, with little experience).

Threat hunting can address these challenges.  Threat hunting creates a proactive and iterative process to find threats and by doing so deeply understand the networks, processes in place, and gaps in security postures. It allows for a fine-grained understanding and reporting of these environments. If you have unknown opportunistic malware or a sophisticated actor operating in your environment, then you are best positioned to detect it. Many proactive analysts have successfully done this on an ad-hoc basis. Hunting is about moving the blue team from art to science.

There’s huge opportunity to take this concept to iteratively increase our visibility – to turn anecdote into risk level understanding.  To make defending ICS environments doable.

Some examples that illustrate today’s challenges:

• Cognitive Vignette: There’s this one analyst that knows everything; she knows your system in and out and can scope and troubleshoot root cause. She’s been asked to train several folks over the last few years but with little success. How can you pull that information out of her brain and spread it into the team effectively? Hint: this is a tacit knowledge problem.
• Organizational Vignette: Everyone on the team knows the recent new deployment was a flop. It detects less than the previous system, requires constant upkeep, and is a general pain. Alyce, the lead analyst just gave her two-week notice. Yet your bosses boss hails it as a success. The audit team is happy with the new reports and won’t recognize that serious gaps in protection into the plant network exist. Your boss has no way to illustrate the new weaknesses; nothing empirical. He trumpets the success while downplaying the unexpected work and new timelines to return to the same protective posture as before. Everyone is waiting for the day when the fog evaporates, and the director sees the true situation for what it is. How can you speed up getting out of this rut? Hint: this is an organizational learning  problem.
• Process Vignette: The vulnerability assessment of all the cyber assets in your control center is complete. Bad news, since the last year’s assessment there are several new IP addresses, a ‘test’ firewall ruleset that allowed egress traffic, and several mischaracterizations of assets in your documentation. How can you fix the mess while preventing the same problems next year? Hint: this is indicative of process problems.
• Threat Vignette: You found malware on a vibration monitoring HMI after plugging in a technicians USB, and the anti-virus lit up. Sarah looked at it (two days before she announced her two-week notice) and noted the malware has been on there since bringing the plant operational back in 2009. She confirmed it’s been attempting to communicate to the Internet, but all the requests have been dropped by the firewall. Alyce saw a misconfiguration in the firewall that prevented the activity from alerting the firewall team. How did this go unnoticed for eight years? Hint: As in the real-world, we often have several systemic issues. These systemic problems are increasing risks in unknown ways.

Hunting is regarded as the latest trend in finding threats and reducing dwell times in IT networks, but it is simply the codification of what has worked over the years into formal approaches that can drive sustainable success. Hunting requires the assumption that breaches can and do happen. While ICS environments are definitely a target, and definitely have breaches, it becomes a hard sell to hunt in an ICS environment and routinely find zero threats. It is a high effort process, and if it is measured only in finding evil, it will have a very low return on investment.

But hunting in ICS requires accepting more assumptions to create a return on investment:

The assumption that your security program has unknown blind spots.

Threat hunting helps to prove your people, processes, and technology actually does what you say they can. Threat hunting first and foremost recognizes that red teaming is not nearly as effective as the blue team validating their systems work. As the last vignette suggested, hunting gives organizations the empirical data they need to escalate issues when a new deployment doesn’t meet expectations.

The assumption that your staff requires hands-on experience to couple with the training they get.

Hunting, if done right, can create the framework for an analyst to learn the environment and skills needed to scope, troubleshoot, understand, and synthesize new information about the environment. It’s how you can successfully move the knowledge out of one person and into the team. How you put into action the investments in training you make.

To conclude with three immediate benefits of hunting:

• Proves your network understanding. Initial hunts by the TOC in new customer environments quickly develop a sense of how well managed the environment is. We’ve come across HMIs inappropriately querying backend SQL servers, IPv6 tunnels, faulty network cards, misconfigured network settings, undocumented assets, and more. These initial findings underscore how proactive collection and analysis can be used to understand the network and assets. Understanding these assets is critical to begin to defend and secure them.
• Proves your security visibility. Applying hunting methodologies to your security controls will likely demonstrate weaknesses in both your visibility and protection. Hunting can and should lead to lessons for improving and a roadmap to do so. Too many times a team will not recognize this weakness until it matters – when actually responding to an incident. Hunts create a proactive feedback mechanism where lessons can be iteratively improved upon.
• Matures your ability to respond. The overlap of data, skills and thought patterns between hunting, incident response, and forensics is significant. Performing hunting will mature and grow your ability to detect malicious activity but at the same time also improve your ability to respond to an event – whether triggered through hunting or another means.

In the next post of this threat hunting series, we look at how the unique characteristics of industrial networks can be used to an advantage as network defense professionals.