Developing and Executing a Fully Informed OT Threat Hunt 

Written in partnership with Michael Gardner, who previously worked as an Intelligence Technical Account Manager at Dragos, Inc.

Threat hunting is an intimidating topic in security operations discussions – and becomes even more so when plans are made to traverse the information technology – operational technology (IT-OT) boundary in scheduled hunts. Despite the intimidation factor, OT-based threat hunts can be a fruitful process to improve security posture and awareness of operational networks. This blog will guide security analysts through one method of executing a successful threat hunt, beginning with planning and ending with a strong report on findings. 

For more on OT threat hunts, listen in on Control Loop: The OT Cybersecurity podcast, Episodes 25-27, where I discuss threat hunting in depth with Michael Gardner and Mark Urban.

Hypothesis Development 

Executing a fruitful threat hunt requires advanced planning and knowledge of the adversary. This is where the development of hypotheses becomes most beneficial to hunt execution. Hunters should work with intelligence analysts and sources to determine what types of threats are most likely to target their organization. Industry threat landscape knowledge and obtainment of intelligence sources satisfying intelligence requirements can also help make these determinations. It’s important to prioritize hunting efforts based on the most likely threats to impact the organization first (i.e., intelligence requirements your stakeholders care about), so some rudimentary analysis goes a long way to a successful hunt. Once an intelligence-driven hypothesis is generated – for example, “Adversaries are leveraging Jaguar Tooth to exploit a vulnerability in Cisco routers” – hunters can move on to planning, most importantly by determining where to hunt. 

Where to Look 

A hunter lives in organizational first-party data sources. Having a collection of these sources is critical to a successful hunt – but identifying gaps in the collection may be a success in and of itself (more on that later). It’s important to scope out where you will look for adversary activity in that vast data collection.  

In the hypothesis example: “Adversaries are leveraging Jaguar Tooth to exploit a vulnerability in Cisco routers,” we’re looking at adversary exploitation of Cisco routers. The most important thing here would be consulting our asset inventory and collection management framework (CMF) to determine where vulnerable or historically vulnerable Cisco routers may sit within our tech stack and how we log data around those routers. In our example, we’ll assume having Cisco routers in both the corporate/enterprise environment and some below the DMZ in operational networks. (A DMZ, or demilitarized zone, is a perimeter network that adds an extra layer of security to an organization’s internal local-area network to protect it from untrusted traffic.) For this example, the corporate/enterprise environment data will be found in Splunk, a software for searching, monitoring, and analyzing machine data. Data around routers in OT is in the Dragos Platform. Therefore, Splunk and the Dragos Platform will be our pivotal hunt sources for looking for this activity.

However, you may need to identify some things that could be improved in the asset inventory and CMF. Hypothetically, let’s say we identified 11 routers in a specific domain that do not yet have logging enabled. Boom! That, right there, is an actionable, impactful hunt finding for reporting later. We need to ensure we’re logging onto those devices, that vulnerability management is monitoring their versions and patch status, and that we can pull logs to review for historic exploitation attempts.  

This is a very simple example, and there may be other data sources to consider. For example, an intelligence provider may provide indicators of compromise associated with past exploitation attempts – let’s say we have some command and control (C2) IP addresses. We will also want to consider firewall egress and ingress data, proxy logs, host logs, and more in our hunt activity. The list can go on infinitely and fully depends on the hypotheses you’ve set and your goals for proving them. Throughout your hunt, you should document your actions and their results to prepare for a retrospective when the hunt is complete. 

Satisfying Our Hypothesis  

We mentioned it already, but a successful hunt does not always mean you found adversary activity and, therefore, “proved” your hypothesis. Successful hunts identify areas for process improvement, gaps in visibility, development of the analyst’s skillsets, further domain knowledge, and more. But some hunts may lead to discovering suspected adversary activity, whether hypothesized or not.  

Rule #1, with this discovery, is: Do Not Panic! You’ve been documenting your actions and findings, and you developed an intelligence-driven hypothesis early on, which contains background on this type of activity, but you’re just hunting. These findings mean you need a second opinion.  

Package up your findings in a ticket or tentative hunt report. Engage your Incident Response team to validate these findings. Offer close collaboration as incident management begins to help determine the extent and scope of potential impact. 

Meeting Threats Where They Are 

Organizational maturity, especially in OT, varies drastically across the board.  If you are not ready to begin dedicated threat hunting cycles in OT, that does not mean you’re inherently at risk more than anyone else, but moving toward a state of maturity where threat hunting is enabled is a drastic improvement to your security posture. There are three basic questions to consider for movement on this maturity curve: 

• Do you have visibility into the environments that adversaries target?
• Do you know what and where adversaries target in an environment like yours?
• Do you maintain the ability to respond to an incident in these environments?

Don’t Forget IT and Stage 1 

When doing a threat hunt, there are many places to start to look. One example is from the “outside-in,” meaning you are looking at what is targeting your network. You examine what threat groups are doing to target or impact industrial infrastructure. One of the more obvious places to start is to ask what ports/services does operational technology have exposed to the internet? Is this device vulnerable? And more importantly, why does the industrial organization have OT-specific assets hanging on the internet? But as threat hunters, we would be negligent if we weren’t identifying and analyzing the industrial control systems (ICS) Cyber Kill Chain Stage 1 incidents. For example, is the threat group targeting specific ICS/OT engineers with a purposefully created spear phishing email? We are also examining the capability that might be sent with the spear phishing email. Does it look like it has lateral movement capabilities? Or is the threat group using built-in Windows operating system tools to move throughout the network? Can we identify the intent to impact the OT environment? These are just a few of the questions OT threat hunters can consider during a hunt.  

Know Your Baselines or Find Someone Who Does  

Understanding your network internally and what sits on the internet will benefit your security team greatly. Conducting any threat hunt without a baseline won’t allow you to decipher between normal and anomalous activity. This doesn’t just include knowing every asset on your network (which we know is difficult) but is more focused on understanding what type of activity is permitted on systems and network enclaves. Threat groups aren’t necessarily getting more sophisticated, but we are noticing a continued increase in threat group tactics to blend into the network. For example, this means using built-in Windows system tools to navigate and make changes on a victim’s network.  

The Definition of Success 

A key goal always seems to be “find adversary activity.” Unfortunately, the reality is, that isn’t the case.  A successful threat hunt is not based on finding adversary activity. But instead, success is based on proving or disproving your hypothesis. A few follow on, post-hunt questions can be considered before creating a new hypothesis to investigate:  

• Were we looking in the correct place? 
• Did we have the correct data? 
• How stale were the indicators we were using? 
• Did I stick to the hypothesis? 
• Did I scale my hunt appropriately? 
• Was my hunt applicable? 
• Did I define a scope that made sense to my organization?   

The list could go on and on. However, at this point, it is best to create a new hypothesis, mentally answer the above questions (and any other questions that came out at the end of the previous hunt) and conduct a new threat hunt.  

The hardest part as a threat hunter is to be aware of “analysis paralysis.” What we mean here is to stick to answering your hypothesis but knowing that at some point during the threat hunt, there must be an “end.” Concluding a threat hunt is just as important as starting one. Without a conclusion, you won’t understand what new data or other indicators you need to explore further. 

Your Findings 

The product of a threat hunt can contain many items; typically, it culminates in a report of your findings. These reports fall into three categories – strategic, operational, and tactical.  

The simplest form is:  

• Strategic – high-level details that the executive and C-suite would understand. 
• Operational – tactics, techniques, and procedures (TTP) of threat(s). 
• Tactical – indicators of compromise (IOC) and their details. 

Strategic reports can still cover all the details of the threat but do not have to be a deep dive into all aspects. The most important items to address in your report are the WHO and the WHY. It should cover the analysis conducted, recommendations, and lessons learned during the threat hunt. 

An operational report will touch on the adversary’s TTPs and the associated threat. In this report, you will touch on the HOW and WHERE. At times you will find some of these details in strategic reporting, but again that is more high-level and less into the technical details that you find in operational reporting. This will cover detections, data sources, and new processes for future threat hunts. 

The last piece is tactical. This is where you can get into the nitty-gritty details, digging into the WHAT is associated with the incident/threat/adversary. In this report, you will discuss the IOCs and their significance. All the details should be included in this type of reporting. This will cover all the details related to the IOCs and mapping to the TTPs. 

Hunting for OT Threats with Dragos Threat Intelligence 

Dragos WorldView is an annual subscription service that delivers actionable strategic, operational, and tactical reporting on industrial infrastructure threats. With a primary focus on adversary activity and capabilities used in OT and ICS, Dragos also provides threat intelligence on early-stage adversary activities to help bridge the visibility gap between OT and IT teams. The outcome – finished threat intelligence that is packaged and delivered for use across multiple settings and audiences, including as a starting point for ICS/OT threat hunting. After that, Dragos Threat Intelligence indicators, vulnerabilities, and detections are codified for operational technology facilities leveraging the Dragos Platform. 

Get free sample reports of Dragos WorldView to see the types of intelligence we provide.