This blog is part two of a three-part blog series adapted from Principal ICS Security Analyst Austin Scott’s DEFCON 2019 presentation. The full DEFCON slide presentation can be viewed here:
In the second part of this three-part series on ICS Purple Teaming, I will discuss the unique challenges when working within ICS environments.
ICS Assessment Specific Challenges
Regardless of the specific type of assessment performed, ICS assessments face some unique challenges compared to their IT assessment cousins.
ICS Safety and Reliability
Safety & reliability are of paramount importance within ICS environments.
Most ICS sites have a strict culture of safety. Any behavior that is considered to be unsafe can get a contractor barred for life from a site. Safety infractions can include:
- Improper Personal Protective Equipment (PPE) for the area you are working (Safety glasses, H2S sensors, steel-toed boots)
- Not having the required safety training (H2S alive, Confined Space training, First Aid etc.) to access a site
- Wandering into restricted areas
- Taking photos (as fire eye sensors can be triggered by a camera flash) without permission
- Taking a phone or electronics in a Class I, Division 1 area (an area where ignitable concentrations of flammable gases exist)
- Not holding hand rails when ascending or descending stairs
- Touching industrial equipment without the authorization of the site operators (or touching ANYTHING without permission).
- Speeding in parking lots or site access areas
The site operations team and engineering team are very sensitive to the reliability of the system. Putting an ICS system into an unknown or unrecoverable state can be:
- Dangerous to the people at the site
- Damaging to equipment
- The cause of costly outages (each hour of downtime is sometimes measured in millions of dollars)
Performing any action that even has a remote possibility of harming an industrial process is out of the question. Most sites we work in do not allow us to introduce any packets into the ICS Purdue Level 1 or Level 2 network. However, we do have alternative “passive” methods of effectively collecting and reporting on data in critical ICS networks. Passive data collection methods often include:
- Packet captures from span ports of ICS network equipment
- Manual data collection using batch scripts/bash scripts
- Reviewing data in virtual images, backup files, testing or training environments
We will use “active” methods opportunistically if we can guarantee the running process will not be impacted and have explicit permission from the site operations team.
If Testing ICS Networks is Risky, Why do it?
Some might ask, why should we take the risk of performing an assessment against an operating ICS network if it is so dangerous?
We perform assessments on ICS networks because activity groups are actively targeting them.
Constant communication, careful planning, and experience working in these environments is required to be successful. Furthermore, during engagements, we actively seek ways of mitigating risk such as:
- Testing in a lab/training environment when possible
- Setting up a test environment or virtualized environment
- Testing during a planned outage window when possible
ICS Specialized Equipment and Crown Jewel Assets
There are many proprietary protocols, engineering tools, and wireless and operational technologies that are unique to ICS. It is critical for the Red Team to understand these technologies and the risk they could pose to the operating assets and the company as a whole. During our assessment, we will work with the operations and engineering team to identify the Crown Jewel assets of a site. A Crown Jewel asset is a piece of equipment that is of critical importance to the safety, reliability, or profitability of the site.
For example, at an oil and gas production site, a customer might have a dozen disposal wells used for waste disposal. The same site might also have only three custody transfer metering systems which act like the cash register for the entire plant. The operations team would care much more about a vulnerability found in their custody transfer meter than in a disposal well. Without a custody transfer meter, the site is unable to accurately ship product which could cause the entire facility to grind to a halt. Whereas, losing a few disposal wells is not likely to have a significant impact on the entire facility in the short-term. Therefore, the custody transfer meter is a Crown Jewel asset and the disposal well is not. Extra care must be taken when working around the identified Crown Jewel assets. At the same time, finding vulnerabilities and methods of remote access to these Crown Jewels is a priority.
ICS Communications Overhead
Communication with the site operations team (operators, engineering, IT, cybersecurity, physical security) is critical during an ICS assessment. Most site operations teams want to know:
- What you plan to do
- Where you need access to in the plant
- When you plan to do it
- Who you need access to in order to complete various tasks
Providing an agenda of the assessment well ahead of the assessment site time allows the site operations team to make the required resources available. Finally, remember that no plan survives the battlefield and the plan typically changes the moment the TOC team arrives on site.
When I used to perform system integration for SCADA and PLC equipment, an experienced engineer once told me: “Everything takes twice as long to complete as normal when you are on an ICS site.” This was true for ICS system integration projects, and it is also true for ICS cybersecurity assessments. The TOC team often has operators looking over our shoulders while we work or request that we project what we are doing to a conference room screen. Daily standup sync and status calls are normally performed at least once per day while working at an operating asset.
Communications overhead and coordination of resources often will consume up to a third of the time on site.
ICS Cultural Challenges
It is not uncommon to encounter varying levels of distrust between the different engineering disciplines or between the site operations team and the corporate security teams at ICS sites. Many sites (especially remote ones) have limited interactions with people from “corporate” and run as their own self-sustaining plants. Furthermore, plants have often been bought, sold and traded by their owners and perhaps never fully integrated into the larger corporate culture. Trust is difficult to earn and is easily lost, but is a foundational requirement for a successful assessment of an operating ICS. Entering into an assessment where the site operations team does not trust the ICS assessment team is very difficult.
Ready to put your insights into action?
Take the next steps and contact our team today.