ICS/OT Incident Response in Times of Lockdown

The restrictions put in place to slow the spread of the COVID-19 pandemic have forced us to reassess how to react to cyber incidents in OT environments. As travel restrictions were being put in place, the Dragos Incident Response team began to create plans, procedures, and tooling to enable us to still perform IR services to our customers during these challenging times. This article aims to give some guidance on how to adapt your incident response posture to the current situation.

If your current incident response plan mostly relies on your trusted IR service provider to dispatch analysts on-site and perform most of the work, it is time for a reassessment. As the pandemic continues, travel and access restrictions will prevent analysts from getting on site or slow them down significantly. Thus, you should confer with your internal and external incident responders to determine

if and how forensic data acquisition in your ICS/OT environment can be done remotely
how on-site personnel can be enabled to perform forensic data acquisition if remote acquisition is not an option
if and how analysis of collected data can be performed remotely
if regulations do not permit data to leave your environment (e.g. NERC CIP), what additional procedures must be put in place to allow for incident responders to be deployed on-site

Let’s look at these points in more detail.

Remote vs. Local Forensic Data Acquisition

In ICS/OT environments, we tend to default to local acquisition of forensic data, as in many cases there is neither enough bandwidth to support remote acquisition, nor has the use of remote acquisition software been cleared by ICS/OT equipment vendors or ICS/OT operations. The feasibility of remote data acquisition should be evaluated through a risk assessment, and that still holds true in times of a pandemic. What has changed, though, are the risks imposed by travel–both for the analysts traveling to a site and the personnel on-site. Thus, the pros and cons of remote data acquisition should be reevaluated, and quite possibly, remote acquisition might be in favor a lot more than before.

But when it comes to remote access to ICS/OT environments, the rule do not enable remote access by default still stands. In addition to performing a risk assessments of the pros and cons for your specific site(s), also refer to the standards relevant to your organization regarding remote access and VPNs. Dragos Principal Cyber Risk Advisor Jason Christopher‘s blog, A Matter of Trust. Remote Access for ICS, takes a more in-depth look into this topic.

When local data acquisition is the preferred route, you should ensure that data collection tools and media (e.g., USB thumbdrives) are pre-deployed at least to your most important sites and local maintenance personnel are getting trained in working with this tools; training is straightforward and easy, as there are a number of commercial and free tools available that are easy to use once configured correctly. USB thumbdrives pre-loaded with the collection software of your choice, combined with an acquisition checklist for on-site staff and a small collection exercise guided by the incident response team, will enable your maintenance personnel to perform these tasks effectively. This will empower them to react faster to incidents once travel restrictions are lifted and help reduce the workload of analysts–leading to faster incident response.

Remote Analysis

Analysis can’t be delegated to anyone else than incident response analysts, so regulations that ensure data comes to the analysts, instead the other way around, is key. There are three methods to give analysts working remotely access to forensic data, each with different pros and cons:

Remote Storage Vault: forensic data is uploaded to a remote server controlled by the incident response provider, then transferred into analysis systems.

Pros:

Relatively easy to set up and use if secure cloud providers are used
Data transfer is reasonably fast, depending on Internet access
Easy for analysts to access the data and import into their analysis platforms
Scales very well

Cons:

Depending on tools and storage providers used, extra care must be taken to ensure data is encrypted during transfer and at rest“ when stored in the remote vault
Not feasible for transfer of large datasets, like full disk images, if only limited Internet access is available
If cloud services are used for intermittent data storage, this might clash with organizations’ data protection regulations

Local Analysis Server: the incident response team places a server containing their analysis tools at the client’s site. The server is not connected to the client’s IT or ICS/OT network. Forensic data is transferred to the server via removable media. Analysts access the data on the server via secure remote access (e.g., VPN) over dedicated Internet connection.

Pros:

Data does not leave the client’s premises; they are on a server outside the client’s direct control
Works well, even if only low bandwidth Internet access is available
Works well with local acquisition methods
Enables much better sharing of tasks and results, compared to each analyst working on their individual computers

Cons:

Scalability depends on server sizing and can’t be expanded easily
Administration of the server potentially adds to the analysts’ workloads

Physical media: forensic data is stored on removable media and shipped to the incident responders.

Pros:

Especially if local acquisition is performed anyway, the only additional step required is shipping the media (and everyone has a standard process for shipping)
No additional technology required to be set up or maintained
Not dependent on Internet access bandwidth; while shipping might be slow compared to uploads, bandwidth is virtually unlimited

Cons:

Slow, at least for smaller datasets
Media might get lost during shipping; as there is probably no copy, data has to be acquired again (if that is even possible)
It must be ensured that acquisition tools encrypt data when writing to removable media; otherwise, additional steps for encrypting all data have to be taken, which takes additional time and is often prone to oversight

There is no one-size-fits-all solution for remote analysis. Your organization will have to determine together with your incident responders what works best. In large organizations, with multiple sites in different countries, it might be necessary to offer different remote analysis methods.

Limitations Imposed by Regulations

Some regulations, like NERC CIP, prohibit sensitive data leaving the site at all. In such cases, remote analysis is not an option, and even within the organization, remote data acquisition might be limited at best; the incident response plan must now take into account additional protective measures for on-site personnel, as well as external incident responders to enable travel to the affected sites and collaboration between internal and external staff on-site. This should include:

Providing PPE like face masks and hand sanitizers
Testing external staff before allowing them on-site
Providing workspaces for external analysts to perform their tasks, while adhering to enhanced safety regulations like workspace distancing
Setting up a process with the incident response provider that accommodates for self-isolation of incident responders before and after the engagement

Expect onsite incident response to be significantly slower, due to these limitations. Isolation requirements also mean there will be fewer incident responders deployed on-site.

In summary, responding to ICS incidents in times of lockdown requires a somewhat different approach, and remote data acquisition and analysis should be employed as much as reasonably possible. On the upside, once restrictions are lifted, safe and secure remote capabilities will speed up incident response significantly and lead to a better overall IR posture.

If regulations do not permit data leaving the environment, incident response plans must factor in the additional protection measure required during the pandemic.