OT Cybersecurity Best Practices for SMBs: Value of Change Management to Securing OT

This is our monthly blog detailing best practices for operational technology (OT) cybersecurity for under-resourced organizations by Dragos OT-CERT (Operational Technology – Cyber Emergency Readiness Team), which provides free resources to help small and medium businesses (SMBs) create or enhance their OT cybersecurity program. This month’s best practice recommendations cover the following categories from the OT-CERT OT Cybersecurity Fundamentals Self-Assessment Survey: Asset Inventory, Configuration Security, Cyber Risk Management, and Cybersecurity Incident Response. Hopefully, you filled out the survey and identified your gaps – these best practices can be implemented to begin to address those gaps. If not, there’s no time like the present – join OT-CERT and get started today.

Larger Organizations Take Note

If you have been increasing your security posture and reduced risk of a significant cyber attack in your enterprise, including your OT environment, that’s excellent news! However, does your risk assessment include the possibility of a cyber attack on one of your critical suppliers, and the impact that would have on your company’s operations? Could you still produce your product or provide services to your customers? Read on to ensure that you are quantifying the likelihood and impact of that risk correctly in light of the current threat environment. And strengthen your supply chain security risk posture by promoting OT-CERT to your suppliers!

Why Do You Need a Change Management Program for Your OT Environment?

Change is inevitable, except from vending machines. – Robert C. Gallagher

And your ICS/OT (Industrial Control Systems / Operational Technology) environment is no different. In this blog we explain why you should implement a change management process for your OT environment. And stay tuned for our next blog, which will provide a template and explain how your change management process should work.

There are many reasons to create a Change Management program for your OT environment. The most important reason is communication. Communications should begin well before the change is implemented and the key issues discussed should be recorded for later reference. Employees transfer, knowledge is lost, and patch cycles never cease. A record of change will never move to another company or win the lottery.  A Change Management program will do more than keep track of changes in a spreadsheet. A properly run Change Management program could identify potential security risks before there’s an issue, encourage cohesion between teams, and promote impromptu educational sessions about operations.

Below are a few examples of the types of situations that a Change Management program can assist with.

Recommendation #1 – Mitigate risks caused by siloes between IT and OT.

In many industries, big and small, IT and OT teams are siloed from each other. Communication between the teams is critical, and a change management process is one way to facilitate important communications between teams. It is important that both teams communicate planned changes before they are implemented to allow the other team to consider the potential risks to their operations. For example:

• If your IT team is not aware of when the next pigging of a natural gas line will occur, they could make changes to the IT environment that causes unintentional impacts to your equipment, which could lead to serious consequences.
• Major storms are forecasted several days in advance, but the network team for a rural electric co-op does not have to go out and repair lines, so they might not understand the need to hold off network changes prior to and during degraded operations.

Recommendation #2 – Reference change logs when investigating incidents.

Change logs can be used to determine whether an issue is a “blip” related to the environment or if it is the result of a change that was implemented, enabling the teams to better respond. For example:

• The Robot Line A in an assembly plant is offline for retooling and configuration changes to PLCs (Programmable Logic Controllers). The team should use change registers or meeting minutes to record the changes made to these critical assets. Furthermore, the planned changes should be communicated so that alerts created by the modification commands and uploads or downloads to equipment will not result in unnecessary investigations.
• It is 3 AM, and all of the indicators on HMI screens have gone gray. If there were no changes planned per the Change Management process, the plant operator will have more confidence to call (and wake) the infrastructure team for assistance.

Recommendation #3 – Use change logs to determine whom to call.

A record of changes, along with who implemented the change, can aid in troubleshooting and indicate whom to call to better understand a new feature. For example:

• The backups are no longer working on the Engineering Workstation after the infrastructure team did some work. Let’s reach out to the person that made the change and ensure their change did not affect our scripting or WinSCP.
• The new anti-virus software is blocking the applications on assets in the plant – HMI servers, Historians, HMI clients or DCS – from opening. According to the change management tracker, Don made changes to the A/V.  Let’s reach out to Don and figure out if his changes may have affected those systems.

Recommendation #4 – Reference change logs during disaster recovery.

In a disaster recovery situation, knowing the latest version of software and settings for each device will make recovery easier. For example:

• A fire has destroyed a rail switching system at the local rail yard. The backup images the company keeps are 2 months old, so let’s check the change management tool to see what we might be missing.
• The IT systems are under a ransomware attack due to not patching a recently released critical Microsoft patch. Let’s check the change management log to see when our latest patching cycle was and if it included that particular patch.

Recommendation #5 – Identify security implications of a change before it is made.

Your change management process can minimize the chances of creating new unintentional security issues because of a change.

• A new SCADA system is coming online, and the vendor/integrator has not supplied any local or domain group policies with the new software. Stakeholders and system owners deny the change until the vendor/integrator supplies the policies as required.
• IT Security wants to phase out the current host-based firewall in favor of the built-in Windows Firewall. During the change approval discussion between system owners and stakeholders, it was identified that in the OT environment, the host-based firewall is also the EDR (Endpoint Detection Response) solution. If the change were to move forward, then the OT environment would be defenseless against malware. Therefore, the change will be denied and IT Security will need to keep renewing the license.

Recommendation #6 – Use change logs to determine if software-related activity is unauthorized and/or malicious.

For example:

• During a breach investigation, it was discovered that an unfamiliar piece of software was installed on several Hyper-V VMs within the last two weeks. But according to the change log, there have been no changes to any severs/workstations in the last 45 days. We should investigate further.
• After a recent upgrade of the SCADA software, a new piece of software was found. It appears to be related to the upgrade, but according to the change management log, it was not tested. We should contact the vendor and perform testing to ensure it does not open security holes.

Stay Up to Date with SMB Cybersecurity Resources: Join Dragos OT-CERT!

Dragos OT-CERT offers FREE resources to help SMBs build their own manufacturing / ICS / OT cybersecurity program without hiring any cybersecurity experts. OT-CERT membership is free and globally available to OT asset owners and operators. Resources are oriented toward small and medium businesses and resource-challenged organizations with OT environments that lack in-house security expertise. Members have access to a growing library of resources such as reports, webinars, training, best practices blogs, assessments, toolkits, tabletop exercises, and more.

Currently available resources include:

• OT Cybersecurity Fundamentals Self-Assessment Survey
• OT Asset Management Toolkit
• Self-Service OT Ransomware Tabletop Exercise Toolkit
• Collection Management Framework for Incident Response
• OT Cybersecurity Incident Response Toolkit
• OT Data Backups Guidance
• Host-Based Logging and Centralized Logging Toolkits
• Access to an introductory ICS/OT cybersecurity module in Dragos Academy

If you haven’t joined Dragos OT-CERT don’t delay! Membership is open to organizations that own or operate a manufacturing / ICS / OT environment. Please join and spread the word to your community and supply chain so we can all work together to raise the security posture of the entire ecosystem – we are only as strong as our weakest link.

We look forward to working with you to safeguard civilization!