There is understandable and significant concern around the compromise of the SolarWinds software used by thousands of organizations around the world for network monitoring and management. This compromise is a highly sophisticated supply chain exposure that led to 18,000 organizations receiving the affected software. Far fewer than the 18,000 organizations had follow on activity from the adversary; public data currently supports the number to be in the dozens though the situation is evolving. However, those insights are from Enterprise networks where there is data to support those conclusions; for the ICS/OT networks there are further considerations we want to highlight:
- You may own SolarWinds even if you do not think it is present in your organization. It is not uncommon for plant personnel to purchase software on their own and beyond known approved lists. It is even more common to find SolarWinds used by ICS original equipment manufacturers (OEMs) directly or in white-labeled security products. Dragos is aware of at least two global OEMs that are using compromised SolarWinds packages across maintenance and remote connections into ICS networks. We have notified the appropriate personnel within these organizations.
- Regulations, standards, and frameworks have pushed a prevention-focused strategy in the industrial community. While Enterprise networks may have network visibility and logging, many ICS networks do not. In all of Dragos’ 2019 assessments, there were none that were doing centralized logging and network visibility prior to engaging with us. Once you confirm the compromised SolarWinds software is in your environment, if you do not have network visibility such as east-west traffic analysis and DNS logging in the ICS/OT networks, it will be incredibly hard to determine if you were breached post-compromise.
We know that for compromised instances of SolarWinds Orion, a malicious adversary had uncontested access as early as October 2019, offering ample time to plan for a multistage attack. This is an ongoing incident and there may be other access points revealed in the future as well.
If you are a Dragos customer here is what has happened already:
- On December 14th Dragos Threat Intelligence released an initial Advisory Alert to WorldView threat intelligence customers. This alert overviews the activity known to-date and includes indicators of compromise (IOCs) that can be downloaded and imported into Dragos Platform. Due to the nature of this activity, our account teams have sent this report to all customers’ main point of contact via email regardless of whether they are subscribed to WorldView.
- Beyond IOCs, the Dragos Platform has existing detections for the threat behaviors described in reporting such as lateral movement, active directory compromises, and beaconing including leveraging some of the tools mentioned post-compromise like CobaltStrike. Please monitor your detections and investigate them as you normally would.
- On December 14th Dragos OT Watch, our managed hunting team leveraging customers’ Dragos Platform installs, confirmed SolarWinds Orion installations across a range of customer verticals and issued guidance as appropriate.
- Our Incident Response team is actively working with customers to hunt for signs of compromise. On the Dragos Customer Portal, you will find the IR team has released a package that can help identify host-based signs of compromise without requiring the installation of any third-party software.
- We have published a SolarWinds playbook that can be imported by Dragos Platform customers. It is also available via the Customer Portal; we will update this playbook as necessary.
If you are not a Dragos customer:
We recommend reading both CISA Alert AA20-35A and FireEye’s research on SUNBURST. These provide context and insights into the tactics, techniques, and procedures observed to-date in the campaign.
Key Takeaway: if you do not have monitoring and centralized logging in your ICS/OT networks then you need to go hunting. Simply checking for IOCs will be insufficient. Confirm if you have the compromised version of SolarWinds. If you do, and do not have monitoring/logging going back 9+ months, you should assume that you’re compromised and go hunt.
How to determine if you are directly affected
- Verify within your asset inventory or CMDB if you have SolarWinds Orion installed in your environment.
- If a complete asset inventory is not available, it might be prudent to verify management servers for Solar Winds Orion installations. Known affected SolarWinds versions include 2019.4 HF5 through 2020.2.1.
- Identify if the backdoor DLL is present on the system. Identify if further tools are present on host memory or disk.
- Verify your IT communications protocols (DNS, firewall, web proxy) for outgoing DNS requests and communication to the domains and IP addresses provided in the appendix.
- If part of your IT/OT network operations are maintained by partners, they should follow the steps outlined above to check if the environment is at risk.
What to do if you detect a compromised Solar Winds Orion Server
- Finding the backdoor does not confirm that adversary conducted operations. Dragos recommends activating incident response plans and performing a combination of host and network forensic analysis to include:
- Unexpected user account authentication
- Lateral movement between key assets such as the domain controller
- Network activity or network attempts to associated IP addresses or domains
- If the affected server is a physical machine, disconnect it from the network, but leave it running (If it is a virtual machine, pause it). This step ensures we preserve valuable data for incident analysis and response.
- You should now follow the plan and playbooks to collect forensic data from the affected systems.
- If you do not have an incident response plan, ensure that you can preserve the data and call a trusted provider. You can reach out to your account team for more information or contact us directly at ir@dragos.com. If you don’t know who your account team is, contact sales@dragos.com.
Further Considerations for ICS/OT installations
At first inspection, a Solarwinds installation in a DMZ may not appear high risk. However, it is important to understand SolarWinds’ role in the overall architecture. It is very possible that a Solarwinds Orion installation is designed to actively poll SNMP of field equipment or PLCs at Purdue Level 1 or Level 2. If there are no firewalls between Solarwinds and the monitored devices, this could allow the adversary to directly interact with field equipment or PLCs. Even if there are firewalls between Solarwinds and the devices their access control may be overly permissive, still allowing for unfettered interactions with equipment. Finally, beyond firewall rules, if the SNMP traps allow for setting data values (changing configuration), the firewall rules are not effective and could still permit device modification or even remote exploitation.
Considerations when conducting your assessment across one or more facilities:
- Make sure you assess all networks to include backup, failover, and test networks.
- Understand what third party networks exist. OEM, support vendors, or maintenance and diagnostic services are known to use Solarwinds.
- Treat your corporate network as a third-party network to your industrial environments. Particularly, if you own critical infrastructure and have a compromised Solarwinds installation you will want to disprove not only if there was a breach on the corporate network, but that it did not extend into your industrial environment or focus on exfiltration of critical infrastructure data, plans, schematics, security posture, etc.
NERC CIP Considerations
With an estimated impact of 18,000 customers, it is probable that some of the nearly 2,000 NERC CIP regulated power utilities in North America have been impacted—if not directly, then indirectly via their supply chain. The supply chain, in particular, is a difficult conversation and relatively new for utilities. Under CIP-013, which became mandatory and enforceable in October 2020, any new contracts required provisions for vendor incident notifications, remote access, and additional procurement language to improve both security practices across vendors, as well as improved collaboration during a cyber incident. A prudent first step in managing the “blast radius” of SUNBURST would involve each utility asking their vendors if they utilize SolarWinds Orion, especially if the vendor has access to Bulk Electric System Cyber Systems (BES CS) or Bulk Electric System Cyber System Information (BES CSI). For the vendors that do, each utility should coordinate a response to limit or remove access, where reliable operations would not be impacted, and voluntarily perform a threat hunt where access cannot be revoked for reliability reasons.
There may also, however, be utilities that have installed SolarWinds in their High Impact Control Centers, or maybe even a Medium Impact facility, where they can leverage the larger suite of SolarWinds NERC CIP Compliance reporting. In those cases, auditors will be looking for a few potential tasks and may review artifacts based on each utility’s unique response. Beyond CIP-013, auditors may have a discussion with utilities regarding malicious communications potentially identified in CIP-005 and the tools used (especially if SolarWinds is the only installed solution), as well as how the hotfixes and patches were installed (or other mitigation plans approved) where BES Cyber Systems using SolarWinds Orion products were identified. While CIP-007 allows 35-day windows for both evaluating the patch and mitigating/installing the patch, there may be additional scrutiny in how the utility responds due to the high visibility of this incident.
This brings up the last potential impact for NERC CIP utilities: incident response. Under the current CIP-008-5 requirements, a potential investigation for SolarWinds Orion in a NERC CIP regulated BES Cyber System could become a “Cyber Security Incident.” That said, unless it impacted the Reliability Task of the BES CS, it would not be a “Reportable Cyber Security Incident.” However, starting in January 2021—mere days away—if the same attack on SolarWinds (or another vendor) were uncovered, this could potentially impact the new undefined term “attempt to compromise” in CIP-008-6, featuring the latest version of NERC CIP incident response requirements. In which case, each utility would have new reporting responsibilities to both NERC and DHS, including follow-up reports, that should be exercised and fully understood. While it does not impact today’s incident, it will undoubtedly have impacts on future supply chain attacks.
Ready to put your insights into action?
Take the next steps and contact our team today.