CrowdStrike update causes global impacts

Executive summary 

A faulty update in CrowdStrike's cyber security Falcon Sensor software caused thousands of computers worldwide to crash causing massive operational impact to businesses. This was caused by a coding error and was not a cyber-attack. 

CrowdStrike rolled back the faulty update and published a manual workaround to fix the issue. As at the time of writing. the workaround is not complex, but it is not scalable as it needs to be applied manually, system by system. With systems spread between on premises and cloud it can take significant time to recover. We have provided details of mitigation measures in the body of this document (based on the latest information provided by Crowdstrike and Microsoft Azure at the time of writing; check their sites for updates). 

Users of Microsoft (MS) Azure with Virtual Machines using Windows Client and Windows Server, who had the CrowdStrike Falcon agent installed, were also affected. MS have published some advice to mitigate the issues.  

Simultaneously, MS O365 also warned that services were degraded, meaning that some users may be unable to access various apps and services. It is not known if all MS issues were linked to the CrowdStrike problems.  

It is highly likely that multiple service providers and supply chain businesses will be affected by the issues, meaning that virtually all businesses will have some level of impact. Although a comprehensive fix is likely in the short term, these issues are likely to persist at a lower level for some time. 

Businesses should consider how to handle customer communications and queries because of the outage. 

If businesses have suffered losses or potential losses because of business interruption, they should be examining their insurance policies for potential claims, and ensuring that they notify their insurers as quickly as possible. 

Businesses should also consider whether they need to notify their regulators, and review their contracts with suppliers and B2B customers to confirm what contractual processes may need to be followed.

Incidents such as this are uncommon but will almost certainly attract executive attention. Cybersecurity leadership should be prepared to resist urges to downgrade security updates as a result. 

The incident underlines the risks of adding software to endpoints as updates cannot always be trusted. Cloud outages should be considered as part of business continuity planning, once immediate fixes are implemented. This could include running test scenarios in future to assess response capabilities. 

Major global IT outages often give rise to claims and litigation further down the line, as parties seek to recover any losses that they may have suffered because of the downtime. The actions that your business takes in the early stages of any IT outage can be critical in any litigation that follows. This briefing sets out the steps that can be taken in the first hours and days in order to maximise your chances of successful outcome. 

This note is based on the information available at 14: 30 UTC on 19 July 2024. 

What happened? 

From 18 July 2024, thousands of Microsoft windows machines across the globe experienced the “Blue Screen of Death” (BSOD) issue when booting up.  

Reports indicate that the issue was caused by a faulty update to the CrowdStrike Falcon Sensor cyber security software, widely used in businesses.^{1 ,2} 

The issue has affected banks, airlines, rail providers, supermarkets and many more businesses, causing widespread operational issues. 

Microsoft Azure users affected 

Simultaneously the issue has affected cloud service MS Azure. On 09:10 UTC 19 July, MS Azure stated³ that they were aware of customers using Windows Client and Windows Server software on Virtual Machines which had the CrowdStrike Falcon agent running being affected by the BSOD issue.  

Microsoft O365 degradation  

Simultaneously, MS O365 warned that services were degraded, meaning that some users may be unable to access various apps and services. It is not known if MS issues were linked to the CrowdStrike problems.  

The issues were detected from 18 July 2024 at 21:56 UTC and were ongoing to at least 19 July 2024 at 08:30 UTC.⁴ 

The fallout

As the software is widely used by many businesses, the issue will almost certainly have severe short-term direct, and indirect supply-chain impacts for the business operations of many companies globally and low-level impacts for some time. 

Businesses should also consider how to handle customer communications and queries as a result of the outage. Other important first steps will include contacting insurers, regulators and your supply chain (see below).

As the technical workarounds provided below are largely manual, the recovery time for many businesses will likely also affect operations unless a quicker workaround can be provided. 

The incident underlines the risks of adding software to endpoints as updates cannot always be trusted. Cloud outages should be considered as part of business continuity planning, once immediate fixes as implemented. This could include testing similar scenarios in future to test response capabilities. 

Although this incident has clearly demonstrated the risks of downtime from third-party cybersecurity providers, it is a highly unusual and uncommon occurrence and is unlikely to be repeated in the short to medium term. Cybersecurity leaders should therefore be ready to resist pressure to downgrade security and their security posture due to executive fear of updates. 

Technical actions to resolve the issues  

CrowdStrike has reported making changes to resolve the issue, however, they have published workaround steps for those hosts that were unable to stay online to receive the “Channel File Changes” still crashing as follows:  

• Boot Windows into Safe Mode or the Windows Recovery Environment 
• Navigate to the C:\Windows\System32\drivers\CrowdStrike directory  
• Locate the file matching “C-00000291*.sys”, and delete it. 
• Boot the host normally. 

Our cybersecurity team have received at least one report that the fix works for on-premises servers but there were difficulties for those using MS Azure cloud services.  

Microsoft Azure recommended those able to restore from a backup. Those that could not were advised to repair the Operating System disk offline by following these instructions:  

• Attach an unmanaged disk to a VM for offline repair 
• Disks that are encrypted may need these additional instructions: 
• Unlocking an encrypted disk for offline repair 
• Once the disk is attached, customers can attempt to delete the following file. Windows/System/System32/Drivers/CrowdStrike/C00000291*.sys 
• The disk can then be attached and re-attached to the original VM.  

Microsoft has recommended that customers that were continuing to experience issues reach out to CrowdStrike for additional assistance. 

Key Steps

There are a number of steps that can be taken in the first hours and days in order to maximise the chances of successful outcome. 

Insurance

Clients should review their insurance policies urgently to see what cover they have in place and whether insurers should be notified of the incident and the potential losses that may arise. Insurance policies contain conditions, particularly in relation to notification and actions to be taken following an incident, which must be strictly complied with to preserve the claim under the policy. Delays in notifying insurers could prejudice those claims. Cyber policies will be the most relevant. However, insurance policies covering business interruption and liability to third parties should also be reviewed.

Technical fix 

It may sound obvious, but a quick fix will help to mitigate losses, and it's always better not to suffer a loss than to later bring a claim to recover that loss. Organisations should dedicate the resources needed by their core IT teams and trusted third party vendors to get critical systems back online as quickly as possible and reduce the impact of the outage. It is important to focus on this before looking at who is to blame and what legal remedies may be available – but it's also important to keep track of steps taken and any costs incurred, as these may form part of a damages claim later. 

Communicate with customers

Organisations should use all available comms channels to communicate with their stakeholders. If unable to contact customers directly, social media and other comms channels can be used to get more general updates out. It's important to let customers know which systems may be affected, what is being done to try to get systems back online and how they can contact the organisation for further information. However, it's also important to avoid creating hostages to fortune by giving information based on incomplete information or making promises about when systems will return to normal, and steer clear of any statements that might be deemed to accept liability for outages. 

Communication should be two-way if possible. We advise organisations to test the temperature of their customer base: what are they complaining about, what losses might they be suffering, who is likely to bring a claim later? Critical information can be gathered during this period that may be useful in bringing or defending a claim in the future so it's important to record all of that information so that it can be relied upon later. 

Speak to your regulators where necessary

Businesses operating in a regulated environment may need to notify their regulators, for example where customers are unable to access accounts for a protracted period.

Don't make it worse! 

At times like this, organisations need to be vigilant – bad actors often use the chaos of a major IT outage to gain access to systems. CrowdStrike have already issued guidance to only rely on information about fixes from trusted sources, so ensure teams are aware of that and that they are not relying on unsolicited advice which may be coming from bad actors. 

Organisations can use manual or analogue workarounds where possible, but should ensure they don't compromise safety and security as this could create further problems down the line, particularly with regulators. 

Immediate review of contractual arrangements 

There is a risk of additional liability by failing to comply with contractual requirements in the event of IT outages. Businesses should check B2C and B2B contracts and follow any pre-agreed processes, for example: 

• whether they are required to notify B2C or B2B customers or other third parties such as regulators in the event of service interruption, system downtime or force majeure
• whether there is potential for a data loss incident, and if so whether companies need to take steps to notify third parties (including data regulators and data subjects)
• if you are required to put fixes or make alternative back-up systems available, ensure these are put in place. 

Early indications appear to suggest that this is not a cyber-attack and there are no third-party bad actors, which may make it harder for vendors to avoid liability to customers. 

Other legal issues

Organisations should review contracts with B2C customers to establish the extent of any potential liability and understand what contractual tools they may be able to rely on to reduce or extinguish liability, such as force majeure clauses.

They should also review contracts with vendors/suppliers to establish whether they might be able bring a claim to recover any losses that they or their B2C customers may have suffered. Suppliers will wish to review their contracts to understand the extent to which they may be shielded by force majeure or other clauses.

If there has been an inability to access personal data on systems, this could constitute a contravention of data protection law, and if any damage has resulted, businesses might be vulnerable to complaints or claims. In that situation, businesses should also consider whether they are required to make notifications to their data regulator.