Recovery options for Azure Virtual Machines (VM) affected by CrowdStrike Falcon agent

We are aware of an issue that started on 19 July 2024 at 04:09UTC, which resulted in customers experiencing unresponsiveness and startup failures on Windows machines using the CrowdStrike Falcon agent, affecting both on-premises and various cloud platforms.  

CrowdStrike has released a public statement on Windows Sensor Update - crowdstrike.com addressing the matter, and it includes recommended steps for a workaround. For environments specific to Azure, further instructions are provided below: 
 
Recovery Option:  

We have noticed that some Azure VMs are successfully updating via the CrowdStrike Falcon agent after multiple manual Virtual Machine restarts. Customers can attempt to do so as follows: 

• Using the Azure Portal - attempting 'Restart' on affected VMs 

• Using the Azure CLI or Azure Shell (https://shell.azure.com) 

https://learn.microsoft.com/en-us/cli/azure/vm?view=azure-cli-latest#az-vm-restart 

Please note in some cases multiple reboots may be required. Usually, the reboot of instance allows the CrowdStrike Falcon agent to update successfully. If a reboot doesn't resolve the issue, then leverage additional recovery options to resolve the issue. 

Additional options for recovery: 

Option 1: 

We recommend customers that can restore from a backup, preferably from before 19 July 2024 at 04:09UTC, when this faulty update started rolling out. 

• Customers leveraging Azure Backup can follow the following instructions: How to restore Azure VM data in Azure portal 

Option 2:  

Customers can attempt to remove the C-00000291*.sys file on the disk directly and potentially not need to perform detach and reattach disc. 

Open Azure AZ CLI and run the following steps :

1.Create rescue VM with 

// Creates a rescue VM, same size as the original VM in the same region. Asks for Username and password.  

// Makes a copy of the OS Disk of the problem VM 

// Attaches the OS Disk as Data disk to the Rescue VM 

//az vm repair create -g {your-resource-group} -n {vm-name} --verbose 

"az vm repair create -g RGNAME -n VMNAME -- verbose" 

 **NOTE: For encrypted VM run the following command:** 

"az vm repair create -g RGNAME -n BROKENVMNAME --unlock-encrypted-vm --verbose" 

2.Then run: 

// Runs the mitigation script on the Rescue VM which fixes the problem (on the os-disk copy attached as a data disk) 

//az vm repair run -g {your-resource-group} -n {vm-name} --run-id win-crowdstrike-fix-bootloop -verbose 

"az vm repair run -g RGNAME -n BROKENVMNAME -- run-id win-crowdstrike-fix-bootloop -- run-on-repair -- verbose" 

3.Final step is to run: 

// Removes the Fixed OS-Disk Copy from the rescue VM 

// Stops the problem VM but it is not deallocated 

// Attaches the fixed OS-Disk to the original VM 

// Starts the original VM 

// Gives prompts to delete the repair vm 

//az vm repair restore -g {your-resource-group} -n {vmname} --verbose 

"az vm repair restore -g RGNAME -n BROKENVMNAME" --verbose 

Note: These steps would work for both managed and unmanaged disks. In case, if you run into capacity issues, please retry after some time. 

Option 3:  

Customers can attempt repairs on the OS disk by following these instructions: 

Troubleshoot a Windows VM by attaching the OS disk to a repair VM through the Azure portal 

Once the disk is attached, customers can attempt to delete the following file: 

Windows/System32/Drivers/CrowdStrike/C-00000291*.sys 

The disk can then be attached and re-attached to the original VM. 

Note: We can confirm the affected update has been pulled by CrowdStrike. Customers that are continuing to experience issues should reach out to CrowdStrike for additional assistance. 

Additionally, we're continuing to investigate additional mitigation options for customers and will share more information as it becomes known.