Loading...

Restoring Soft-Deleted Blobs with multithreading in Azure Storage Using C#

Restoring Soft-Deleted Blobs with multithreading in Azure Storage Using C#

Blob soft delete is an essential feature that safeguards your data against accidental deletions or overwrites. By retaining deleted data for a specified period, it ensures data integrity and availability, even in the event of human error. However, restoring data in the soft delete state can be more labor-intensive, as the undelete API must be called for each individual deleted blob. Currently, there is no option to bulk undelete all blobs.

 

In this blog, we provide a sample C# code that will help you restore soft-deleted data efficiently. The code leverages multiple threads to expedite the restoration process, making it particularly effective if you have a large number of blobs to restore. Additionally, this program can be configured to undelete blobs within a specific container or directory, rather than scanning the entire storage account.

 

To run this program, follow these steps:

  • Install .NET SDK: Ensure you have the .NET SDK installed on your machine.
  • Connect to Azure Account:

 

Connect-AzAccount

 

  • Add NuGet Source:

 

dotnet nuget add source https://api.nuget.org/v3/index.json -n nuget.org

 

  • Create a New Console Application:

 

dotnet new console --force

 

  • Add the following code to Program.cs.

 

using Azure.Core; using Azure.Identity; using Azure.Storage.Files.DataLake; using Azure.Storage.Files.DataLake.Models; var StorageAccountName = "xxxx"; var ContainerName = "xxxx"; var DirectoryPath = ""; var Concurrency = 500; var BatchSize = 500; static DataLakeServiceClient GetDatalakeClient(string accountName) { DataLakeClientOptions clientOptions = new DataLakeClientOptions() { Retry = { Delay = TimeSpan.FromMilliseconds(500), MaxRetries = 5, Mode = RetryMode.Fixed, MaxDelay = TimeSpan.FromSeconds(5), NetworkTimeout = TimeSpan.FromSeconds(30) }, }; // only works for prod. DataLakeServiceClient client = new( new Uri($"https://{accountName}.blob.core.windows.net"), new DefaultAzureCredential(), clientOptions); return client; } Console.WriteLine("Starting the program"); var client = GetDatalakeClient(StorageAccountName); var throttler = new SemaphoreSlim(initialCount: Concurrency); List<Task> tasks = new List<Task>(); List<string> containerNames = new List<string>(); if (string.IsNullOrEmpty(ContainerName)) { var containers = client.GetFileSystems(); foreach (var container in containers) { containerNames.Add(container.Name); } } else { containerNames.Add(ContainerName); } var totalSuccessCount = 0; var totalFailedCount = 0; foreach (var container in containerNames) { Console.WriteLine($"Recoverying for container {container}"); var fileSystem = client.GetFileSystemClient(container); var deletedItems = fileSystem.GetDeletedPaths(pathPrefix: DirectoryPath); var count = 0; var totalSuccessCountForContainer = 0; var totalFailedCountForContainer = 0; foreach (PathDeletedItem item in deletedItems) { await throttler.WaitAsync(); count++; try { var task = (fileSystem.UndeletePathAsync(item.Path, item.DeletionId)); var continuedTask = task.ContinueWith(t => { throttler.Release(); if (t.IsFaulted) { Interlocked.Increment(ref totalFailedCount); Interlocked.Increment(ref totalFailedCountForContainer); Console.WriteLine($"Failed count for container {totalFailedCountForContainer}, total failed count {totalFailedCount}, path {DirectoryPath + item.Path} due to {t.Exception.Message}"); } else { Interlocked.Increment(ref totalSuccessCount); Interlocked.Increment(ref totalSuccessCountForContainer); Console.WriteLine($"Success count for container {totalSuccessCountForContainer}, total success count {totalSuccessCount}"); } }); tasks.Add(continuedTask); } catch (Exception ex) { Console.WriteLine("Failed to create task: " + ex.ToString()); } finally { if (count == Math.Max(Concurrency, BatchSize)) { count = 0; await Task.WhenAll(tasks); tasks.Clear(); } } } await Task.WhenAll(tasks); Console.WriteLine($"Recover finished for container {container}"); }

 

 

Replace xxxx with your specific storage account and container name. If you need to restore a particular directory, provide the directory name; otherwise, leave it empty to scan the entire container. The code is configured to run with 500 threads by default, but you can adjust this number according to your needs.

 

  • Add Required Packages:

 

dotnet add package Azure.Identity dotnet add package Azure.Storage.Blobs

 

  • Build the Project:

 

dotnet build --configuration Release

 

 

  • Run the Program:

 

dotnet <path_to_dll>

 

 

Once the application is running, you can monitor the console window to track its progress and identify any potential issues or failures.

Published on:

Learn more
Azure PaaS Blog articles
Azure PaaS Blog articles

Azure PaaS Blog articles

Share post:

Related posts

Azure Storage - TLS 1.0 and 1.1 retirement

Overview TLS 1.0 and 1.1 retirement on Azure Storage was previously announced for Nov 1st, 2024, and it was postponed recently to 1 year later...

1 year ago

Efficient Management of Append and Page Blobs Using Azure Storage Actions

  Overview In Azure Storage, Blob Lifecycle Management (BLM) allows you to automate the management of your data based on rules defined by...

1 year ago

[Azure AI Search] Internal Server Error when creating CMK encrypted objects

Scenario Customers follow the Microsoft doc to create CMK encrypted objects (data source, index etc.), but get the 500 Internal Serv...

1 year ago

Optimizing Azure Table Storage: Automated Data Cleanup using a PowerShell script with Azure Automate

Scenario This blog’s aim is to manage Table Storage data efficiently. Imagine you have a large Azure Table Storage that accumulates logs from ...

1 year ago

Optimizing Azure Table Storage: Automated Data Clean-up using a PowerShell script with Azure Automat

Scenario This blog’s aim is to manage Table Storage data efficiently. Imagine you have a large Azure Table Storage that accumulates logs from ...

1 year ago

Performing simple Azure Table Storage REST API operations using curl command.

The blog provides guidance to perform simple Table Storage REST API operations such as Create table, Delete Table, Insert entity, Delete entit...

1 year ago

Bulk delete all the old jobs from the batch account

Deleting a Job also deletes all Tasks that are part of that Job, and all Job statistics. This also overrides the retention period for Task dat...

1 year ago

Utilizing Azure Storage and Runbooks for scheduled automated backups of Azure SQL Databases

In this article, we are going to provide detailed steps to create a scheduled Azure SQL Database backup to storage account using automation. T...

1 year ago

[Azure Service Bus] JMS messages getting dead-lettered

The article discusses a problem where numerous messages end up in the dead letter queue (DLQ) when the JMS service bus consumer connects to th...

1 year ago

Subscribe to Azure Storage Blob Lifecycle Policy Events

The LifecyclePolicyCompleted event is generated when the actions defined by a lifecycle management policy are performed. Refer - Opt...

2 years ago
Stay up to date with latest Microsoft Dynamics 365 and Power Platform news!
* Yes, I agree to the privacy policy