How to integrate Azure Batch pool with Azure Storage Account configured with firewall

The Azure Storage Account is a necessary dependent component of Azure Batch account to store resource files, application packages and output files. In many cases, users want to use Azure Storage Account with firewall to enhance the security of the Azure Storage Account. However, this may cause some issues when executing Azure Batch jobs. This Blog will introduce two workarounds of configuring Azure Storage Accounts with firewall and Azure Batch account.

Background：

• When executing Azure Batch Jobs, users may encounter errors related to the associated Azure Storage Account.

       Underlying mechanism:

• When creating Azure Batch pool, new VMs (Batch nodes) will be provisioned. If the user does not assign a static public IP to the Batch pool, a random public IP will be assigned. Whenever users resize the number of node to 0 and resize out again, the public IP address of these new Batch nodes will change. Therefore, if the associated Azure Storage Account has firewall configured, it’s hard for users to manage the allow list of the firewall.

• Additionally, if the Azure Storage Account and the Azure Batch pool are in the same region, the outbound traffic from Azure Batch node will go via Azure backbone internet (via private IPs) instead of the public IP. Meanwhile, Azure Storage Firewall is not allowed to add private IP in the allow list, which will also cause the traffic to Storage Account to be denied.

Therefore, based on different settings, we need to manage the Azure Batch pool/Storage Account configurations carefully to make it work.

Below are the two scenarios that Azure Storage Account is configured with firewall.

Scenario 1:

• The Azure Batch pool and Storage Account are in the same region, the Batch pool has VNET.

• When an Azure Batch pool has configured Virtual Network, users can check the configuration from Azure Portal -> Batch Account -> Pool -> Properties. The subnet info is displayed as shown below:

• In the associated Azure Storage firewall settings, add this subnet in the allow list:

• If this subnet never enables the service endpoint, when you choose this subnet, a notification will be displayed as shown below:

• Therefore, it’s better to check the batch pool’s subnet whether the Service Endpoint for Storage Account is enabled:

• After the above configurations, the Batch nodes in the pool can access the Storage Account successfully.

Scenario 2:

• The Azure Batch pool and Azure Storage Account are in different regions.

Steps:

1. Create a Batch pool in Virtual network with a static Public IP address referring to the document Create a Batch pool with specified public IP addresses - Azure Batch | Microsoft Docs
2. Write down the Public IP address because it will be assigned to the Azure Batch Pool public Loadbalancer’s IP and be used for the communication with the Azure Storage account.
3. After configuring the above steps, when you check your pool’s properties, it will be like the screenshot below:

• As the Azure Batch pool and Azure Storage Account are in different regions, the outbound traffic will go through the public internet via the above user assigned public IP.
• Therefore, users only need to add the public IP assigned to the batch pool in the Storage firewall allow list.

• Then users can test to run the jobs with the new-created batch pool.

Important:

• If the Batch account and the Storage Account are in the same region, no matter if the Batch pool has a static public IP address or not, the outbound traffic will always go via the Azure backbone network.
• According to the document Deploy application packages to compute nodes - Azure Batch | Microsoft Docs , if the users need to upload application package, all above solutions will not work. The storage account must not configure any firewall.