Azure Storage Blog articles

Azure Storage Blog articles

https://techcommunity.microsoft.com/t5/azure-storage-blog/bg-p/AzureStorageBlog

Azure Storage Blog articles

Public Preview: Create additional 5000 Azure Storage accounts within your Subscription

Published

Public Preview: Create additional 5000 Azure Storage accounts within your Subscription

Azure Storage is excited to announce public preview of 5000 account limit features, which allows customers to create additional 5000 storage accounts per subscription per region. This is 20x increase from the current limit of 250. Many of our customers wish need several hundreds or thousands of storage accounts to address their storage needs.  This helps customers build scenarios which need hundreds or thousands of storage accounts within a single subscription instead of creating more subscriptions. (Internal name PartitionedDns Feature)

 

This blog will help you to

  • understand the feature in-depth, provides detailed steps on how to use &
  • faq+known issues are at the end.

 

Feature details : 

Azure Storage will have two limits

  • Standard Storage Account Limit : 250
  • Additional Azure DNS Zone Accont Limit: 5000
ManuYareshimi_0-1654269839023.png

With this feature, Azure Storage is introducing two new concepts,

  1. A new storage account limit called 'Azure DNS Zone' which allows customers to create upto 5000 additional storage accounts. To leverage this new 5000 account limit, customers will need to create storage account with dnsEndpointType as 'AzureDNSZone'. 
  2. dnsEndpointType is a new attribute added to storage account and can have values 'standard' or 'AzureDNSZone'. 

When dnsEndpointType is 'AzureDNSZone', the endpoint of the storage account will be created on one of the zones of the new DNS++ architecture of Azure and endpoint url will have a zone in it.

E.g. myAccountname.[dnszone].[service type].storage.azure.net
       where dnszone can be z00 to z99 and [service type] is blob, queue, table etc
 

Here are the difference the two dnsEndpointType - standard and AzureDNSZone 

 

dnsEndpointType = Standard

dnsEndpointType = AzureDNSZone

1

Quota is 250.

Can create upto maximum of 250 accounts

Quota is 5000.

Can create upto maximum of 5000 accounts

2

You could derive the Endpoint url follow the standard scheme of :

myAccountname.[service type].core.windows.net 

 

 

 

 

 

 

E.g:

"primaryEndpoints": {

      "dfs": "https://myaccountname.dfs.core.windows.net /",

      "web": "https://myaccountname.web.core.windows.net /",

      "blob": "https://myaccountname.blob.core.windows.net /",

      "queue": "https://myaccountname.queue.core.windows.net /",

      "table": "https://myaccountname.table.core.windows.net /",

      "file": "https://myaccountname.file.core.windows.net /"

    }

 

You cannot derive the endpoint url and your application or 3rd party tool will need to be updated to get the new endpoints & use it .

At the time of creation of the storage account the endpoint will be assigned a specific dns zone. The endpoint url will be

myAccountname.[dnszone].[service type].storage.azure.net

where dnszone can be z00 to z99.

 

Below is an example of account where it was assigned dns zone  “z18”

"primaryEndpoints": {

      "dfs": "https://myaccountname.z18.dfs.storage.azure.net/",

      "web": "https://myaccountname.z18.web.storage.azure.net/",

      "blob": "https://myaccountname.z18.blob.storage.azure.net/",

      "queue": "https://myaccountname.z18.queue.storage.azure.net/",

      "table": "https://myaccountname.z18.table.storage.azure.net/",

      "file": "https://myaccountname.z18.file.storage.azure.net/"

    } 

 

Your application or 3rd party tool will need to be updated to get and use the new endpoints. To get your endpoints either,

  • Call GetProperties api & results has the endpoint url.
  • Or goto Azure Portal and look at endpoints on the left in your storage account.
    • Save and use this endpoint to connect to your storage account

 

Exclusion : 

  1. Azure Storage Classic accounts are not supported for Azure DNS Zone account limit i.e 5000 accounts per region per subscription.

A. Regions where this public preview is available :

Only available in the all public regions and not sovereign or government regions. If you try in other regions, you will get an error message (Error "code": "FeatureNotSupportedForAccount","message": "Partitioned Dns is not supported for the account.”)

 

B. How to leverage this feature :

0. Pre-requisite - Register for the preview via registering PartitionedDnsPublicPreview feature for your subscription. Here are the instructions Set up preview features in Azure subscription.

Specify PartitionedDnsPublicPreview as the feature name and Microsoft.Storage as the provider namespace

1.  Step 1 – Create Storage Account with AzureDnsZone

  1. Arm Template, Powershell / CLI: Add dnsEndpointType = AzureDnsZone (see detailed steps in F. Steps for Common Tasks/Jobs )
  2. Azure Portal: In Create a Storage account workflow, under Networking tab, for Endpoint Type, choose azurednszone.

 

ManuYareshimi_2-1654269839026.jpeg

 

 

2. Step 2 - Use the new endpoints in your connection strings in your application or Storage client SDKs / Tools

Here is sample connection string format

DefaultEndpointsProtocol=https;AccountName=<storageaccount>;AccountKey=***;BlobEndpoint=https://<storageaccount>.<zone>.blob.storage.azure.net ;FileEndpoint=https://<storageaccount>.<zone>.file.storage.azure.net;QueueEndpoint=https://<storageaccount>.<zone>.queue.storage.azure.net; TableEndpoint=https://<storageaccount>.<zone>.table.storage.azure.net;

 

You can get your endpoints / connection string via Azure Portal , powershell, cli or rest api.

  1. You could use api GetProperties from storage resource provider API. Storage Accounts - Get Properties - REST API (Azure Storage Resource Provider) | Microsoft Docs
  2. You could use Azure Portal- Goto your storage account. On left goto ‘Access Keys’ blade, you will see the complete connection string and if you goto endpoints you will see the endpoints.

ManuYareshimi_4-1654269839027.png

 

C. SDK/Tools supporting 5000 Account Limit (partitioneddns) feature

Azure Portal

https://portal.azure.com/

Storage Management API

2021-09-01 or later.

.Net

Azure.Storage.Common 12.8.0 or later

Azure.Storage.Files.DataLake 12.7.0 or later

Java

azure-storage-blob v12.10.0 or later

azure-storage-common v12.10.0 or later

azure-storage-file-datalake v12.4.0  or later

JS

No support yet 

Estimated date to add support is middle of June 2022.

Python

- azure-storage-blob v12.7.0 (including 12.7.0b1)

- azure-storage-file-datalake v12.2.0

Powershell Install Az.Storage 4.4.2 and later
CLI Azure CLI 2.37.0 or above
AzCopy v10.9.0 or later
Storage Explorer 1.19.0 or later
Management plane SDK 24.0.0 or later 

Reference links for SDK / Tools

.Net - https://www.nuget.org/packages/Azure.Storage.Blobs/

Java - https://mvnrepository.com/artifact/com.azure/azure-storage-blob 

Python - https://pypi.org/project/azure-storage-blob/

AzCopy - https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10

Storage Explorer - https://azure.microsoft.com/en-us/features/storage-explorer/

Azure Powershell -  https://github.com/Azure/azure-powershell/

Azure CLI - https://github.com/Azure/azure-cli

 

D. Your applications or tools will need an update 

Your applications and tools will need to use the supported sdks and be updated to get the new endpoint by calling GetProperties api & then use the endpoint to connect to the storage account.

New endpoint is of this format  myAccountname.[dnszone].[service type].storage.azure.net
       where dnszone can be z00 to z99 and [service type] is blob, queue, table etc

 

E. For ISV(Independent Software Vendor) or Services vendor

This feature may affect you. Please review our recommendation for you appropriate scenario.

Scenario

What is the impact?

Recommendation

1. Your service/product connects to your Customer created Azure Storage account

a. Collects only storage account name and No endpoint url, and figures endpoint url

Failure / Error to connect to Storage Account with new of endpointType =AzureDNSZone, which gets new endpoint schema assigned while creation of storage account.

Add ability for Customers to specify endpoint url and use it if customer provides endpoint url.

b. Collects endpoint url from Customer/user

No impact

2. Your service/product creates Storage Account for your / your customers’ needs and you don’t intend to use the 5000 Account Limit feature.

No Impact

N/A

3. Your service/product wishes to leverage this feature to create upto 5000 accounts with your subscription

If you wish to participate in public preview, please follow B. How to leverage this feature  section

  • Update your service/product to get the new endpoint GetProperties api and save it on your end for the next time you need it.
  • Use this endpoint to connect to your storage account

 

 

F. Steps for Common Tasks/Jobs 

1. Powershell:

Releases –

a. Install the correct version (with admin privileges) :

i. Install the latest PowershellGet

Install-Module PowerShellGet –Repository PSGallery –Force

ii. Close and reopen the Powershell console

iii. Install Az.Storage preview module

Install-Module Az.Storage -Repository PsGallery -RequiredVersion 4.4.2-preview -AllowClobber -AllowPrerelease -Force

d. Create an account with dnsEndpointType == AzureDnsZone

New-AzStorageAccount -ResourceGroupName $rgname -Name $accountName -SkuName Standard_LRS -Location eastus2euap -Kind StorageV2 -DnsEndpointType AzureDnsZone

e. Get the connection string

i. Create storage Context

# only have dataplane access

$ctx = New-AzStorageContext -BlobEndpoint "https://accountname.z10.blob.core.windows.net/" -TableEndpoint "https://accountname.z10.table.core.windows.net/" -QueueEndpoint "https://accountname.z10.queue.core.windows.net/" -FileEndpoint "https://accountname.z10.file.core.windows.net/" -StorageAccountName “accountname” -StorageAccountKey $key

# if have management plane access:

$ctx = (Get-AzStorageAccount -ResourceGroupName $resourceGroupNam -Name $accountName).Context 

ii.Get Connection string

$connectionString = $ctx.ConnectionString

f. Upload a file(myfile.txt) – no change

Set-AzStorageBlobContent -File $localSrcFile -Container $containerName -Blob $blobName -Force  -Context $ctx

g. Download a file(myfile.txt) – no change

Get-AzStorageBlobContent -Container $containerName -Blob $blobName  -Destination $localDestFile -Force -Context $ctx

h. Delete the previous created account – no change

Remove-AzStorageAccount -ResourceGroupName $rgname -Name $accountName

2. With CLI: 

Releases –

  • CLI extension storage-preview 0.8.2 released to support `--dns-endpoint-type` for creating storage account
  • CLI 2.36.0 will be released to support specifying endpoint for each service

a. Install the correct version

az extension add -n storage-preview

b. Create an account with dnsEndpointType == AzureDnsZone

az storage account create -n $accountName -g $rgName -l eastus2euap --dns-endpoint-type AzureDnsZone

c. Get the service endpoint

az storage account show –n $accountName --query “primaryEndpoints”

az storage account show -g $resourcegroupName -n $accountName --query “primaryEndpoints”

d. Get Connection String

i. Use show-connection-string

az storage account show-connection-string -g $resourcegroupName -n $accountName

ii. Alternate approach is to add the endpoint

az storage account show-connection-string –blob-endpoint “https://myaccountname.z18.blob.storage.azure.net/” -g $resourcegroupName -n $accountName

c. Upload a file

$blobEndpoint = az storage account show –n $accountName –query “primaryEndpoints.blob”

az storage blob upload --file $localFile --name $blobName --container $containerName --blob-endpoint $blobEndpoint

d. Download a file

$blobEndpoint = az storage account show –n $accountName –query “primaryEndpoints.blob”

az storage blob download --file $localFile --name $blobName --container $containerName --blob-endpoint $blobEndpoint

e. Delete the previous created account

az storage account delete –n $accountName –g $rgName

 

3. With .net management plane sdk:

a. Install the correct version

Install-Package Microsoft.Azure.Management.Storage -Version 24.0.0

b. Create an account with dnsEndpointType == AzureDnsZone

See test case: https://github.com/Azure/azure-sdk-for-net/blob/30cc1b933378780cccf22c00fcb68ccd7c09e6e6/sdk/storage/Microsoft.Azure.Management.Storage/tests/Tests/StorageAccountTests.cs#L3015

c. Get the connection string

N/A (Management plane SDK doesn’t support create dataplane connection string directly)

d. Upload a file(myfile.txt)

N/A (Management plan SDK doesn’t support dataplane operation)

e. Download a file(myfile.txt)

N/A (Management plan SDK doesn’t support dataplane operation)

f. Delete the previous created account

No change, see sample in https://github.com/Azure/azure-sdk-for-net/blob/30cc1b933378780cccf22c00fcb68ccd7c09e6e6/sdk/storage/Microsoft.Azure.Management.Storage/tests/Tests/StorageAccountTests.cs#L201

 

G. Frequently asked questions

  1. What if my subscription is registered for private preview and public preview feature?
    • Please email us [email protected], we need to unregister your subscription from the private preview of this feature.
    • Or you could
  2. I don’t have a need to add more than 250 storage account, should I participate in public preview.
    • If you foresee that in future you may need to add more than 250 storage account then, please participate.
  3. Will we be impacted with the launch of this feature?
    • This feature is available only for subscriptions registering for the public preview feature(PartitionedDnsPublicPreview). If your subscription is not registered then, there is no impact.
    • Once your subscription is registered you can leverage additional 5000 Account limit to create storage accounts with dnsEndpointType as AzureDNSZone.
  4. Will enabling this feature on my subscription, impact my existing storage accounts ?
    • No, this will have no impact to your existing storage accounts. 
  5. Will enabling this feature on my subscription, impact my current application or process interacting with Azure Storage?
    • If you create new storage accounts with dnsEndpointType = AzureDnsZone then, the endpoint url of will be of new format
      • myAccountname.[dnszone].[service type].storage.azure.net
      • where dnszone can be z00 to z99
    • Your application or process will need to get endpoint url from GetProperties api and use it to connect to your storage account. Your application can cache or save the endpoint url as it does not change for a storage account
  6. Can I change the dnsEndpointType after creation of storage account?
    • No, this wont be possible once dnsEndpointType is either standard or AzureDNSZone, it cannot be changed. 
  7. Will the endpoint url of the storage account change?
    • No, once the endpoint url is assigned while creation of storage account it will not change for the life of the storage account.
  8. How does this feature affect secondary endpoints?
    • Secondary endpoint will be created in the same way. For dnsEndpointType is AzureDNSZone, the secondary endpoint will be as follows,
      • myAccountname-secondary.[dnszone].[service type].storage.azure.net
  9. Will Firewall rules need to change ?
    • Customers will need to add *.storage.azure.net to the firewall exception rules
  10. How to asks questions ?
  11. How to give feedback or report issues ?
    • We would appreciate your feedback and reporting issues. Please email us at [email protected] 

​H. Known Issues:

The following services are not compatible with this feature. We recommend not to use these services / features with 5000 Account limit feature in preview period. As these teams add compatibility, we will update dates on this blog. If you have any question, please email us at [email protected] 

  1. Azure Blob Filesystem driver (ABFS)
  2. ADX - Azure Data Explorer (Kusto)
  3. Azure Data Factory
  4. Azure Databricks / Databricks
  5. Azure Devops Terraform Plugin
  6. Azure Event Grid
  7. Azure Event hub
  8. Azure File Sync
  9. Azure Files AD
  10. Azure Function
  11. Azure Service Bus
  12. Azure Monitor - Diagnostics
  13. Azure Storage - Blob Inventory
  14. Azure Storage - Object Replication Rules
  15. Power Apps - Platform Connectors

Thank you for taking time to read this blog. We hope this feature helps you and your business. 

Continue to website...

More from Azure Storage Blog articles