Share data with Microsoft Purview for Azure storage with private endpoints or VNET restrictions

This is a follow-up post on a recently published article on sharing data in near real-time with Microsoft Purview in-place data sharing for Azure Storage. I highly encourage you to read Share data near real-time with Microsoft Purview in-place data sharing for Azure Storage first before you pore over this article on how to share Azure storage solutions such as Azure Blob Storage and Azure Data Lake Gen2 with VNET traffic restrictions or even with no public endpoints i.e. private endpoints only.

Now, using Microsoft Purview in place data share you can also create shares that can connect to Azure storage PaaS such as blob storage and data lake storage with VNET restrictions and to further up the ante - even with storage accounts with no public endpoints.

This is a critical feature that has been made available only in Microsoft Purview data share and is not part of the standalone Azure data share services.

In this article, I will present a few quick pointers and considerations that you have to know before you get started with Purview data sharing using restricted Azure storage accounts.

Premise - Share data with Microsoft Purview for Azure storage with private endpoints or VNET restrictions.

Solution - In a nutshell, with Purview private endpoints i.e. account, ingestion, and portal (being optional) you can deliver new data shares while connecting to storage accounts with VNET restrictions and/or no public endpoint i.e. private endpoints only. This assumes that the private endpoint for the storage account and purview are created in the same VNET or, this is created in a networking landing zone in case you follow the learnings of the enterprise scale landing zone. 

Here's a quick overview of key steps which will help you set up a data share with restricted storage accounts - 

Step 1 - Create purview private endpoints i.e. account, ingestion, and portal (being optional). This is required for private connectivity between Purview and target storage accounts. This is a required step regardless of how the storage account is configured i.e. with VNET restrictions or with no public endpoint.

You can create the private endpoints in the same VNET i.e. of storage account or in a dedicated VNET which may be part of your networking scaffold in the enterprise scale landing zone. Make sure the different networks i.e. Purview VNET and storage account VNET is peered if you were to follow this topology. 

You can also learn more about this configuration here - Connect to your Microsoft Purview to data sources privately and securely

Here's a quick overview of a typical Purview deployment with private endpoints enabled.

Purview firewall settingsPurview account and portal private endpointsPurview ingestion private endpoint

Step 2 - Depending on the storage account networking configuration, let's break this section into 2 sub parts - 

#1 Deliver data share from a storage account with VNET restrictions

Selected VNET's only

This setup assumes that you have storage accounts with VNET restrictions and in-bound traffic from only selected VNET's are permitted. In this case, you can create Purview endpoints in the same VNET as of storage account or in a separate VNET as long as both are peered. 

You can read more about this configuration here - Configure Azure Storage firewalls and virtual networks

#2 Deliver data share from a storage account with private endpoints only

No public endpoint

This setup assumes that you have storage accounts with no public endpoints and the only way an application can communicate is via private endpoints only. In this case, you will have to create private endpoints for the storage account in a VNET which is accessible by Purview via ingestion private endpoints. So, this can be done in the same VNET as Purview ingestion private endpoints or in a VNET that is accessible and peered to Purview private endpoints in case of enterprise scale landing zone.

If you create a private endpoint for the Data Lake Storage Gen2 storage resource, then you should also create one for the Blob storage resource. That's because operations that target the Data Lake Storage Gen2 endpoint might be redirected to the Blob endpoint. By creating a private endpoint for both resources, you ensure that operations can complete successfully.

You can read more about this configuration here - Use private endpoints - Azure Storage

Step 3 (optional) - I always recommend running nslookup for querying the Domain Name System (DNS) records to obtain the mapping between domain name and IP address. The expectation here is that the name resolution for the storage account and Purview endpoints should return their respective private IP addresses.

If you have followed these steps, with appropriate permissions on the storage account via RBAC's you can then deliver new data shares with your consumer groups with restricted storage accounts which earlier was not possible using Azure data share.

My objective for this post was to introduce you to how you can connect and deliver share with storage accounts that have network restrictions such as VNET restrictions and private endpoints only. Hence, I would not talk about the user journey or customer experience regarding how to create and manage a share. I encourage you to read our post on this subject at How to share data - Microsoft Purview.