Loading...

Mount ADLS Gen2 or Blob Storage in Azure Databricks

Mount ADLS Gen2 or Blob Storage in Azure Databricks

Scenario:
Azure Databricks offers many of the same features as the open-source Databricks platform, such as a web-based workspace for managing Spark clusters, notebooks, and data pipelines, along with Spark-based analytics and machine learning tools. It is fully integrated with Azure cloud services, providing native access to Azure Blob Storage, Azure Data Lake Storage, Azure SQL Database, and other Azure services. This blog shows example of mounting Azure Blob Storage or Azure Data Lake Storage in the Databricks File System (DBFS), with two authentication methods for mount: Access Key and SAS token.

 

Objective:

To become acquainted with Databricks storage mount with ABFS/WASB driver and various authentication methods.

Pre-requisites:

For this example, you would need:

  1. An Azure Databricks Service.
  2. A Databricks Cluster (compute).
  3. A Databricks Notebook.
  4. An Azure Data Lake Storage or Blob Storage.

 

Steps to mount storage container on Databricks File System (DBFS):

  1. Create storage container and blobs.
  2. Mount with dbutils.fs.mount().
  3. Verify mount point with dbutils.fs.mounts().
  4. List the contents with dbutils.fs.ls().
  5. Unmount with dbutils.fs.unmount().

 

[STEP 1]: Create storage container and blobs

Below is the storage structure used in this example. I have created a container “aaa”, a virtual folder “bbb”, in which has 5 PNG files. The storage “charlesdatabricksadlsno” is a blob storage with no hierarchical namespace.

 

databricks-test1.png

databricks-test.png

 

 

[STEP 2]: Mount with dbutils.fs.mount()

We can use below code snippet to mount container "aaa" with Azure Databricks.

 

 

 

storageAccountName = "charlesdatabricksadlsno" storageAccountAccessKey = <access-key> sasToken = <sas-token> blobContainerName = "aaa" mountPoint = "/mnt/data/" if not any(mount.mountPoint == mountPoint for mount in dbutils.fs.mounts()): try: dbutils.fs.mount( source = "wasbs://{}@{}.blob.core.windows.net".format(blobContainerName, storageAccountName), mount_point = mountPoint, #extra_configs = {'fs.azure.account.key.' + storageAccountName + '.blob.core.windows.net': storageAccountAccessKey} extra_configs = {'fs.azure.sas.' + blobContainerName + '.' + storageAccountName + '.blob.core.windows.net': sasToken} ) print("mount succeeded!") except Exception as e: print("mount exception", e)

 

 

 

Some keypoints to note:

  1. I provide two authentication methods for mount: Access Key and SAS token. You may use either (by choosing the 1st or 2nd line that starts with "extra_configs". Instructions for getting the Access Key and SAS token are in the next section.
  2. This mount example does not re-mount an existing mount point. To re-mount, you have to unmount (mentioned in later section) and then mount again.

To get the Access Key, you would go to Azure portal/Access Keys and copy either key1 or key2.

 

charleswang_4-1682178395217.png

To get a SAS token, you can generate in two ways:

  • Generate an account level SAS with all Allowed Resource Types enabled.

charleswang_0-1685953446828.png

 

  • Generate a container level SAS with read and list permissions. For this example, I generate a SAS for container “aaa” which I would later mount on the Databricks cluster.

charleswang_0-1685953796024.png

 

 [STEP 3]: Verify mount point (/mnt/data) with dbutils.fs.mounts()

 

 

 

dbutils.fs.mounts()

 

 

charleswang_5-1682179093078.png

 

[STEP 4]: List the contents with dbutils.fs.ls()

 

 

 

dbutils.fs.ls("/mnt/data/bbb")

 

 

charleswang_0-1682180048076.png

 

[STEP 5]: Unmount with dbutils.fs.unmount()

 

 

 

dbutils.fs.unmount('/mnt/data')

 

 

charleswang_2-1682180294442.png

 

Others:

  • To use ADLS Gen2 storage as mount source, just replace the storage account name, Access Key, and SAS token in the mount step. You may reuse the BLOB endpoint (blob.core.windows.net).
  • If you want to take advantage of the hierarchical namespace feature of ADLS Gen2, such as ACL on the files and folders, you can switch to use ABFS, which stands for Azure Blob File System, and the DFS endpoint (dfs.core.windows.net), from the previous WASBS (Windows Azure Storage Blob) used with BLOB endpoint. The mount source would become: 
abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/
  • from
wasbs://<container-name>@<storage-account-name>.blob.core.windows.net/
  • To prevent mount point authentication error in case an Access Key or SAS token is rotated, you can modify the mount condition such that if there is an existing mount point, it will first unmount before mounting.

 

 

 

storageAccountName = "charlesdatabricksadlsno" storageAccountAccessKey = <access-key> sasToken = <sas-token> blobContainerName = "aaa" mountPoint = "/mnt/data/" if any(mount.mountPoint == mountPoint for mount in dbutils.fs.mounts()): dbutils.fs.unmount(mountPoint) try: dbutils.fs.mount( source = "wasbs://{}@{}.blob.core.windows.net".format(blobContainerName, storageAccountName), mount_point = mountPoint, #extra_configs = {'fs.azure.account.key.' + storageAccountName + '.blob.core.windows.net': storageAccountAccessKey} extra_configs = {'fs.azure.sas.' + blobContainerName + '.' + storageAccountName + '.blob.core.windows.net': sasToken} ) print("mount succeeded!") except Exception as e: print("mount exception", e)

 

 

 

 

References:

Published on:

Learn more
Azure PaaS Blog articles
Azure PaaS Blog articles

Azure PaaS Blog articles

Share post:

Related posts

Azure SDK Release (February 2025)

Azure SDK releases every month. In this post, you find this month's highlights and release notes. The post Azure SDK Release (February 2025) a...

1 day ago

Announcing the Public Preview of the Azure Cosmos DB SDK for Rust!

We’re excited to announce the public preview of the Azure Cosmos DB SDK for Rust, a native Rust SDK that enables developers to interact with A...

1 day ago

Migrating data from DynamoDB to Azure Cosmos DB

Co-authors: RK Iyer, Mangal Dutta Migrating stateful systems, such as databases, is a complex process. A frequent requirement for customers is...

2 days ago

New Azure Cosmos DB Design Patterns Video Series – Learn, Implement, and Contribute!

We’re excited to introduce the Azure Cosmos DB Design Patterns video series—your go-to resource for learning how to design scalable, efficient...

3 days ago

Azure dev quiz

3 days ago
Stay up to date with latest Microsoft Dynamics 365 and Power Platform news!
* Yes, I agree to the privacy policy