403 Access Denied authorization error caused by Sticky Bit in ADLS Gen2

It's common that the user of ADLS Gen2 (Azure Data Lake Storage) need to control the permission of different users. One possible way is to use the ACL (Access Control List). For more details about this permission control mode, it can be found in this official document. This blog will mainly focus on one specific setting of the ACL called Sticky Bit and the authorization failure caused by this setting.

There are four parts in this blog:

• What is Sticky Bit in ADLS Gen2?
• How to identify whether the 403 Access Denied is caused by Sticky Bit?
• How to verify the Sticky Bit setting of a folder?
• How to disable/enable Sticky Bit setting

What is Sticky Bit in ADLS Gen2?

According to Access control lists in Azure Data Lake Storage Gen2 | Microsoft Docs , ACL is a POSIX-like access control system. The description of sticky bit in this document is:

The sticky bit is a more advanced feature of a POSIX container. In the context of Data Lake Storage Gen2, it is unlikely that the sticky bit will be needed. In summary, if the sticky bit is enabled on a directory, a child item can only be deleted or renamed by the child item's owning user, the directory's owner, or the Superuser ($superuser).

The sticky bit isn't shown in the Azure portal.

For example, if there is an ADLS Gen2 storage account and a container called container, and there is folder path like folder/child-folder. User A was configured in ACL setting to have X permission on root and folder, and have WX permission on child-folder, then according to Access control lists in Azure Data Lake Storage Gen2 | Microsoft Docs , this user A should have enough permission to create/upload a new file test.txt into path container/folder/child-folder/test.txt. But if Sticky Bit is enabled on child-folder, then user A will get 403 authorization error.

How to identify whether the 403 Access Denied is caused by Sticky Bit?

When a user tried to upload new files into ADLS Gen2 Storage Account folder and he is using ACL as authorization method, either this user doesn't have enough permission including the parent folders, or this user has enough permission but Sticky Bit is enabled and this user isn't the owner of this folder, the 403 Access Denied error will be returned and the error message will be the same.

The only thing we can do from user side is to verify the ACL setting of the folder, and the parent folders and then compare it with the official document. If the permission configured on the user is already enough, then possibly the 403 error was caused by the Sticky Bit. We can follow next part to verify the Sticky Bit setting.

How to verify the Sticky Bit setting of a folder?

From user side, we can use lots of ways to check this setting, such as REST API call, PowerShell command, Azure CLI etc. Among these ways, the most recommended way is the Azure CLI because it doesn't need user to additionally install any software and the command is easy to understand.

• Please login into your Azure Portal with your account. Please make sure that this account has Storage Blob Data Owner role assignment on the ADLS Gen2 storage account.
• Click on the Cloud Shell button from Azure Portal.

Azure CLI in Azure Portal

• Use the following command to get the ACL and Sticky Bit setting: az storage fs access show -p {folder name} -f {container name} --account-name {storage account name} --auth-mode login​For example, az storage fs access show -p folder -f container --account-name account --auth-mode login means to check the ACL and Sticky Bit setting of container/folder. If we want to check the root directory, which is the container level ACL and Sticky Bit setting, we can use az storage fs access show -p / -f container --account-name account --auth-mode login.
• The result will be such as:

Azure CLI get ACL/Sticky Bit setting result

In the response Json body, we need to focus on permissions. It will normally contain 9 or sometimes 10 bits with an additional + signal. For more information about these letters, please refer to Access control lists in Azure Data Lake Storage Gen2 | Microsoft Docs .

In above example, it indicates that all permissions of all users are enabled, and also the Sticky Bit is enabled. For details about how to read this permission notation, please refer to File-system permissions - Wikipedia , Notation of traditional Unix permissions part. Here I’ll only focus on the Sticky Bit related configuration.

The ninth letter has 4 possible values: “-“, “x”, “t”, “T”. If the value of this letter is “t” or “T”, it means that the Sticky Bit is enabled. The “t” is “x“ with Sticky Bit enabled and “T” is “-“ with Sticky Bit enabled.

rwxrwxrwt can be explained into:

• r,w,x permissions are enabled for Owner
• r,w,x permissions are enabled for Owning group
• r,w,x permissions are enabled for Other users and Sticky Bit is enabled

For you to understand more easily, provide another example here.

rwxr-xr-T can be explained to:

• r,w,x permissions are enabled for Owner
• r and x permissions are enabled for Owning group
• Only r permission is enabled for Other users and Sticky Bit is enabled

And here I’ll also give a piece of information about how to translate above notation to short forms. According to Access control lists in Azure Data Lake Storage Gen2 | Microsoft Docs , short form permission will be calculated every group of 3 letters. (r as 4, w as 2 and x as 1) For example: rw-rwx--x will be equals to 4+2+0 4+2+1 0+0+1, 671. Based on this calculation rule, we’ll only need to add the fourth letter at the beginning. If sticky bit is enabled, we set it as 1. If sticky bit is disabled, we set it as 0.

So the above examples will be:

• rwxrwxrwt => 1777

• rwxr-xr-T => 1754

• rw-rwx--x => 0671

How can user disable/enable this setting?

The operation from user side to disable/enable this setting is the same one. User only needs to set the permissions to the expected value.

The account used to modify this setting must have Storage Blob Data Owner role on the target ADLS Gen2 Storage Account.

Since there are so many possible ways to modify this setting, here is a piece of information about whether every SDK supports this setting at this moment:

• REST API. Confirmed working on 2019-12-12+. Path - Update - REST API (Azure Storage Services) | Microsoft Docs 
• PowerShell Az module. Not supported yet. Use PowerShell to manage ACLs in Azure Data Lake Storage Gen2 | Microsoft Docs 
• CLI. Confirmed working on 2.38.0+ Use Azure CLI to manage ACLs in Azure Data Lake Storage Gen2 | Microsoft Docs 
• .Net SDK. Not supported yet. Use .NET to manage ACLs in Azure Data Lake Storage Gen2 | Microsoft Docs
• Java SDK. Confirmed working on 12.11.0+ Use Java to manage ACLs in Azure Data Lake Storage Gen2 | Microsoft Docs 
• Python SDK. Confirmed working on 12.8.0+. Use Python to manage ACLs in Azure Data Lake Storage Gen2 | Microsoft Docs 
• Javascript SDK. Not supported yet. Use JavaScript (Node.js) to manage ACLs in Azure Data Lake Storage Gen2 | Microsoft Docs 
• HDFS CLI: Confirmed working on 3.3.3+ Using the HDFS CLI with Azure Data Lake Storage Gen2 | Microsoft Docs  and Apache Hadoop 3.3.3 – HDFS Permissions Guide 

Here I’ll give an example of Azure CLI. For other ways, please kindly check the SDK or command documents.

• As previous part, please login into your Azure Portal with your account having Storage Blob Data Owner role assignment on the target ADLS Gen2 storage account.
• Click on the Cloud Shell button from Azure Portal.

Azure CLI in Azure Portal

• Use the following command to set the ACL and Sticky Bit setting: az storage fs access set --permissions {permission notation} -p {folder name} -f {container name} --account-name {storage account name} --auth-mode login​For example, az storage fs access set --permissions rwxrwxrwt -p folder -f container --account-name account --auth-mode login means to set the ACL of container/folder to rwxrwxrwx and to enable the Sticky Bit at same time. If we want to modify the setting of the root directory, which is the container level ACL and Sticky Bit setting, we can use az storage fs access set --permissions rwxrwxrwt -p / -f container --account-name account --auth-mode login.
• The {permission notation} in above command can accept both long and short form notations. This means that the following command is also qualified. az storage fs access set --permissions 1776 -p folder -f container --account-name account --auth-mode login
• The successful result of the command will be like following one:

Azure CLI set ACL/Sticky Bit setting result