NLP Inferencing on Confidential Azure Container Instance

NLP Inferencing on Confidential Azure Container Instance

Thanks to the advancements in the area of natural language processing using machine learning techniques & highly successful new-age LLMs from OpenAI, Google & Microsoft - NLP based modeling & inferencing are one of the top areas of customer interest. These are being widely used in almost all Microsoft products & there is also a huge demand from our customers to utilize these techniques in their business apps. Similarly, there is a demand for privacy preserving infrastructure to run such apps. 

In this blog I am going to show how to run basic NLP text summarization using T5 & BART in a python streamlit.io based app & run it as a container on Confidential ACI container group. Before we go there, here are some details about Azure Confidential Computing for the uninitiated. Azure already encrypts data at rest and in transit, and confidential computing helps protect data in use, including the cryptographic keys.  Azure confidential computing helps customers prevent unauthorized access to data in use, including from the cloud operator, by processing data in a hardware-based and attested Trusted Execution Environment (TEE). 


In this article we will create a python streamlit.io app and deploy it to Confidential ACI. The use case is loosely based on a 2 party multi party computing scenario where party I provides text data in encrypted format because they cannot allow party II to take a look at the text. However, their contract with party II allows party II to summarize the encrypted text and consume the summary. This entire process runs in the Azure tenant that belongs to party I but due to Confidential ACI, party I is at no point able to take a look at the original text data from party II.


Note: In the demo steps below we are not doing any steps that are marked optional and shown by dotted lines.
App workflow is as follows:
a) Party I creates and app that can do Secure Key Release based on policies. There can be multiple keys based on the application design - in this example, we release one key to decrypt the encrypted file system used to store data from party II. (In real life there can be an application/model key as well as several keys used for encrypting/decrypting the Confidential data).
b) Party I deploys the application to confidential ACI that is capable to doing the secure key release, decrypt the file system containing Party II's data and mount the file system to the container. In this entire process, any guest user or the cloud provider cannot access the application or the mounted file system in any way since it is always encrypted - even when in-memory (running on AMD SEV-SNP based confidential VMs).
c) Now, the application code does a summarization of the Text data from Party II and is able to display the summary on the application page as shown below. The code is audited and is measured as a part of the launch of the container (all layers of the container including the code are measured and the measurement is used for Secure Key Release; so no tampering is possible. In real life, when the keys for data decryption are located in the Party II's Azure tenant, the key release policies cannot be tampered with/altered by Party I or Azure operators else it will prevent the container from launching successfully).
d) The encrypted data can be alternatively also be stored in Party I's tenant and can also be loaded cross-tenant at runtime as in this example. Post processing the data will have to be deleted on-demand and that can also be measured as a part of the encrypted file system. The encrypted file system may be located one of the storage accounts in either of the tenants and the same rules apply.

To start with, we need to create and deploy this app on ACI. Please follow the instructions given below.



Before starting please create the following pre-requisites in your portal or using CLI:


a) Create your app and push it to Azure Container Registry (please see the appendix section to see how I created my app and ACR entry)

b) Create a Managed HSM instance Quickstart - Provision and activate an Azure Managed HSM | Microsoft Learn

c) Create an Azure Storage Account & create a container inside it Create a storage account - Azure Storage | Microsoft Learn

d) Create a SAS token for the container you created in Step c. https://learn.microsoft.com/en-us/azure/cognitive-services/translator/document-translation/how-to-guides/create-sas-tokens 

e) Create a user-assigned managed identity Manage user-assigned managed identities - Microsoft Entra | Microsoft Learn

f) Give the following permissions to your AAD user and the newly created user assigned managed identity on the MHSM 

  • Managed HSM Crypto User
  • Managed HSM Crypto Officer

Demo Steps (windows or linux environment will work; I am using Windows 11 here with WSL2 for Ubuntu Shell):

a) Open WSL2 on Windows and run the following installs:



sudo apt update sudo apt install golang-go sudo apt install gccgo-go



Now, switch to /mnt






b) Clone our main github repo as follows: (you may need to install git if not installed)



git clone https://github.com/microsoft/confidential-sidecar-containers.git



c) Switch to the folder containing the next shell script as follows:



cd confidential-sidecar-containers/examples/encfs/generatefs/



d) Install python to run this script if not already done



sudo apt install python3 sudo apt install python-is-python3 python -V sudo apt install cryptsetup



e) Run the shell script  ./generatefs.sh






This should generate the image (.img file & .bin file). We will use these in later steps.



ls -l od -x encfs.img



This step creates an encrypted file system that will be uploaded in the next step to the azure storage account as a block blob.

f) Now install the azure cli if not already installed



curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash



Upload the encfs.img to the blob storage. (Before this steps please create an Azure Storage Account & create a container inside it, then create a SAS token)



cd .. az storage blob upload --file generatefs/encfs.img --container-name <your container name> --name <specify a name you would like for the block blob of the image we generated in the previous step> --account-name <your storage account name> --sas-token '<SAS token starting with sp within single quotations>'



g) Post the step above your encfs should show in the portal.


h) Now run the following to figure out the key that was used to encrypt the file system:



python hexstring.py keyfile.bin



The following is a sample template of the file called encfs-sidecar-args.json which is located in the encfs folder (/confidential-sidecar-containers/examples/encfs/)



{ "azure_filesystems": [ { "mount_point": "/mnt/remote/share", "azure_url": "<URL of the blob we uploaded above; grab it from Azure portal>", "azure_url_private": true, "key": { "kid": "<name the key that will be uploaded to M-HSM/AKV later>", "authority": { "endpoint": "<MAA Endpoint>" }, "akv": { "endpoint": "<Your Managed HSM Endpoint>" } } } ] }




Please fill in the rest of the details.

i) Install base64 if not already installed:




sudo apt-get install cl-base64



Use the following to convert the above json file into base64 (since this is going to be sent as arguments in another json tempalate)



base64 -w0 encfs-sidecar-args.json



i) Take the base64 value and update the following section (value field) in the file at aci-arm-template.json at (/confidential-sidecar-containers/examples/encfs)



"environmentVariables": [ { "name": "EncfsSideCarArgs", "value": "<Use the hash you generated in the previous steps>" } ]



Then, please follow instructions here azure-cli-extensions/src/confcom/azext_confcom at main ยท Azure/azure-cli-extensions (github.com) to install the confcom extension.



az extension add --name confcom



j) Next let's modify the aci-arm-template.json file.



"identity": { "type": "UserAssigned", "userAssignedIdentities": { "<full property value of the user assigned managed identity you created in the pre-requisites>": {} } }



At this point, we need to generate the CCE policy using confcom we installed in the previous step. But before that, we need to ensure that we add information about our container and replace the already provided container information in the aci-arm-template.json file.



{ "name": "<name that you pick for the new container in Confidential ACI>", "properties": { "command": [], "image": "<your_ACR_instance_url/your_image:tag>", "resources": { "requests": { "cpu": or anything you like, please be aware of limits>, "memoryInGb": <6 or anything you like, please be aware of limits> } }, "volumeMounts": [ { "name": "remotemounts", "mountPath": "/mnt/remote" } ], "ports": [ { "port": <your app port> } ] } }



I would also add the following value but it is optional.



"imageRegistryCredentials": [ { "server": "<your_ACR_server_URL>", "username": "<your acr user name under Access Keys>", "password": "<your ACR password under Access Keys>" } ]



Please ensure that the CCE policy value is an empty string for now. This will be generated when we run the next command.



"confidentialComputeProperties": { "ccePolicy": "" }



k) Now, let's run the command to fill in the CCE policy using policygen.



az confcom acipolicygen -a aci-arm-template.json --debug-mod -->> You may remove debug-mod for production releases & ensure that no one can access the container (NOT EVEN GUEST USERS on your side)



Note: Ensuring no debug-mode and auditing are currently something that needs to be performed by the parties together to ensure there is no violation of confidentiality in the infrastructure code.

After this is run, we will see that the same template file aci-arm-template.json has a large hash value in the ccepolicy field.

The output of this will also have the value to be filled into in the next step.



"claim": "x-ms-sevsnpvm-hostdata"




This needs to be filled into the file called importkeyconfig.json at the same location (/confidential-sidecar-containers/examples/encfs)

l) Now, the next step is to upload the key which will be used to decrypt the file system to MHSM and this key will be released to the container side car using the Secure Key Release process.

In order to upload the key we need to fill up another json template called importkeyconfig.json located in(/confidential-sidecar-containers/examples/encfs). Here is a sample.




{ "key": { "kid": "<your key name that will be uploaded to MHSM>", "authority": { "endpoint": "<Microsoft Azure Attestation URL>" }, "akv": { "endpoint": "acidemomhsm.managedhsm.azure.net", "api_version": "api-version=7.3-preview", "bearer_token": "<Generate bearer token to authenticate to the MHSM as instructed & place it here>" } }, "claims": [ [ { "claim": "x-ms-sevsnpvm-hostdata", "equals": "<Use the value generated by >" }, { "claim": "x-ms-compliance-status", "equals": "azure-compliant-uvm" }, { "claim": "x-ms-sevsnpvm-is-debuggable", "equals": "false" } ] ] }



Once the values are filled in, now we need to upload the Key into MHSM. 

Before we run the actual step, we will need to generate a bearer token (short lived) and use it in the template above as shown.

To generate the bearer token, we can use the following:




az account get-access-token --resource https://managedhsm.azure.net



 Now, the template should have all placeholders filled in with real value. Let's run it as follows:




go run ../../tools/importkey/main.go -c importkeyconfig.json -kh <key hash generated in step h above>




Check and validate that the key has been uploaded to MHSM. 


m) The time has come to finally create the container group using the template aci-arm-template.json. This can be done from the CLI or the portal. I am going to use a template deployment approach from the portal.

Step 1: Copy the contents of the template from VSCODE or whatever editor you used so far.

Step 2: Go to Azure Portal and search "Deploy a custom template"



Step 3:  Click on Build Your Own Template


Step 4: Now replace the contents in the editor with the contents of your aci-arm-template.json & Save

Step 5: It will show a page like this 



Step 5a: Deploy!! by clicking Review + Create

Once this is done, you should be able to run your app from a portal.



Here is how to create & deploy the app - the files can be also found on our official github page.





Published on:

Learn more
Azure Confidential Computing Blog articles
Azure Confidential Computing Blog articles

Azure Confidential Computing Blog articles

Share post:

Related posts

Playwright in Action: From Setup to Best Practices | Azure Developers JavaScript Day 2024

  Another excellent session was held during the Azure Developers JavaScript Day 2024 event! This time, the focus was on Playwr...

13 hours ago

What’s new across Azure Governance services, Microsoft Build 2024

Azure Governance Services have been undergoing exciting developments, with exciting new releases available in the past six months, aimed at pr...

1 day ago

Azure SDK Release (May 2024)

Stay up-to-date on the latest improvements and features with this comprehensive overview of Azure SDK's May 2024 release. The monthly release ...

1 day ago

Automatically enable system managed identity for App Service apps with Azure Policy

A common challenge when updating app service apps with the standard App service ARM template is the mandatory "serverFarmId" property. Th...

1 day ago

Automating Azure Remediation for Policy Initiatives with Azure PowerShell

Introduction and Current Challenges: Policy remediation is a critical aspect of Azure Policy, a service in Microsoft Azure used to create, ass...

1 day ago
Stay up to date with latest Microsoft Dynamics 365 and Power Platform news!
* Yes, I agree to the privacy policy