Announcing the General Availability of AMD-based Confidential VMs on Azure Databricks

We are excited to announce the general availability of AMD-based confidential virtual machines (VMs) for cluster nodes on Azure Databricks. Confidential VMs are part of the Azure confidential computing (ACC) portfolio and provide a trusted execution environment (TEE) for Azure Databricks clusters, protecting data while in use in memory. It is important to note that Azure already encrypts data at rest and in transit, and the introduction of confidential VMs provides an additional layer of security for sensitive data in use, helping organizations meet compliance requirements and protect their most valuable data.

By using Azure confidential computing on Azure Databricks, you gain the capability to encrypt your data end-to-end. This is valuable not only for confidential workloads but also for any scenario where you need to protect highly sensitive data residing in memory and prevent unauthorized access or tampering. The solution also supports Azure Managed HSM, a hardware security module that allows the customer to manage their own encryption keys for data at-rest, in-use, and in-transit.

To use confidential VMs on Azure Databricks, customers need to select one of the confidential VM types when creating a cluster. This type of cluster can then be used for any workload that requires the protection of highly sensitive data in memory.

For compute-optimized needs, DCasv5 confidential VMs are available, and for memory-optimized needs, ECasv5 confidential VMs can be used. These VMs are currently available in the following regions: East US, West US, North Europe, West Europe, Southeast Asia, Central India, East Asia, Switzerland North, Japan East and Italy North, and coming to additional regions soon.

Databricks partnership in Confidential Computing:

"We are thrilled to have collaborated with Microsoft to introduce Azure Databricks support for Azure confidential computing,” said David Meyer, Senior Vice President of Product, Databricks. “With the Databricks Lakehouse, customers can build an end-to-end data platform with increased confidentiality and privacy by encrypting data in use thanks to AMD-based confidential virtual machines.”

“Azure Databricks with confidential computing is our first choice for the robust protection of confidential customer data across multiple industries.

Our successful collaboration with Microsoft and Databricks enables our customers not only to unlock significant value from their data. It also emphasizes data privacy and ownership throughout the large-scale data analysis of sensitive information.”

Provision a Confidential VM cluster in Azure Databricks