Welcome to the April 2023 Azure Synapse Analytics Update! This month, we have a new ARM template to deploy Azure Data Explorer DB with Cosmos DB connection, as well as additional updates in Apache Spark for Synapse, Synapse Data Explorer, and Data Integration. 

 

Read on for more details and don’t forget to watch the video! 

 

 

Table of contents

• Apache Spark for Synapse
  • Delta Lake – Low Shuffle Merge
• Synapse Data Explorer
  • Ingest data from Azure Events Hub to ADX free tier
  • New ARM template to deploy Azure Data Explorer DB with Cosmos DB connection
  • New look and feel for Query command bar in ADX Web
• Data Integration
  • Capture changed data from Cosmos DB analytical store (Public Preview)

 

Apache Spark for Synapse

Delta Lake – Low Shuffle Merge

Low Shuffle Merge optimization for Delta tables is now available in Apache Spark 3.2 and 3.3 pools. You can now update a Delta table with advanced conditions using the Delta Lake MERGE command. It can update data from a source table, view, or DataFrame into a target table. The current algorithm of the MERGE command is not optimized for handling unmodified rows. With Low Shuffle Merge optimization, unmodified rows are excluded from expensive shuffling execution and written separately.

 

To learn more about this new command, read Low Shuffle Merge optimization on Delta tables. 

 

Synapse Data Explorer

Ingest data from Azure Events Hub to ADX free tier

Are you looking for a powerful and cost-effective (or possibly even free...) way to analyze large volumes of near real-time streaming data? Azure Data Explorer now supports integration with Events Hub in ADX free tier.

 

Events Hub is a big data streaming platform, which can process millions of events per second in near real-time.  Connecting your Event Hub data to Azure Data Explorer is easy and straightforward and can be done in just a few simple steps, using an intuitive "One-Click" ingestion wizard.  

 

 

To learn more about ingesting data from Azure Events Hub to ADX free tier, read Free Event Hub data analysis with Azure Data Explorer. 

 

New ARM template to deploy Azure Data Explorer DB with Cosmos DB connection

A new ARM template that deploys an Azure Data Explorer DB with a Cosmos DB connection is now available. This vastly simplifies the deployment of an ADX cluster with:

• a System Assigned Identity
• a database
• an Azure Cosmos DB account (NoSql)
• an Azure Cosmos DB database
• an Azure Cosmos DB container
• a data connection between the Cosmos DB container and the Kusto database (using the System Assigned identity)

 

 

To learn more about this new ARM template, read Deploy Azure Data Explorer DB with Cosmos DB connection with ARM Template.

 

New look and feel for Query command bar in ADX Web

The ADX Web Query command bar has undergone a major redesign to provide an improved user experience. The new design is not only visually appealing but also makes it easier and faster for users to access the commands they need.

 

To learn more about the new command bar and other ADX Web updates, read ADX Web updates – March 2023. 

 

Data Integration

Capture changed data from Cosmos DB analytical store (Public Preview)

When you perform data integration and ETL processes in the cloud, your jobs can perform better and be more effective when you only read changed data from your source. We are excited to share that Azure Cosmos DB analytical store now supports change data capture (CDC) for Azure Cosmos DB API for NoSQL and Azure Cosmos DB API for MongoDB.  

 

Available in Public Preview, this will allow you to efficiently consume continuous and changed (inserted, updated, and deleted) data from the analytical store. Seamlessly integrated with Azure Synapse Analytics and Azure Data Factory, it is a scalable, no-code experience for high data volume and will not consume provisioned RUs or affect the performance of your transactional workloads while providing lower latency and a lower TCO.  

 

You can consume incremental analytical store data from a Cosmos DB container using either Azure Synapse Analytics or Azure Data Factory after you enable the Cosmos DB account for Synapse Link and you have enabled analytical store on a new container or an existing container. 

 

 

To learn more about capturing Change Data:

Watch Azure Cosmos DB analytical store Change Data Capture

Read Capture Changed Data From your Cosmos DB analytical store (Preview) and Now in preview – Change Data Capture (CDC) with Azure Cosmos DB analytical store.

 

Thanks for reading! That's all we have for you this month. We look forward to hearing your comments and questions. We'll see you here for the next monthly update!