Seequality

Don’t count rows in ETL, use Delta Log metrics!

Collecting statistics during your ETL process can be highly beneficial. These statistics can prove useful in various scenarios. For example, t...

1 day ago

Avoiding Issues: Monitoring Query Pushdowns in Databricks Federated Queries

A foreign catalog in Databricks is a specialized type of catalog that enables users to access and query data stored in external databases as i...

2 months ago

Terraforming Databricks #3: Lakehouse Federation

In today’s post, the third in the Terraforming Databricks series, we’ll break down the process of setting a connection to an Azure...

3 months ago

Microsoft Fabric: Using Workspace Identity for Authentication

One of the newest features available in Microsoft Fabric is the ability to use Workspace Identity to authenticate with external Data Lake Stor...

3 months ago

Terraforming Databricks #2: Catalogs & Schemas

In the first post of this series, we discussed the process of creating a metastore, which is essential for enabling workspaces for the Unity C...

4 months ago

Terraforming Databricks #1: Unity Catalog Metastore

Over the past two years, we have participated in numerous projects where Databricks was implemented on the Azure cloud from the ground up. Eac...

4 months ago

Executing SQL queries from Azure DevOps using Service Connection credentials

Executing SQL queries from Azure DevOps to an Azure SQL Database can be a complex and challenging task, particularly when it comes to establis...

4 months ago

Utilizing YAML Anchors in Databricks Asset Bundles

We all know what YAML is – it’s like JSON, just with indentation instead of brackets. Easier to write and read. That’s it, i...

4 months ago

Setup Git credentials for Service Principal in Azure Databricks

Jobs can be executed using notebooks or Python code from a remote Git repository or a Databricks Git folder. This approach simplifies the crea...

5 months ago

Databricks: MERGE WITH SCHEMA EVOLUTION

Anyone who has ever designed an ETL process involving more than a few tables of data has likely encountered the need to build a metadata-drive...

5 months ago