Azure Data Factory: How to split a file into multiple output files with Bicep

Introduction

In this article we will see how to split a csv file located in an Azure Storage Account through Azure Data Factory Data Flow.

We will do it through Azure Bicep in order to demonstrate the benefits of Infrastructure as Code (IaC) including:

• Reviewing the planned infrastructure that will be deployed through the what-if feature.
• Reproducible and testable infrastructures with templating deployments.

 

A complete procedure to deploy the following resources is available here: https://github.com/JamesDLD/bicep-data-factory-data-flow-split-file

• Azure Storage Account
• Uploading a test file that will be splitted
• Azure Data Factory
• Azure Data Factory Linked Service to connect Data Factory to our Storage Account
• Azure Data Factory Data Flow that will split our file
• Azure Data Factory Pipeline to trigger our Data Flow
• Bonus: Azure Data Factory Pipeline to cleanup the output container for your demos

Bicep code to create our linked service

The following Bicep code demonstrates how to create a Storage Account Linked Service through Azure Bicep.

Every body might not be concerned by the following limitation but in order to make this demo accessible to everyone we will create the Storage Account linked service through a connection string instead of using a managed identity which is definitely what I usually recommend.

If your blob account enables soft delete, system-assigned/user-assigned managed identity authentication isn’t supported in Data Flow.

If you access the blob storage through private endpoint using Data Flow, note when system-assigned/user-assigned managed identity authentication is used Data Flow connects to the ADLS Gen2 endpoint instead of Blob endpoint. Make sure you create the corresponding private endpoint in ADF to enable access.

Source: Data Flow/User-assigned managed identity authentication

Let’s have a look at the Bicep code!

The only trick here is to grab an existing storage account and pass its connection string through Bicep without having any secret in your code.