Azure Data Factory: How to split a file into multiple output files with Bicep
Bicep code to create our Azure Data Factory Data Flow
Based on the following reference “Microsoft.DataFactory factories/linkedservices” we will create the Azure Data Factory Data Flow that will split our file into multiple files.
When using the az deployment what-if option we can see the following changes. This is really convenient to see the asked changes before applying them.
The Data Flow looks like the following screenshot where we can see the number of partition that will be created. In our context it corresponds to the number of csv files that will be generated from our input csv file.
The other trick here is to play with a file name pattern to manage the target files names.
The output files in this sample will be set to fit with the input file name, the current date and the output file iteration.
Split the file through the Pipeline
Through the procedure located here https://github.com/JamesDLD/bicep-data-factory-data-flow-split-file we have created an Azure Data Factory pipeline named “ArmtemplateSampleSplitFilePipeline”, you can trigger it to launch the Data Flow that will split the file.
The following screenshot illustrates the split result done through Azure Data Factory Data Flow.
Conclusion
Considering Bicep or any other Infrastructure as Code (IaC) tool ensures to gain efficiency and agility, its a real ramp up when designing infrastructures and it makes them reproducible and testable.
See You in the Cloud
Jamesdld
Published on:
Learn moreRelated posts
Selecting the Optimal Container for Azure AI: Docker, ACI, or AKS?
Deploying Azure AI services in containers like Docker, Azure Container Instances (ACI), or Azure Kubernetes Service (...
Azure Elastic SAN for Azure VMware Solution: now Generally Available
Have you been looking to expand your storage on Azure VMware Solution (AVS), but do not need the extra compute performance and the associated ...
Introducing Pull Request Annotation for CodeQL and Dependency Scanning in GitHub Advanced Security for Azure DevOps
In the world of software development, security is paramount. As developers, we strive to write clean, efficient, and most importantly, secure ...
Accelerate metadata heavy workloads with Metadata Caching preview for Azure Premium Files SMB & REST
Azure Files previously announced the limited preview of Metadata caching highlighting improvements on the metadata latency (up to 55...
How to Choose the Right Models for Your Apps | Azure AI
With more than 1700 models to choose from on Azure, selecting the right one is key to enabling the right capabilities, at the right price poin...
MMR Call Redirection for Azure Virtual Desktop, Windows 365 now available
Today, I am pleased to share the launch of Multimedia Redirection (MMR) Call Redirection for Azure Virtual Desktop and Windows 365. Call Redir...
Liquid Cooling in Air Cooled Data Centers on Microsoft Azure
With the advent of artificial intelligence and machine learning (AI/ML), hyperscale datacenters are increasingly accommodating AI accelerators...
Introducing Azure Product Retirement Livestreams
The Azure Retirements team, in collaboration with key partner groups, is excited t...
Azure Developer CLI (azd) – October 2024
This post announces the October release of the Azure Developer CLI (`azd`), including configurable api-version for ACA. The post Azure Develop...