Loading...

Azure Data Factory CI-CD using YAML template

Azure Data Factory CI-CD using YAML template

Purpose of this Blog

  • Defined  the end to end Git integration and  Deployment flow using YAML template from a project implementation perspective
  • This can be easily implemented as you just need to include this YAML template and do a few changes as this has been successfully implemented  for Deployments across some projects.

Developer Workflow(CI/CD)

Integration of Code from Data Factory UI(Continuous Integration)


1. A sandbox Data Factory is created for development of data pipelines with Datasets and Linked Services.
The Data Factory is configured with Azure Dev-ops Git.(Collaboration and publish branch) and the root folder where the data factory code is committed

Akshay_Attota_0-1643924075912.png


2. A feature branch is created based on the main/collaboration branch for development. The branch in the Data Factory UI is changed to feature branch.

Akshay_Attota_1-1643924115116.png


3. After the developer tests the pipelines and is satisfied with the changes , a pull request is raised to merge
the changes with the main/collaboration branch

Akshay_Attota_2-1643924275798.png

 

4. After the PR gets approved by the concerned product leads, the changes made in feature branch are
merged into main branch.

Akshay_Attota_3-1643924335774.png


5. The branch is changed to main branch in Data Factory UI . The changes are published to main branch by
clicking on publish button.

Akshay_Attota_9-1643925485472.png


6. The changes are reflected in ARM template located in adf_publish branch

Akshay_Attota_8-1643925443625.png


Note: If some parameters are missing in the arm template follow the steps below :

I. then we should go to manage Akshay_Attota_7-1643925098371.png in ADF UI

II. click on  the ARM template

III. click on Edit parameter configuration in the DataFactory UI and include parameters that were missing in the ARM template in the json configuration file.

IV. We need to include this json configuration file in the main branch at root folder configured in Git configuration, let us consider we keep it in  'src/DataFactory/{DataFactoryName}/arm-template-parameters-definition.json'

 

- Follow this documentation on how to use the use custom parameters with the Resource Manager template : Custom parameters in a Resource Manager template - Azure Data Factory | Microsoft Docs 

 

Akshay_Attota_5-1643924912773.png


7. A Release pipeline is built in YAML to deploy the Data Factory using ARM template(We need to give
location of ARM template file) to Dev ,QA ,UAT and Production environments

 

Automated publish of ADF ARM Template :

YAML Release pipeline(Continuous Deployment)

  • Run the release pipeline for the specified target environment.
  • This will download the previously generated ARM template. It will also download secure connection strings
    from Azure Key-Vault. .Then it will deploy to your
    Target Data Factory using ARM template deployment.
  • The code below shows how to run your release agent on a Microsoft hosted agent

 

 

 

 

stages: - stage: Release displayName: Release stage jobs: - job: Release displayName: Release job pool: vmImage: 'Windows-2019'

 

 

 

 

 


Trigger:

  • This Pipeline will be triggered based on the changes made in  adf_publish branch

 

 

 

 

trigger: branches: include: - adf_publish

 

 

 

 


Resources:

  • The branch where the ARM template is located is mentioned.
    Referring to the previous point where it is mentioned that the ARM template will be picked from adf_publish (Publish) branch
  • Example:

 

 

 

 

resources: repositories: - repository: <repo name> type: git name: <repo name> ref: adf_publish

 

 

 

 

 

Steps:
1. Deployment Variables (variable declared)

 

 

 

 

#Deployment variables variables: KeyVaultName: <keyvaultname> azureSubscription: <ServiceConnection> SourceDataFactoryName : <Source Data Factory name from which code is published> DeployDataFactoryName : <Target Data Factory Name> DeploymentResourceGroupName : <Target Resource Group Name>

 

 

 

 


2. Keyvault task to fetch the secrets(parameters to override ARM paramters) (PowerShell Task):

 

 

 

 

- task: AzureKeyVault@1 inputs: azureSubscription: '$(azureSubscription)' KeyVaultName: $(KeyVaultName) SecretsFilter: '<secrets needed for overriding ARM parameters>' RunAsPreJob: true - checkout: <Repo Name>

 

 

 

 

 

3. Stopping the triggers of Target Data Factory before Deployment(Powershell Task)
Script:

 

 

 

 

- task: AzurePowerShell@5 displayName: Stop Triggers inputs: azureSubscription: '$(azureSubscription)' ScriptType: 'InlineScript' Inline: $triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName "$(DeployDataFactoryName)" -ResourceGroupName "$(DeploymentResourceGroupName)"; $triggersADF | ForEach-Object { Stop-AzDataFactoryV2Trigger – ResourceGroupName "$(DeploymentResourceGroupName)" -DataFactoryName "$(DeployDataFactoryName)" -Name $_.name -Force } azurePowerShellVersion: 'LatestVersion'

 

 

 

 


4. Deploying the ARM template to the Target Data Factory by passing some values to the parameters in
Incremental Mode. (Powershell Task)
Script:

 

 

 

 

- task: AzurePowerShell@5 displayName: Deploy ADF inputs: azureSubscription: '$(azureSubscription)' ScriptType: 'InlineScript' Inline: 'New-AzResourceGroupDeployment -ResourceGroupName "$(DeploymentResourceGroupName)" -TemplateParameterFile "$(System.DefaultWorkingDirectory)/$(SourceDataFactoryName)/ARMTemplateParametersForFactory.json" -TemplateFile "$(System.DefaultWorkingDirectory)/$(SourceDataFactoryName)/ARMTemplateForFactory.json" -factoryName "$(DeployDataFactoryName)" #<parameter-overridden> : <value-to-be-overridden> there are parameters in arm template and overriden by key vault secrets #<parameter-overridden> : <value-to-be-overridden> -Mode "Incremental"' azurePowerShellVersion: 'LatestVersion'

 

 

 

 

 

5. Starting the Triggers after Successful Deployment(Powershell Task)
Script:

 

 

 

 

- task: AzurePowerShell@5 displayName: Restart inputs: azureSubscription: '$(azureSubscription)' ScriptType: 'InlineScript' Inline: $triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName "$(DeployDataFactoryName)" -ResourceGroupName "$(DeploymentResourceGroupName)"; $triggersADF | ForEach-Object { Start-AzDataFactoryV2Trigger -ResourceGroupName "$(DeploymentResourceGroupName)" -DataFactoryName "$(DeployDataFactoryName)" -Name $_.name -Force } azurePowerShellVersion: 'LatestVersion'

 

 

 

 

 

Full ADF CD Template:

 

 

 

 

 

name: Release-$(rev:r) trigger: branches: include: - adf_publish resources: repositories: - repository: <repo name> type: git name: <repo name> ref: adf_publish variables: KeyVaultName: <keyvaultname> azureSubscription: <ServiceConnection> SourceDataFactoryName : <Source Data Factory name from which code is published> DeployDataFactoryName : <Target Data Factory Name> DeploymentResourceGroupName : <Target Resource Group Name> stages: - stage: Release displayName: Release stage jobs: - job: Release displayName: Release job pool: vmImage: 'Windows-2019' steps: - task: AzureKeyVault@1 inputs: azureSubscription: '$(azureSubscription)' KeyVaultName: $(KeyVaultName) SecretsFilter: '<secrets needed for overriding ARM parameters>' RunAsPreJob: true - checkout: <repo name> - task: AzurePowerShell@5 displayName: Stop Triggers inputs: azureSubscription: '$(azureSubscription)' ScriptType: 'InlineScript' Inline: $triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName "$(DeployDataFactoryName)" -ResourceGroupName "$(DeploymentResourceGroupName)"; $triggersADF | ForEach-Object { Stop-AzDataFactoryV2Trigger – ResourceGroupName "$(DeploymentResourceGroupName)" -DataFactoryName "$(DeployDataFactoryName)" -Name $_.name -Force } azurePowerShellVersion: 'LatestVersion' - task: AzurePowerShell@5 displayName: Deploy ADF inputs: azureSubscription: '$(azureSubscription)' ScriptType: 'InlineScript' Inline: 'New-AzResourceGroupDeployment -ResourceGroupName "$(DeploymentResourceGroupName)" -TemplateParameterFile "$(System.DefaultWorkingDirectory)/$(SourceDataFactoryName)/ARMTemplateParametersForFactory.json" -TemplateFile "$(System.DefaultWorkingDirectory)/$(SourceDataFactoryName)/ARMTemplateForFactory.json" -factoryName "$(DeployDataFactoryName)" #<parameter-overridden> : <value-to-be-overridden> there are parameters in arm template and overriden by key vault secrets #<parameter-overridden> : <value-to-be-overridden> -Mode "Incremental"' azurePowerShellVersion: 'LatestVersion' - task: AzurePowerShell@5 displayName: Restart Triggers inputs: azureSubscription: '$(azureSubscription)' ScriptType: 'InlineScript' Inline: $triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName "$(DeployDataFactoryName)" -ResourceGroupName "$(DeploymentResourceGroupName)"; $triggersADF | ForEach-Object { Start-AzDataFactoryV2Trigger -ResourceGroupName "$(DeploymentResourceGroupName)" -DataFactoryName "$(DeployDataFactoryName)" -Name $_.name -Force } azurePowerShellVersion: 'LatestVersion'

 

 

 

 

 

 

 

 

Published on:

Learn more
Azure Data Blog articles
Azure Data Blog articles

Azure Data Blog articles

Share post:

Related posts

Skyrocket Your Efficiency: Dive into Azure Cloud-Native solutions

This blog invites you to explore the power of cloud-native solutions, which can transform the way businesses operate and innovate. As part of ...

2 hours ago

[Mitigated] Azure Lab Services - Maintenance update outage

Hi, We are experiencing a service outage due to ongoing maintenance since around July 21st, 4 pm PDT. The service is currently not available i...

1 day ago

Azure Lab Services - Maintenance update outage

Hi, We are experiencing a service outage due to ongoing maintenance since around July 21st, 4 pm PDT. The service is currently not available i...

1 day ago

Dataverse: Create Custom Integration To Azure Cosmos DB for PostgreSQL

In a world where integration is common to do. Especially, because clouds is a common term, for sure, there are requests to integrate Dataverse...

2 days ago

Recovery options for Azure Virtual Machines (VM) affected by CrowdStrike Falcon agent

We are aware of an issue that started on 19 July 2024 at 04:09UTC, which resulted in customers experiencing unresponsiveness and startup failu...

2 days ago

Use cases of Advanced Network Observability for your Azure Kubernetes Service clusters

Introduction  Advanced Network Observability is the inaugural feature of the Advanced Container Networking Services (ACNS) suite bringing...

3 days ago

Azure Update Manager to support CIS hardened images among other images

What’s coming in by end of July 2024: Azure Update Manager will add support for 35 CIS hardened images. This is the first time that Update Man...

3 days ago

Mastering your cloud journey: Essentials to Innovating, Migrating and Modernizing, on Azure

We are living during a time of rapid growth in AI technologies and seeing cloud complexity increase as a result of those advanced workloads, w...

4 days ago
Stay up to date with latest Microsoft Dynamics 365 and Power Platform news!
* Yes, I agree to the privacy policy