Azure Data Factory CI-CD using YAML template

Purpose of this Blog

• Defined  the end to end Git integration and  Deployment flow using YAML template from a project implementation perspective
• This can be easily implemented as you just need to include this YAML template and do a few changes as this has been successfully implemented  for Deployments across some projects.

Developer Workflow(CI/CD)

Integration of Code from Data Factory UI(Continuous Integration)

1. A sandbox Data Factory is created for development of data pipelines with Datasets and Linked Services.
The Data Factory is configured with Azure Dev-ops Git.(Collaboration and publish branch) and the root folder where the data factory code is committed

2. A feature branch is created based on the main/collaboration branch for development. The branch in the Data Factory UI is changed to feature branch.

3. After the developer tests the pipelines and is satisfied with the changes , a pull request is raised to merge
the changes with the main/collaboration branch

 

4. After the PR gets approved by the concerned product leads, the changes made in feature branch are
merged into main branch.

5. The branch is changed to main branch in Data Factory UI . The changes are published to main branch by
clicking on publish button.

6. The changes are reflected in ARM template located in adf_publish branch

Note: If some parameters are missing in the arm template follow the steps below :

I. then we should go to manage  in ADF UI

II. click on  the ARM template

III. click on Edit parameter configuration in the DataFactory UI and include parameters that were missing in the ARM template in the json configuration file.

IV. We need to include this json configuration file in the main branch at root folder configured in Git configuration, let us consider we keep it in  'src/DataFactory/{DataFactoryName}/arm-template-parameters-definition.json'

 

- Follow this documentation on how to use the use custom parameters with the Resource Manager template : Custom parameters in a Resource Manager template - Azure Data Factory | Microsoft Docs 

 

7. A Release pipeline is built in YAML to deploy the Data Factory using ARM template(We need to give
location of ARM template file) to Dev ,QA ,UAT and Production environments

 

Automated publish of ADF ARM Template :

• The "Automated publish" improvement takes the validate all and export Azure Resource Manager (ARM) template functionality from the ADF UI and makes the logic consumable via a publicly available npm package @microsoft/azure-data-factory-utilities. This allows you to programmatically trigger these actions instead of going to the ADF UI and do a button click 'Publish'. This will give your CI/CD pipelines a truer continuous integration experience.(Quoted from blog)
• Please follow this tech community blog for more details : Automated publish improvement in ADF's CI/CD flow - Microsoft Tech Community
• Please follow this Microsoft Documentation for the same : Automated publishing for continuous integration and delivery - Azure Data Factory | Microsoft Docs

YAML Release pipeline(Continuous Deployment)

• Run the release pipeline for the specified target environment.
• This will download the previously generated ARM template. It will also download secure connection strings
  from Azure Key-Vault. .Then it will deploy to your
  Target Data Factory using ARM template deployment.
• The code below shows how to run your release agent on a Microsoft hosted agent

 

 

 

 

stages: - stage: Release displayName: Release stage jobs: - job: Release displayName: Release job pool: vmImage: 'Windows-2019'

 

 

 

 

 

Trigger:

• This Pipeline will be triggered based on the changes made in  adf_publish branch

 

 

 

 

trigger: branches: include: - adf_publish

 

 

 

 

Resources:

• The branch where the ARM template is located is mentioned.
  Referring to the previous point where it is mentioned that the ARM template will be picked from adf_publish (Publish) branch
• Example:

 

 

 

 

resources: repositories: - repository: <repo name> type: git name: <repo name> ref: adf_publish

 

 

 

 

 

Steps:
1. Deployment Variables (variable declared)

 

 

 

 

#Deployment variables variables: KeyVaultName: <keyvaultname> azureSubscription: <ServiceConnection> SourceDataFactoryName : <Source Data Factory name from which code is published> DeployDataFactoryName : <Target Data Factory Name> DeploymentResourceGroupName : <Target Resource Group Name>

 

 

 

 

2. Keyvault task to fetch the secrets(parameters to override ARM paramters) (PowerShell Task):

 

 

 

 

- task: AzureKeyVault@1 inputs: azureSubscription: '$(azureSubscription)' KeyVaultName: $(KeyVaultName) SecretsFilter: '<secrets needed for overriding ARM parameters>' RunAsPreJob: true - checkout: <Repo Name>

 

 

 

 

 

3. Stopping the triggers of Target Data Factory before Deployment(Powershell Task)
Script:

 

 

 

 

- task: AzurePowerShell@5 displayName: Stop Triggers inputs: azureSubscription: '$(azureSubscription)' ScriptType: 'InlineScript' Inline: $triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName "$(DeployDataFactoryName)" -ResourceGroupName "$(DeploymentResourceGroupName)"; $triggersADF | ForEach-Object { Stop-AzDataFactoryV2Trigger – ResourceGroupName "$(DeploymentResourceGroupName)" -DataFactoryName "$(DeployDataFactoryName)" -Name $_.name -Force } azurePowerShellVersion: 'LatestVersion'

 

 

 

 

4. Deploying the ARM template to the Target Data Factory by passing some values to the parameters in
Incremental Mode. (Powershell Task)
Script:

 

 

 

 

- task: AzurePowerShell@5 displayName: Deploy ADF inputs: azureSubscription: '$(azureSubscription)' ScriptType: 'InlineScript' Inline: 'New-AzResourceGroupDeployment -ResourceGroupName "$(DeploymentResourceGroupName)" -TemplateParameterFile "$(System.DefaultWorkingDirectory)/$(SourceDataFactoryName)/ARMTemplateParametersForFactory.json" -TemplateFile "$(System.DefaultWorkingDirectory)/$(SourceDataFactoryName)/ARMTemplateForFactory.json" -factoryName "$(DeployDataFactoryName)" #<parameter-overridden> : <value-to-be-overridden> there are parameters in arm template and overriden by key vault secrets #<parameter-overridden> : <value-to-be-overridden> -Mode "Incremental"' azurePowerShellVersion: 'LatestVersion'

 

 

 

 

 

5. Starting the Triggers after Successful Deployment(Powershell Task)
Script:

 

 

 

 

- task: AzurePowerShell@5 displayName: Restart inputs: azureSubscription: '$(azureSubscription)' ScriptType: 'InlineScript' Inline: $triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName "$(DeployDataFactoryName)" -ResourceGroupName "$(DeploymentResourceGroupName)"; $triggersADF | ForEach-Object { Start-AzDataFactoryV2Trigger -ResourceGroupName "$(DeploymentResourceGroupName)" -DataFactoryName "$(DeployDataFactoryName)" -Name $_.name -Force } azurePowerShellVersion: 'LatestVersion'

 

 

 

 

 

Full ADF CD Template:

 

 

 

 

 

name: Release-$(rev:r) trigger: branches: include: - adf_publish resources: repositories: - repository: <repo name> type: git name: <repo name> ref: adf_publish variables: KeyVaultName: <keyvaultname> azureSubscription: <ServiceConnection> SourceDataFactoryName : <Source Data Factory name from which code is published> DeployDataFactoryName : <Target Data Factory Name> DeploymentResourceGroupName : <Target Resource Group Name> stages: - stage: Release displayName: Release stage jobs: - job: Release displayName: Release job pool: vmImage: 'Windows-2019' steps: - task: AzureKeyVault@1 inputs: azureSubscription: '$(azureSubscription)' KeyVaultName: $(KeyVaultName) SecretsFilter: '<secrets needed for overriding ARM parameters>' RunAsPreJob: true - checkout: <repo name> - task: AzurePowerShell@5 displayName: Stop Triggers inputs: azureSubscription: '$(azureSubscription)' ScriptType: 'InlineScript' Inline: $triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName "$(DeployDataFactoryName)" -ResourceGroupName "$(DeploymentResourceGroupName)"; $triggersADF | ForEach-Object { Stop-AzDataFactoryV2Trigger – ResourceGroupName "$(DeploymentResourceGroupName)" -DataFactoryName "$(DeployDataFactoryName)" -Name $_.name -Force } azurePowerShellVersion: 'LatestVersion' - task: AzurePowerShell@5 displayName: Deploy ADF inputs: azureSubscription: '$(azureSubscription)' ScriptType: 'InlineScript' Inline: 'New-AzResourceGroupDeployment -ResourceGroupName "$(DeploymentResourceGroupName)" -TemplateParameterFile "$(System.DefaultWorkingDirectory)/$(SourceDataFactoryName)/ARMTemplateParametersForFactory.json" -TemplateFile "$(System.DefaultWorkingDirectory)/$(SourceDataFactoryName)/ARMTemplateForFactory.json" -factoryName "$(DeployDataFactoryName)" #<parameter-overridden> : <value-to-be-overridden> there are parameters in arm template and overriden by key vault secrets #<parameter-overridden> : <value-to-be-overridden> -Mode "Incremental"' azurePowerShellVersion: 'LatestVersion' - task: AzurePowerShell@5 displayName: Restart Triggers inputs: azureSubscription: '$(azureSubscription)' ScriptType: 'InlineScript' Inline: $triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName "$(DeployDataFactoryName)" -ResourceGroupName "$(DeploymentResourceGroupName)"; $triggersADF | ForEach-Object { Start-AzDataFactoryV2Trigger -ResourceGroupName "$(DeploymentResourceGroupName)" -DataFactoryName "$(DeployDataFactoryName)" -Name $_.name -Force } azurePowerShellVersion: 'LatestVersion'