This video will focus on how to use Azure Data Factory CI/CD Pipelines and promote ETL Pipelines from Dev environment to UAT then production. How to create an Azure DevOps pipeline to deploy Azure Data Factory pipelines, link services, triggers, datasets, etc to several Azure Data Factories.

Learn a complete tutorial for creating Azure Data Factory CI/CD pipelines with Git enabled that use different branches for developing new features for ETL pipelines and other Azure Data Factory components.

This will cover, creating three Azure Data Factories, two Azure Storage accounts, Azure Release pipeline with Azure Artifacts based on Git repo and deploying Azure Data Factory ARM templates files to another environment.

It will show how to develop all the Azure Data Factory components in the feature branch. Then how to raise a pull request to merge those changes with the master branch. After that publish Azure Data Factory development which will create adf_publish branch which will include the ARM template that describes the current Azure Data Factory Environment. And by using this ARM template, we can deploy it to UAT and Production environments.

Commands:


Stop trigger

armTemplate "$(System.DefaultWorkingDirectory)/[artifact source alias]/[SC folder]/ARMTemplateForFactory.json" ResourceGroupName "[RG name]" DataFactoryName "[Target ADF Env]" predeployment $true deleteDeployment $false


Start trigger

armTemplate "$(System.DefaultWorkingDirectory)/[artifact source alias]/[SC folder]/ARMTemplateForFactory.json" ResourceGroupName "[RG name]" DataFactoryName "[Target ADF Env]" predeployment $true deleteDeployment $true


Stop trigger

armTemplate "$(System.DefaultWorkingDirectory)/_adf_publish_source/ADFRadwanDev/ARMTemplateForFactory.json" ResourceGroupName "adfrg" DataFactoryName "ADFRadwanUAT" predeployment $true deleteDeployment $false


Start trigger

armTemplate "$(System.DefaultWorkingDirectory)/_adf_publish_source/ADFRadwanDev/ARMTemplateForFactory.json" ResourceGroupName "adfrg" DataFactoryName "ADFRadwanUAT" predeployment $true deleteDeployment $true

factoryName "[target ADF name]" AzureBlobStorage1_connectionString "" AzureBlobStorage2_connectionString ""

factoryName [target ADF name] AzureBlobStorage1_connectionString "" AzureBlobStorage2_connectionString ""

factoryName "ADFRadwanUAT" AzureBlobStorage1_connectionString "DefaultEndpointsProtocol=https;AccountName=storageaccradwan1;AccountKey=xx;EndpointSuffix=core.windows.net" AzureBlobStorage2_connectionString "DefaultEndpointsProtocol=https;AccountName=storageaccradwan2;AccountKey=xxx;EndpointSuffix=core.windows.net"

armTemplate "$(System.DefaultWorkingDirectory)/_adf_publish_source/ADFRadwanDev/ARMTemplateForFactory.json" ResourceGroupName "adfrg" DataFactoryName "ADFRadwanUAT" predeployment $true deleteDeployment $false


 About the Author

Video:
   • Mohamed Radwan is a Principal DevOps ...  
Blog:
https://mohamedradwan.com
Linkedin
  / mohamedahmedradwan  
GitHub
https://github.com/MohamedRadwanDevOps
Twitter:
  / mradwan06  

DevOps #Azure #Azure_Data_Factory_tutorial #Azure_training #Azure_DevOps