This video will focus on how to use Azure Data Factory CI/CD Pipelines and promote ETL Pipelines from Dev environment to UAT then production. How to create an Azure DevOps pipeline to deploy Azure Data Factory pipelines, link services, triggers, datasets, etc to several Azure Data Factories.
Learn a complete tutorial for creating Azure Data Factory CI/CD pipelines with Git enabled that use different branches for developing new features for ETL pipelines and other Azure Data Factory components.
This will cover, creating three Azure Data Factories, two Azure Storage accounts, Azure Release pipeline with Azure Artifacts based on Git repo and deploying Azure Data Factory ARM templates files to another environment.
It will show how to develop all the Azure Data Factory components in the feature branch. Then how to raise a pull request to merge those changes with the master branch. After that publish Azure Data Factory development which will create adf_publish branch which will include the ARM template that describes the current Azure Data Factory Environment. And by using this ARM template, we can deploy it to UAT and Production environments.
Commands:
Stop trigger
armTemplate "$(System.DefaultWorkingDirectory)/[artifact source alias]/[SC folder]/ARMTemplateForFactory.json" ResourceGroupName "[RG name]" DataFactoryName "[Target ADF Env]" predeployment $true deleteDeployment $false
Start trigger
armTemplate "$(System.DefaultWorkingDirectory)/[artifact source alias]/[SC folder]/ARMTemplateForFactory.json" ResourceGroupName "[RG name]" DataFactoryName "[Target ADF Env]" predeployment $true deleteDeployment $true
Stop trigger
armTemplate "$(System.DefaultWorkingDirectory)/_adf_publish_source/ADFRadwanDev/ARMTemplateForFactory.json" ResourceGroupName "adfrg" DataFactoryName "ADFRadwanUAT" predeployment $true deleteDeployment $false
Start trigger
armTemplate "$(System.DefaultWorkingDirectory)/_adf_publish_source/ADFRadwanDev/ARMTemplateForFactory.json" ResourceGroupName "adfrg" DataFactoryName "ADFRadwanUAT" predeployment $true deleteDeployment $true
factoryName "[target ADF name]" AzureBlobStorage1_connectionString "" AzureBlobStorage2_connectionString ""
factoryName [target ADF name] AzureBlobStorage1_connectionString "" AzureBlobStorage2_connectionString ""
factoryName "ADFRadwanUAT" AzureBlobStorage1_connectionString "DefaultEndpointsProtocol=https;AccountName=storageaccradwan1;AccountKey=xx;EndpointSuffix=core.windows.net" AzureBlobStorage2_connectionString "DefaultEndpointsProtocol=https;AccountName=storageaccradwan2;AccountKey=xxx;EndpointSuffix=core.windows.net"
armTemplate "$(System.DefaultWorkingDirectory)/_adf_publish_source/ADFRadwanDev/ARMTemplateForFactory.json" ResourceGroupName "adfrg" DataFactoryName "ADFRadwanUAT" predeployment $true deleteDeployment $false
About the Author
Video:
• Mohamed Radwan is a Principal DevOps ...
Blog:
https://mohamedradwan.com
Linkedin
/ mohamedahmedradwan
GitHub
https://github.com/MohamedRadwanDevOps
Twitter:
/ mradwan06
DevOps #Azure #Azure_Data_Factory_tutorial #Azure_training #Azure_DevOps