Creating templates in Azure DevOps Pipelines

Building Azure DevOps YAML pipelines and continuously adding the same pipeline.yaml file? Over time; this file can grow and grow – copy and pasting the same job/task but changing variables? I recommend that you have a look at creating templates within your Azure DevOps pipelines to template common jobs/tasks that you can reuse within other pipelines!

Why create templates?

Some reasons into why I consider creating templates as part of my Azure DevOps pipelines:-

  • Reusability
  • Avoids duplication
  • Reduces complexity and size of creating a single pipeline
  • Set structure and pattern that I can follow throughout my pipelines
  • Used as a guide to create further stages
  • Saves time and allows me to create generic templates that I can use on multiple projects
  • Adding/removing a change in one file rather than a number of stages – changing the same config each time

I recommend checking out this blog post, Azure DevOps Pipelines – Keeping your pipelines DRY (Don’t Repeat Yourself) – it is following my thoughts and suggestions of keeping your pipelines DRY!

Azure DevOps Pipeline with no templates

First, lets have a look at an Azure DevOps Pipeline that doesn’t use templates, as you can see – one huge YAML file, I would not call this a clean approach. (GitHub URL)

Over time, this will grow – this is when I recommend to introduce templating. A template dedicated to each task!

name: $(BuildDefinitionName)_$(date:yyyyMMdd)$(rev:.r)

trigger: none

pr: none

variables:
  - group: azurefestivecalendar-production
  - name: backendServiceArm
    value: 'tamopstf2'
  - name: backendAzureRmResourceGroupName
    value: 'tamopstfstates'
  - name: backendAzureRmStorageAccountName
    value: 'tfstatedevops'
  - name: backendAzureRmContainerName
    value: 'azuredevopstemplates'
  - name: backendAzureRmKey
    value: 'terraform.tfstate'
  - name: environment
    value: 'production'

stages :
  - stage: validate
    jobs:
    - job: validate
      continueOnError: false
      steps:
      - task: TerraformInstaller@0
        displayName: 'install'
        inputs:
          terraformVersion: '0.13.3'
      - task: TerraformTaskV1@0
        displayName: 'init'
        inputs:
          provider: 'azurerm'
          command: 'init'
          backendServiceArm: 'tamopstf'
          backendAzureRmResourceGroupName: '${{ variables.backendAzureRmResourceGroupName}}'
          backendAzureRmStorageAccountName: '${{ variables.backendAzureRmStorageAccountName}}'
          backendAzureRmContainerName: '${{ variables.backendAzureRmContainerName }}'
          backendAzureRmKey: 'terraform.tfstate'
          workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'
      - task: TerraformTaskV1@0
        displayName: 'validate'
        inputs:
          provider: 'azurerm'
          command: 'validate'
          
  - stage: plan
    dependsOn: [validate]
    condition: succeeded('validate')
    jobs:
      - job: terraform_plan_develop
        steps:
              - checkout: self
              - task: TerraformInstaller@0
                displayName: 'install'
                inputs:
                  terraformVersion: '0.13.3'
              - task: TerraformTaskV1@0
                displayName: 'init'
                inputs:
                  provider: 'azurerm'
                  command: 'init'
                  backendServiceArm: 'tamopstf'
                  backendAzureRmResourceGroupName: '${{ variables.backendAzureRmResourceGroupName}}'
                  backendAzureRmStorageAccountName: '${{ variables.backendAzureRmStorageAccountName}}'
                  backendAzureRmContainerName: '${{ variables.backendAzureRmContainerName }}'
                  backendAzureRmKey: 'terraform.tfstate'
                  workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'
              - task: TerraformTaskV1@0
                displayName: 'plan'
                inputs:
                  provider: 'azurerm'
                  command: 'plan'
                  commandOptions: '-input=false -var-file="../$(Environment)/$(Environment).tfvars"'
                  environmentServiceNameAzureRM: 'tamopstf'
                  workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'

  - stage: apply
    dependsOn: [plan]
    condition: succeeded('plan')
    jobs:
      - job: terraform_apply_production
        steps:
              - checkout: self
              - task: TerraformInstaller@0
                displayName: 'install'
                inputs:
                  terraformVersion: '0.13.3'
              - task: TerraformTaskV1@0
                displayName: 'init'
                inputs:
                  provider: 'azurerm'
                  command: 'init'
                  backendServiceArm: 'tamopstf'
                  backendAzureRmResourceGroupName: '${{ variables.backendAzureRmResourceGroupName}}'
                  backendAzureRmStorageAccountName: '${{ variables.backendAzureRmStorageAccountName}}'
                  backendAzureRmContainerName: '${{ variables.backendAzureRmContainerName }}'
                  backendAzureRmKey: 'terraform.tfstate'
                  workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'
              - task: TerraformTaskV1@0
                displayName: 'plan'
                inputs:
                  provider: 'azurerm'
                  command: 'plan'
                  commandOptions: '-input=false -var-file="../$(Environment)/$(Environment).tfvars"'
                  environmentServiceNameAzureRM: 'tamopstf'
                  workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'
              - task: TerraformTaskV1@0
                displayName: 'apply'
                inputs:
                  provider: 'azurerm'
                  command: 'apply'
                  commandOptions: '-input=false -auto-approve -var-file="../$(Environment)/$(Environment).tfvars"'
                  environmentServiceNameAzureRM: 'tamopstf'
                  workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'

  - stage: test
    dependsOn: [apply]
    condition: succeeded('apply')
    jobs:
      - job: Inspec_Testing
        steps:
          - task: UseRubyVersion@0
            displayName: 'Install Ruby'
            inputs:
              versionSpec: '>= 2.5'
              addToPath: true
 
          - script: gem install inspec-bin
            displayName: 'Install inspec-bin'
                       
          - task: Bash@3
            displayName: 'Run inspec tests'
            inputs:
              targetType: inline
              script: |
                export AZURE_SUBSCRIPTION_ID=$(AZURE_SUBSCRIPTION_ID)
                export AZURE_CLIENT_ID=$(AZURE_CLIENT_ID)
                export AZURE_CLIENT_SECRET=$(AZURE_CLIENT_SECRET)
                export AZURE_TENANT_ID=$(AZURE_TENANT_ID)
                inspec exec ./azure-inspec-tests/ -t azure:// --chef-license=accept --reporter cli junit:inspectestresults.xml
 
          - task: PublishTestResults@2
            displayName: Publish inspec test results
            condition: succeededOrFailed()
            inputs:
              testResultsFiles: '**/inspectestresults.xml'
              mergeTestResults: true

  - stage: bash
    dependsOn: [apply]
    condition: succeeded('apply')
    jobs:
      - job: bash_echo
        steps:
              - task: Bash@3
                displayName: 'Echo Test'
                inputs:
                  targetType: inline
                  script: |
                    Echo "Test Script!"

  - stage: cli
    dependsOn: [apply]
    condition: succeeded('apply')
    jobs:
      - job: azcli_resourcegroup_create
        steps:
              - task: AzureCLI@2
                displayName: 'Deploy Resource Group'
                inputs:
                  azureSubscription: 'tamopstf'
                  scriptType: bash
                  scriptLocation: inlineScript
                  addSpnToEnvironment: true
                  inlineScript: |
                    #!/bin/bash
                    az group create -l uksouth -n bicep-rg 

Identifying jobs/tasks that can be templated

Lets identity common tasks within the pipeline:-

  • Terraform Init/Validate
  • Terraform Plan
  • Terraform Apply
  • Inspec Testing
  • Bash Script
  • Azure CLI

Current folder structure

Azure-DevOps-Pipelines-Using-Templates
    └── azure-inspec-tests
    └── pipelines
    └── production
    └── terraform

I will add a folder into pipelines – to store my templates

Azure-DevOps-Pipelines-Using-Templates
    └── azure-inspec-tests
    └── pipelines
         └── templates
    └── production
    └── terraform

How are templates referenced within Azure DevOps Pipelines?

Using – template reference , template below

stages :   
  - stage: terraform_validate
    jobs:
      - template: templates/terraform-validate.yaml

Moving pipeline from one file to using Azure DevOps Pipeline Templates

I will now show a template for each task identified:

  • Terraform Init/Validate
  • Terraform Plan
  • Terraform Apply
  • Inspec Testing
  • Bash Script
  • Azure CLI

Terraform Init/Validate

  jobs:
    - job: terraform_validate
      continueOnError: false
      steps:
      - task: TerraformInstaller@0
        displayName: 'install'
        inputs:
          terraformVersion: '${{ parameters.terraform_version }}'
      - task: TerraformTaskV1@0
        displayName: 'init'
        inputs:
          provider: 'azurerm'
          command: 'init'
          backendServiceArm: '${{ parameters.backendServiceArm }}'
          backendAzureRmResourceGroupName: '${{ parameters.backendAzureRmResourceGroupName }}'
          backendAzureRmStorageAccountName: '${{ parameters.backendAzureRmStorageAccountName }}'
          backendAzureRmContainerName: '${{ parameters.backendAzureRmContainerName }}'
          backendAzureRmKey: '${{ parameters.backendAzureRmKey }}' 
          workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'
      - task: TerraformTaskV1@0
        displayName: 'validate'
        inputs:
          provider: 'azurerm'
          command: 'validate'

Terraform Plan

  jobs:
    - job: terraform_plan
      steps:
        - task: TerraformInstaller@0
          displayName: 'install'
          inputs:
            terraformVersion: '${{ parameters.terraform_version }}'
        - task: TerraformTaskV1@0
          displayName: 'init'
          inputs:
            provider: 'azurerm'
            command: 'init'
            backendServiceArm: '${{ parameters.backendServiceArm }}'
            backendAzureRmResourceGroupName: '${{ parameters.backendAzureRmResourceGroupName }}'
            backendAzureRmStorageAccountName: '${{ parameters.backendAzureRmStorageAccountName }}'
            backendAzureRmContainerName: '${{ parameters.backendAzureRmContainerName }}'
            backendAzureRmKey: '${{ parameters.backendAzureRmKey }}' 
            workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'
        - task: TerraformTaskV1@0
          displayName: 'plan'
          inputs:
            provider: 'azurerm'
            command: 'plan'
            commandOptions: '-input=false -var-file="../vars/${{ parameters.environment }}/${{ parameters.environment }}.tfvars"'
            environmentServiceNameAzureRM: '${{ parameters.backendServiceArm }}'
            workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'

Terraform Apply

  jobs:
    - job: terraform_apply
      steps:
        - task: TerraformInstaller@0
          displayName: 'install'
          inputs:
            terraformVersion: '${{ parameters.terraform_version }}'
        - task: TerraformTaskV1@0
          displayName: 'init'
          inputs:
            provider: 'azurerm'
            command: 'init'
            backendServiceArm: '${{ parameters.backendServiceArm }}'
            backendAzureRmResourceGroupName: '${{ parameters.backendAzureRmResourceGroupName }}'
            backendAzureRmStorageAccountName: '${{ parameters.backendAzureRmStorageAccountName }}'
            backendAzureRmContainerName: '${{ parameters.backendAzureRmContainerName }}'
            backendAzureRmKey: '${{ parameters.backendAzureRmKey }}' 
            workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'
        - task: TerraformTaskV1@0
          displayName: 'plan'
          inputs:
            provider: 'azurerm'
            command: 'plan'
            commandOptions: '-input=false -var-file="../vars/${{ parameters.environment }}/${{ parameters.environment }}.tfvars"'
            environmentServiceNameAzureRM: '${{ parameters.backendServiceArm }}'
            workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'
        - task: TerraformTaskV1@0
          displayName: 'apply'
          inputs:
            provider: 'azurerm'
            command: 'apply'
            commandOptions: '-input=false -auto-approve -var-file="../vars/${{ parameters.environment }}/${{ parameters.environment }}.tfvars"'
            environmentServiceNameAzureRM: '${{ parameters.backendServiceArm }}'
            workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'

Inspec Testing

  jobs:
    - job: Inspec_Testing
      steps:
        - task: UseRubyVersion@0
          displayName: 'Install Ruby'
          inputs:
            versionSpec: '${{ parameters.ruby_versionspec }}'
            addToPath: true

        - script: gem install inspec-bin
          displayName: 'Install inspec-bin'
                      
        - task: Bash@3
          displayName: 'Run inspec tests'
          inputs:
            targetType: inline
            script: |
              export AZURE_SUBSCRIPTION_ID=$('${{ parameters.AZURE_SUBSCRIPTION_ID }}')
              export AZURE_CLIENT_ID=$('${{ parameters.AZURE_CLIENT_ID }}')
              export AZURE_CLIENT_SECRET=$('${{ parameters.AZURE_CLIENT_SECRET }}')
              export AZURE_TENANT_ID=$('${{ parameters.AZURE_TENANT_ID }}')
              inspec exec ./azure-inspec-tests/ -t azure:// --chef-license=accept --reporter cli junit:inspectestresults.xml

        - task: PublishTestResults@2
          displayName: Publish inspec test results
          condition: succeededOrFailed()
          inputs:
            testResultsFiles: '**/inspectestresults.xml'
            mergeTestResults: true

Bash Script

  jobs:
    - job: bash_echo
      steps:
        - task: Bash@3
          displayName: 'Echo Test'
          inputs:
            targetType: inline
            script: |
              Echo "${{ parameters.bash_input }}"

Azure CLI

  jobs:
    - job: azcli_resourcegroup_create
      steps:
            - task: AzureCLI@2
              displayName: 'Deploy Resource Group'
              inputs:
                azureSubscription: 'tamopstf'
                scriptType: bash
                scriptLocation: inlineScript
                addSpnToEnvironment: true
                inlineScript: |
                  #!/bin/bash
                  az group create -l uksouth -n ${{ parameters.resource_group }}

Awesome, all the jobs/tasks have now been templated, how does the main Azure DevOps pipeline now look?

name: $(BuildDefinitionName)_$(date:yyyyMMdd)$(rev:.r)

trigger: none

pr: none

variables:
  - group: azurefestivecalendar-production
  - name: backendServiceArm
    value: 'tamopstf2'
  - name: backendAzureRmResourceGroupName
    value: 'tamopstfstates'
  - name: backendAzureRmStorageAccountName
    value: 'tfstatedevops'
  - name: backendAzureRmContainerName
    value: 'azuredevopstemplates'
  - name: backendAzureRmKey
    value: 'terraform.tfstate'
  - name: environment
    value: 'production'
  - name: terraform_version
    value: '0.13.3'
  - name: ruby_versionspec
    value: '>= 2.5'

stages :   
  - stage: terraform_validate
    jobs:
      - template: templates/terraform-validate.yaml
        parameters:
          backendServiceArm: ${{ variables.backendServiceArm }}
          backendAzureRmResourceGroupName: ${{ variables.backendAzureRmResourceGroupName }}
          backendAzureRmStorageAccountName: ${{ variables.backendAzureRmStorageAccountName }}
          backendAzureRmContainerName: ${{ variables.backendAzureRmContainerName }}
          backendAzureRmKey: ${{ variables.backendAzureRmKey }}
          environment: ${{ variables.environment }}
          terraform_version: ${{ variables.terraform_version }}

  - stage: terraform_plan
    dependsOn: [terraform_validate]
    condition: succeeded('terraform_validate')
    jobs:
      - template: templates/terraform-plan.yaml
        parameters:
          backendServiceArm: ${{ variables.backendServiceArm }}
          backendAzureRmResourceGroupName: ${{ variables.backendAzureRmResourceGroupName }}
          backendAzureRmStorageAccountName: ${{ variables.backendAzureRmStorageAccountName }}
          backendAzureRmContainerName: ${{ variables.backendAzureRmContainerName }}
          backendAzureRmKey: ${{ variables.backendAzureRmKey }}
          environment: ${{ variables.environment }}
          terraform_version: ${{ variables.terraform_version }}

  - stage: terraform_apply
    dependsOn: [terraform_plan]
    condition: succeeded('terraform_plan')
    jobs:
      - template: templates/terraform-apply.yaml
        parameters:
          backendServiceArm: ${{ variables.backendServiceArm }}
          backendAzureRmResourceGroupName: ${{ variables.backendAzureRmResourceGroupName }}
          backendAzureRmStorageAccountName: ${{ variables.backendAzureRmStorageAccountName }}
          backendAzureRmContainerName: ${{ variables.backendAzureRmContainerName }}
          backendAzureRmKey: ${{ variables.backendAzureRmKey }}
          environment: ${{ variables.environment }}
          terraform_version: ${{ variables.terraform_version }}

  - stage: inspec_testing
    dependsOn: [terraform_apply]
    condition: succeeded('terraform_apply')
    jobs:
      - template: templates/inspec-testing.yaml
        parameters:
          ruby_versionspec: ${{ variables.backendServiceArm }}
          AZURE_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)
          AZURE_CLIENT_ID: $(AZURE_CLIENT_ID)
          AZURE_CLIENT_SECRET: $(AZURE_CLIENT_SECRET)
          AZURE_TENANT_ID: $(AZURE_TENANT_ID)

  - stage: bash
    dependsOn: [terraform_apply]
    condition: succeeded('terraform_apply')
    jobs:
      - template: templates/bash-script.yaml
        parameters:
          bash_input: "Test Script!"

  - stage: cli
    dependsOn: [terraform_apply]
    condition: succeeded('terraform_apply')
    jobs:
      - template: templates/az-cli.yaml
        parameters:
          resource_group: "bicep-rg"

Notice the difference? The pipeline is now alot cleaner with no bloating!

I have created a number of generic templates that can now be used for the associated tasks within any additional pipelines – reusability! Awesome 🙂

Using this approach will assist you in reducing additional effort required creating new pipelines as you will have a number of tasks already templated!

Git Repository for code used

6 comments

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s