Creating Templates For Azure DevOps Pipeline Jobs

Azure DevOps pipeline templates allow you to create multiple types of templates that you can define and reuse in multiple pipelines. In this blog post I am going to show how you can create template jobs! Each stage will have its own templated job that has multiple tasks. In this example I will be templating a Terraform validate, plan and apply stages

Why create templates?

Some reasons into why I consider creating templates as part of my Azure DevOps pipelines:-

  • Reusability
  • Avoids duplication
  • Reduces complexity and size of creating a single pipeline
  • Set structure and pattern that I can follow throughout my pipelines
  • Used as a guide to create further stages
  • Saves time and allows me to create generic templates that I can use on multiple projects

I recommend checking out this blog post, Azure DevOps Pipelines – Keeping your pipelines DRY (Don’t Repeat Yourself) – it is following my thoughts and suggestions of keeping your pipelines DRY!

Pipeline Structure

Below is the folder & pipeline structure that I will be using:-

  • pipelines folder:- The main pipeline along with the three templated jobs for terraform validate, plan & apply
  • terraform folder:- Terraform resources that I want to deploy
  • vars folder:- .tfvars for various environments, in this example it will just be production.tfvars

Notice the multiple references of .yaml?

pipeline.yaml ( pipeline.yaml GitHub )will be the main pipeline that I will be running within Azure DevOps and the templates/* are the templated pipeline jobs that I will be running within each stage:-

Breakdown of the main pipeline used in Azure DevOps

Lets look at the pipeline that I will be running within Azure DevOps – pipeline.yaml

name: $(BuildDefinitionName)_$(date:yyyyMMdd)$(rev:.r)

trigger: none

pr: none

variables:
  backendServiceArm: 'tamopstf2'
  backendAzureRmResourceGroupName: 'tamopstfstates'
  backendAzureRmStorageAccountName: 'tfstatedevops'
  backendAzureRmContainerName: 'azure-devops-template-pipelines'
  backendAzureRmKey: 'terraform.tfstate' 
  environment: production

stages :   
  - stage: terraform_validate
    jobs:
      - template: templates/terraform-validate.yaml
        parameters:
          backendServiceArm: ${{ variables.backendServiceArm }}
          backendAzureRmResourceGroupName: ${{ variables.backendAzureRmResourceGroupName }}
          backendAzureRmStorageAccountName: ${{ variables.backendAzureRmStorageAccountName }}
          backendAzureRmContainerName: ${{ variables.backendAzureRmContainerName }}
          backendAzureRmKey: ${{ variables.backendAzureRmKey }}
          environment: ${{ variables.environment }}

  - stage: terraform_plan
    dependsOn: [terraform_validate]
    condition: succeeded('terraform_validate')
    jobs:
      - template: templates/terraform-plan.yaml
        parameters:
          backendServiceArm: ${{ variables.backendServiceArm }}
          backendAzureRmResourceGroupName: ${{ variables.backendAzureRmResourceGroupName }}
          backendAzureRmStorageAccountName: ${{ variables.backendAzureRmStorageAccountName }}
          backendAzureRmContainerName: ${{ variables.backendAzureRmContainerName }}
          backendAzureRmKey: ${{ variables.backendAzureRmKey }}
          environment: ${{ variables.environment }}

  - stage: terraform_apply
    dependsOn: [terraform_plan]
    condition: succeeded('terraform_plan')
    jobs:
      - template: templates/terraform-apply.yaml
        parameters:
          backendServiceArm: ${{ variables.backendServiceArm }}
          backendAzureRmResourceGroupName: ${{ variables.backendAzureRmResourceGroupName }}
          backendAzureRmStorageAccountName: ${{ variables.backendAzureRmStorageAccountName }}
          backendAzureRmContainerName: ${{ variables.backendAzureRmContainerName }}
          backendAzureRmKey: ${{ variables.backendAzureRmKey }}
          environment: ${{ variables.environment }}

Within this example, I do not want the pipeline to run during a pull-request(pr) or to be triggered so I have added these references:-

trigger: none

pr: none

Notice in my referenced blog post above Azure DevOps Pipelines – Keeping your pipelines DRY (Don’t Repeat Yourself) that I reference the use of variables and to avoid duplication?

I’ve used variables that I reference within each templated job as below:-

variables:
  backendServiceArm: 'tamopstf2'
  backendAzureRmResourceGroupName: 'tamopstfstates'
  backendAzureRmStorageAccountName: 'tfstatedevops'
  backendAzureRmContainerName: 'azure-devops-template-pipelines'
  backendAzureRmKey: 'terraform.tfstate' 
  environment: production

Within each template I make reference to these variables, lets have a look at this. Checking out any of the stages, they are created using the same process. I am going to show a breakdown of the terraform_plan stage.

Notice the use of variables (${{ variables.backendServiceArm }} etc) as these are now referenced as parameters?

  - stage: terraform_plan
    dependsOn: [terraform_validate]
    condition: succeeded('terraform_validate')
    jobs:
      - template: templates/terraform-plan.yaml
        parameters:
          backendServiceArm: ${{ variables.backendServiceArm }}
          backendAzureRmResourceGroupName: ${{ variables.backendAzureRmResourceGroupName }}
          backendAzureRmStorageAccountName: ${{ variables.backendAzureRmStorageAccountName }}
          backendAzureRmContainerName: ${{ variables.backendAzureRmContainerName }}
          backendAzureRmKey: ${{ variables.backendAzureRmKey }}
          environment: ${{ variables.environment }}

Referencing another pipeline/template is down via

 - template: templates/terraform-plan.yaml

Why parameters? Lets check out a template!

Within each of my templates I’ve created – I can make reference to these variables from the main pipeline!

Looking at the below terraform-plan.yaml you can see reference to this:-

  jobs:
    - job: terraform_plan
      steps:
            - task: TerraformInstaller@0
              displayName: 'install'
              inputs:
                terraformVersion: '0.13.4'
            - task: TerraformTaskV1@0
              displayName: 'init'
              inputs:
                provider: 'azurerm'
                command: 'init'
                backendServiceArm: '${{ parameters.backendServiceArm }}'
                backendAzureRmResourceGroupName: '${{ parameters.backendAzureRmResourceGroupName }}'
                backendAzureRmStorageAccountName: '${{ parameters.backendAzureRmStorageAccountName }}'
                backendAzureRmContainerName: '${{ parameters.backendAzureRmContainerName }}'
                backendAzureRmKey: '${{ parameters.backendAzureRmKey }}' 
                workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'
            - task: TerraformTaskV1@0
              displayName: 'plan'
              inputs:
                provider: 'azurerm'
                command: 'plan'
                commandOptions: '-input=false -var-file="../vars/${{ parameters.environment }}/${{ parameters.environment }}.tfvars"'
                environmentServiceNameAzureRM: '${{ parameters.backendServiceArm }}'
                workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'

Awesome isn’t it?

What is being deployed?

In this example, I am deploying a resource group:- templates-rg , check out production.tfvars – you can template further, per environment! Notice how we can reuse templates? Constantly trying to keep them pipelines DRY!

Round-up

Hope you have enjoyed this blog post regarding Templating Azure DevOps Pipeline Jobs and how you can reference multiple .yaml files within the same pipeline!

The code used has been uploaded to my Github repo here

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s