Azure DevOps Pipelines – Remove the need to hard-code – Thomas Thornton – Microsoft Azure MVP

Creating Azure DevOps Pipelines? If avoidable, try not hard code values within your DevOps pipeline! In this blog post; I am going to go through a small Terraform pipeline from hard-coded values to using parameters & variables

What is hard-code?

Think of it from a pipeline perspective, you create a stage within your pipeline and enter numerous inputs as static values in various tasks; these values cannot be amended in any way, you want another similar stage – you copy and paste and change so values, now your pipeline is growing unnecessary or if you want to reuse the same pipeline it can be tedious trying to change all the hard-coded values before you can re-use the same pipeline

Some more examples of “hardcode”

File names
Storage Account Name
Application version
Boolean values
Static values

Notice how they are all static references? When you run your pipeline; these values will never change.

The pipeline hardcoded

Lets have a look at a small Terraform pipeline that is completed hard-coded:-

name: $(BuildDefinitionName)_$(date:yyyyMMdd)$(rev:.r)

trigger: none

pr: none

stages :        
  - stage: terraform_plan
    jobs:
      - job: terraform_plan
        steps:
              - checkout: self

              - task: TerraformInstaller@0
                displayName: 'install'
                inputs:
                  terraformVersion: '0.13.4'

              - task: TerraformTaskV1@0
                displayName: 'init'
                inputs:
                  provider: 'azurerm'
                  command: 'init'
                  backendServiceArm: 'tamopstf'
                  backendAzureRmResourceGroupName: 'tamopstfstates'
                  backendAzureRmStorageAccountName: 'tfstatedevops'
                  backendAzureRmContainerName: 'azureterraformbuildartifacts'
                  backendAzureRmKey: 'terraform.tfstate'
                  workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'

              - task: TerraformTaskV1@0
                displayName: 'plan'
                inputs:
                  provider: 'azurerm'
                  command: 'plan'
                  commandOptions: '-input=false -var-file="../vars/production/production.tfvars"'
                  environmentServiceNameAzureRM: 'tamopstf'
                  workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'
              
              - task: ArchiveFiles@2
                inputs:
                  rootFolderOrFile: '$(Build.SourcesDirectory)'
                  includeRootFolder: false
                  archiveType: 'tar'
                  tarCompression: 'gz'
                  archiveFile: '$(Build.ArtifactStagingDirectory)/tfbuild.tgz'
                  replaceExistingArchive: true
                  displayName: 'Create Plan Artifact'

              - task: PublishBuildArtifacts@1
                inputs:
                  PathtoPublish: '$(Build.ArtifactStagingDirectory)'
                  ArtifactName: 'tfbuild-tfplan'
                  publishLocation: 'Container'
                  displayName: 'Publish Plan Artifact'    

              - task: DeleteFiles@1
                displayName: 'Remove unneeded files'
                inputs:
                  contents: |
                    .terraform
                    tfplan

  - stage: terraform_apply
    dependsOn: [terraform_plan]
    condition: succeeded('terraform_plan')
    jobs:
      - job: terraform_apply
        steps:
              - checkout: none

              - task: DownloadBuildArtifacts@0
                inputs:
                  artifactName: 'tfbuild-tfplan'
                  displayName: 'Download Plan Artifact'

              - task: ExtractFiles@1
                inputs:
                  archiveFilePatterns: '$(System.ArtifactsDirectory)/tfbuild-tfplan/tfbuild.tgz'
                  destinationFolder: '$(System.DefaultWorkingDirectory)/'
                  cleanDestinationFolder: false
                  displayName: 'Extract Terraform Plan Artifact'

              - task: TerraformInstaller@0
                displayName: 'install'
                inputs:
                  terraformVersion: '0.13.4'

              - task: TerraformTaskV1@0
                displayName: 'init'
                inputs:
                  provider: 'azurerm'
                  command: 'init'
                  backendServiceArm: 'tamopstf'
                  backendAzureRmResourceGroupName: 'tamopstfstates'
                  backendAzureRmStorageAccountName: 'tfstatedevops'
                  backendAzureRmContainerName: 'azureterraformbuildartifacts'
                  backendAzureRmKey: 'terraform.tfstate'
                  workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'

              - task: TerraformTaskV1@0
                displayName: 'plan'
                inputs:
                  provider: 'azurerm'
                  command: 'plan'
                  commandOptions: '-input=false -var-file="../vars/production/production.tfvars"'
                  environmentServiceNameAzureRM: 'tamopstf'
                  workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'

              - task: TerraformTaskV1@0
                displayName: 'apply'
                inputs:
                  provider: 'azurerm'
                  command: 'apply'
                  commandOptions: '-input=false -auto-approve -var-file="../vars/production/production.tfvars"'
                  environmentServiceNameAzureRM: 'tamopstf'
                  workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'

Within this pipeline, there is two stages terraform_plan & terraform_apply, in each task, lets have a look

terraformVersion has been hard-coded; you will most likely be updating terraformVersion on a quite regular basis & depending on how big the pipeline is – you may be changing it in several places

                inputs:
                  terraformVersion: '0.13.4'

The reference to the .tfstate file and its storage location is the next set of properities to be hard-coded, this will change between each pipeline and if you are looking at additional functionality such as for_each loops etc then removing this to be added in a more generic format

                  backendServiceArm: 'tamopstf'
                  backendAzureRmResourceGroupName: 'tamopstfstates'
                  backendAzureRmStorageAccountName: 'tfstatedevops'
                  backendAzureRmContainerName: 'azureterraformbuildartifacts'
                  backendAzureRmKey: 'terraform.tfstate'

commandOptions to where the terraform .tfvars file location should not be hardcoded

                  commandOptions: '-input=false -var-file="../vars/production/production.tfvars"'

                  environmentServiceNameAzureRM: 'tamopstf'

Updating the pipeline with Parameters & Variables

Why use Parameters & Variables?

They will allow you to change the state or values at a faster rate , all the possible changes are in one place – making the pipeline more seemless, allowing you to use the same stages for the ability to deploy to multiple environments!

Variable example in Azure DevOps Pipeline:-

variables:
  terraformVersion: 0.13.4

There is also quite a number of prefined variables within Azure DevOps – these are documented here , an example of this:-

$(System.DefaultWorkingDirectory)

Parameters example in Azure DevOps Pipeline:-

parameters:
- name: enviornment
  displayName: Environment to deploy
  type: string
  default: ../vars/production/production.tfvars
  values:
  - ../vars/production/production.tfvars
  - ../vars/develop/develop.tfvars

Lets look at what was hard-coded above & add into parameters & variables

variables:
  terraformVersion: 0.13.4
  backendServiceArm: 'tamopstf'
  backendAzureRmResourceGroupName: 'tamopstfstates'
  backendAzureRmStorageAccountName: 'tfstatedevops'
  backendAzureRmContainerName: 'azureterraformbuildartifacts'
  backendAzureRmKey: 'terraform.tfstate'

parameters:
- name: enviornment
  displayName: Environment to deploy
  type: string
  default: ../vars/production/production.tfvars
  values:
  - "../vars/production/production.tfvars"
  - "../vars/develop/develop.tfvars"

I have also used a built-in variable $(Build.BuildId), you will notice this in the below complete pipeline

Full pipeline below, notice the use of the variables & parameters? Removing the need for hard-coded values:-

name: $(BuildDefinitionName)_$(date:yyyyMMdd)$(rev:.r)

trigger: none

pr: none

variables:
  terraformVersion: 0.13.4
  backendServiceArm: 'tamopstf'
  backendAzureRmResourceGroupName: 'tamopstfstates'
  backendAzureRmStorageAccountName: 'tfstatedevops'
  backendAzureRmContainerName: 'azureterraformbuildartifacts'
  backendAzureRmKey: 'terraform.tfstate'

parameters:
- name: enviornment
  displayName: Environment to deploy
  type: string
  default: ../vars/production/production.tfvars
  values:
  - "../vars/production/production.tfvars"
  - "../vars/develop/develop.tfvars"

stages :        
  - stage: terraform_plan
    jobs:
      - job: terraform_plan
        steps:
              - checkout: self

              - task: TerraformInstaller@0
                displayName: 'install'
                inputs:
                  terraformVersion: ${{ variables.terraformVersion}}

              - task: TerraformTaskV1@0
                displayName: 'init'
                inputs:
                  provider: 'azurerm'
                  command: 'init'
                  backendServiceArm: ${{ variables.backendServiceArm}}
                  backendAzureRmResourceGroupName: ${{ variables.backendAzureRmResourceGroupName}}
                  backendAzureRmStorageAccountName: ${{ variables.backendAzureRmStorageAccountName}}
                  backendAzureRmContainerName: ${{ variables.backendAzureRmContainerName}}
                  backendAzureRmKey: ${{ variables.backendAzureRmKey}}
                  workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'

              - task: TerraformTaskV1@0
                displayName: 'plan'
                inputs:
                  provider: 'azurerm'
                  command: 'plan'
                  commandOptions: '-input=false -var-file=${{ parameters.enviornment }}'
                  environmentServiceNameAzureRM: ${{ variables.backendServiceArm}}
                  workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'
              
              - task: ArchiveFiles@2
                inputs:
                  rootFolderOrFile: '$(Build.SourcesDirectory)'
                  includeRootFolder: false
                  archiveType: 'tar'
                  tarCompression: 'gz'
                  archiveFile: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).tgz'
                  replaceExistingArchive: true
                  displayName: 'Create Plan Artifact'

              - task: PublishBuildArtifacts@1
                inputs:
                  PathtoPublish: '$(Build.ArtifactStagingDirectory)'
                  ArtifactName: '$(Build.BuildId)-tfplan'
                  publishLocation: 'Container'
                  displayName: 'Publish Plan Artifact'    

              - task: DeleteFiles@1
                displayName: 'Remove unneeded files'
                inputs:
                  contents: |
                    .terraform
                    tfplan

  - stage: terraform_apply
    dependsOn: [terraform_plan]
    condition: succeeded('terraform_plan')
    jobs:
      - job: terraform_apply
        steps:
              - checkout: none

              - task: DownloadBuildArtifacts@0
                inputs:
                  artifactName: '$(Build.BuildId)-tfplan'
                  displayName: 'Download Plan Artifact'

              - task: ExtractFiles@1
                inputs:
                  archiveFilePatterns: '$(System.ArtifactsDirectory)/$(Build.BuildId)-tfplan/$(Build.BuildId).tgz'
                  destinationFolder: '$(System.DefaultWorkingDirectory)/'
                  cleanDestinationFolder: false
                  displayName: 'Extract Terraform Plan Artifact'

              - task: TerraformInstaller@0
                displayName: 'install'
                inputs:
                  terraformVersion: ${{ variables.terraformVersion}}

              - task: TerraformTaskV1@0
                displayName: 'init'
                inputs:
                  provider: 'azurerm'
                  command: 'init'
                  backendServiceArm: ${{ variables.backendServiceArm}}
                  backendAzureRmResourceGroupName: ${{ variables.backendAzureRmResourceGroupName}}
                  backendAzureRmStorageAccountName: ${{ variables.backendAzureRmStorageAccountName}}
                  backendAzureRmContainerName: ${{ variables.backendAzureRmContainerName}}
                  backendAzureRmKey: ${{ variables.backendAzureRmKey}}
                  workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'

              - task: TerraformTaskV1@0
                displayName: 'plan'
                inputs:
                  provider: 'azurerm'
                  command: 'plan'
                  commandOptions: '-input=false -var-file=${{ parameters.enviornment }}'
                  environmentServiceNameAzureRM: ${{ variables.backendServiceArm}}
                  workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'

              - task: TerraformTaskV1@0
                displayName: 'apply'
                inputs:
                  provider: 'azurerm'
                  command: 'apply'
                  commandOptions: '-input=false -auto-approve -var-file=${{ parameters.enviornment }}'
                  environmentServiceNameAzureRM: ${{ variables.backendServiceArm}}
                  workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'

Notice how the pipelines stages are now more generic? If you wanted to reuse the same pipeline, you can just change the required variables and parameters!

Hard-coded pipeline can be found here in GitHub

Updated pipeline can be found here in GitHub