Deploying Terraform using Azure DevOps with Build Artifacts

In this blog post, I am going to show how you can deploy Terraform using Azure DevOps with a Build Artifact that is created during the Terraform plan stage.

Why Build Artifacts for Terraform?

Working in a busy environment, you may be wanting multiple iterations of the Terraform pipeline; these iterations may require an approval process or requirement to be approved prior to being released into a specific environment. Multiple people are updating Terraform code into the master branch; these approvals backup and now the current branch version is different to the original approval requirement..

This is where build artifacts will help! In a later blog, I will show, how you can configure approval gates.

How can Build Artifacts help?

After you run a Terraform plan against the environment, you can .zip these contents along with the additional configurations stored in git into a build artifact on that pipeline.

So, when approval has been granted, you can run the exact version that was planned – whether this be hours or days later! The build artifact is stored within the pipeline, so when Terraform Apply is ran, it will download and unzip this artifact – using this Terraform configuration rather than pulling directly from master branch for the newer version of the Terraform code

Recommended reading

If you are unsure how to run and configure Terraform using Azure DevOps, I recommend my blog post Deploy Terraform using Azure DevOps prior to this blog post, it will go into detail the pre-requirements etc required to successfully run Terraform using Azure DevOps

I want to artifact, what is my Azure pipeline process now?

Lets have a look at the tasks that will be used now during the terraform_plan stage:-

Git Checkout:- Normal process, checkout required branch
Terraform Init:- Initial Terraform using Task:- TerraformTaskV1@0
Terraform Plan:- Terraform Plan using Task:- TerraformTaskV1@0
Archive Files:- Archive the directory that has both the git content and the Terraform Plan

Now the terraform_deploy stage:-

Git Checkout:- Checkout none, this stage will not checkout the required branch. It will use the code that was archived and zipped as part of the above stage
Download Build Artifacts:- Download artifacts from previous step using DownloadBuildArtifacts@0 Task
Extract Build Artifacts:- Extract the build artifacts using ExtractFiles@1 Task
Terraform Init:- Initial Terraform using Task:- TerraformTaskV1@0
Terraform Plan:- Terraform Plan using Task:- TerraformTaskV1@0 that will display the plan taken from above stage
Terraform Apply:- Terraform Apply using Task:- TerraformTaskV1@0 that applies required additions/changes from the Terraform Plan

Stages
    └──terraform_plan
            └──Git Checkout
            └──Terraform Init
            └──Terraform Plan
            └──Archive Files
            └──Publish Build Artifact
            └──Delete Files (Cleanup)
    └──terraform_apply
            └──Checkout: None
            └──Download Build Artifacts
            └──Extract Build Artifacts
            └──Terraform Init
            └──Terraform Plan
            └──Terraform Apply

Lets look at the above inside Azure Pipelines as code

Stage:- terraform_plan
Note:- I have included Terraform Installer task on each stage, so you can run this code directly using Azure DevOps Agent

  - stage: terraform_plan
    jobs:
      - job: terraform_plan
        steps:
              - checkout: self

              - task: TerraformInstaller@0
                displayName: 'install'
                inputs:
                  terraformVersion: '0.13.4'

              - task: TerraformTaskV1@0
                displayName: 'init'
                inputs:
                  provider: 'azurerm'
                  command: 'init'
                  backendServiceArm: 'tamopstf'
                  backendAzureRmResourceGroupName: 'tamopstfstates'
                  backendAzureRmStorageAccountName: 'tfstatedevops'
                  backendAzureRmContainerName: 'azureterraformbuildartifacts'
                  backendAzureRmKey: 'terraform.tfstate'
                  workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'

              - task: TerraformTaskV1@0
                displayName: 'plan'
                inputs:
                  provider: 'azurerm'
                  command: 'plan'
                  commandOptions: '-input=false -var-file="../vars/production/production.tfvars"'
                  environmentServiceNameAzureRM: 'tamopstf'
                  workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'
              
              - task: ArchiveFiles@2
                inputs:
                  rootFolderOrFile: '$(Build.SourcesDirectory)'
                  includeRootFolder: false
                  archiveType: 'tar'
                  tarCompression: 'gz'
                  archiveFile: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).tgz'
                  replaceExistingArchive: true
                  displayName: 'Create Plan Artifact'

              - task: PublishBuildArtifacts@1
                inputs:
                  PathtoPublish: '$(Build.ArtifactStagingDirectory)'
                  ArtifactName: '$(Build.BuildId)-tfplan'
                  publishLocation: 'Container'
                  displayName: 'Publish Plan Artifact'    

              - task: DeleteFiles@1
                displayName: 'Remove unneeded files'
                inputs:
                  contents: |
                    .terraform
                    tfplan

Stage:- terraform_apply

  - stage: terraform_apply
    dependsOn: [terraform_plan]
    condition: succeeded('terraform_plan')
    jobs:
      - job: terraform_apply
        steps:
              - checkout: none

              - task: DownloadBuildArtifacts@0
                inputs:
                  artifactName: '$(Build.BuildId)-tfplan'
                  displayName: 'Download Plan Artifact'

              - task: ExtractFiles@1
                inputs:
                  archiveFilePatterns: '$(System.ArtifactsDirectory)/$(Build.BuildId)-tfplan/$(Build.BuildId).tgz'
                  destinationFolder: '$(System.DefaultWorkingDirectory)/'
                  cleanDestinationFolder: false
                  displayName: 'Extract Terraform Plan Artifact'

              - task: TerraformInstaller@0
                displayName: 'install'
                inputs:
                  terraformVersion: '0.13.4'

              - task: TerraformTaskV1@0
                displayName: 'init'
                inputs:
                  provider: 'azurerm'
                  command: 'init'
                  backendServiceArm: 'tamopstf'
                  backendAzureRmResourceGroupName: 'tamopstfstates'
                  backendAzureRmStorageAccountName: 'tfstatedevops'
                  backendAzureRmContainerName: 'azureterraformbuildartifacts'
                  backendAzureRmKey: 'terraform.tfstate'
                  workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'

              - task: TerraformTaskV1@0
                displayName: 'plan'
                inputs:
                  provider: 'azurerm'
                  command: 'plan'
                  commandOptions: '-input=false -var-file="../vars/production/production.tfvars"'
                  environmentServiceNameAzureRM: 'tamopstf'
                  workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'

              - task: TerraformTaskV1@0
                displayName: 'apply'
                inputs:
                  provider: 'azurerm'
                  command: 'apply'
                  commandOptions: '-input=false -auto-approve -var-file="../vars/production/production.tfvars"'
                  environmentServiceNameAzureRM: 'tamopstf'
                  workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'


The full pipeline

name: $(BuildDefinitionName)_$(date:yyyyMMdd)$(rev:.r)

trigger: none

pr: none

stages :        
  - stage: terraform_plan
    jobs:
      - job: terraform_plan
        steps:
              - checkout: self

              - task: TerraformInstaller@0
                displayName: 'install'
                inputs:
                  terraformVersion: '0.13.4'

              - task: TerraformTaskV1@0
                displayName: 'init'
                inputs:
                  provider: 'azurerm'
                  command: 'init'
                  backendServiceArm: 'tamopstf'
                  backendAzureRmResourceGroupName: 'tamopstfstates'
                  backendAzureRmStorageAccountName: 'tfstatedevops'
                  backendAzureRmContainerName: 'azureterraformbuildartifacts'
                  backendAzureRmKey: 'terraform.tfstate'
                  workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'

              - task: TerraformTaskV1@0
                displayName: 'plan'
                inputs:
                  provider: 'azurerm'
                  command: 'plan'
                  commandOptions: '-input=false -var-file="../vars/production/production.tfvars"'
                  environmentServiceNameAzureRM: 'tamopstf'
                  workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'
              
              - task: ArchiveFiles@2
                inputs:
                  rootFolderOrFile: '$(Build.SourcesDirectory)'
                  includeRootFolder: false
                  archiveType: 'tar'
                  tarCompression: 'gz'
                  archiveFile: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).tgz'
                  replaceExistingArchive: true
                  displayName: 'Create Plan Artifact'

              - task: PublishBuildArtifacts@1
                inputs:
                  PathtoPublish: '$(Build.ArtifactStagingDirectory)'
                  ArtifactName: '$(Build.BuildId)-tfplan'
                  publishLocation: 'Container'
                  displayName: 'Publish Plan Artifact'    

              - task: DeleteFiles@1
                displayName: 'Remove unneeded files'
                inputs:
                  contents: |
                    .terraform
                    tfplan

  - stage: terraform_apply
    dependsOn: [terraform_plan]
    condition: succeeded('terraform_plan')
    jobs:
      - job: terraform_apply
        steps:
              - checkout: none

              - task: DownloadBuildArtifacts@0
                inputs:
                  artifactName: '$(Build.BuildId)-tfplan'
                  displayName: 'Download Plan Artifact'

              - task: ExtractFiles@1
                inputs:
                  archiveFilePatterns: '$(System.ArtifactsDirectory)/$(Build.BuildId)-tfplan/$(Build.BuildId).tgz'
                  destinationFolder: '$(System.DefaultWorkingDirectory)/'
                  cleanDestinationFolder: false
                  displayName: 'Extract Terraform Plan Artifact'

              - task: TerraformInstaller@0
                displayName: 'install'
                inputs:
                  terraformVersion: '0.13.4'

              - task: TerraformTaskV1@0
                displayName: 'init'
                inputs:
                  provider: 'azurerm'
                  command: 'init'
                  backendServiceArm: 'tamopstf'
                  backendAzureRmResourceGroupName: 'tamopstfstates'
                  backendAzureRmStorageAccountName: 'tfstatedevops'
                  backendAzureRmContainerName: 'azureterraformbuildartifacts'
                  backendAzureRmKey: 'terraform.tfstate'
                  workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'

              - task: TerraformTaskV1@0
                displayName: 'plan'
                inputs:
                  provider: 'azurerm'
                  command: 'plan'
                  commandOptions: '-input=false -var-file="../vars/production/production.tfvars"'
                  environmentServiceNameAzureRM: 'tamopstf'
                  workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'

              - task: TerraformTaskV1@0
                displayName: 'apply'
                inputs:
                  provider: 'azurerm'
                  command: 'apply'
                  commandOptions: '-input=false -auto-approve -var-file="../vars/production/production.tfvars"'
                  environmentServiceNameAzureRM: 'tamopstf'
                  workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'

Reviewing the Pipeline in Azure DevOps

Notice the reference to artifact at terraform_plan stage?

We can download the artifact to review, I have downloaded the artifact and screenshot taken below:-

Notice the same folder structure as found in the Azure Repo? Along with additional plan-*.json. Awesome!

If we review the steps in terraform_apply stage, we can see Checkout was ignored

Awesome, I have deployed terraform using the artifact created during the terraform_plan stage

Github Repo here containing required Terraform configuration & Pipelines to deploy the above

2 comments

    1. Hi Lakshay, really happy to hear that feedback! Glad you enjoyed it. Feel free to subscribe to my blog and check out my other posts

      Thanks

      Thomas

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s