Storing and retrieving secrets in Azure KeyVault with Variable Groups in Azure DevOps Pipelines

Using Azure DevOps Pipelines and want to store secrets in Azure KeyVault? Use a variable group with Key Vault integration to retrieve these secrets and use within your DevOps pipeline

Why Azure KeyVault?

Using a variable group, you can store “secrets” but it is stored directly within Azure DevOps, I much prefer to store secrets within Azure KeyVaults – for a number of reasons:-

• Ability to rotate secrets in one place within a Key Vault – you may have a number of secrets in the same Key Vault
• Activity log to see who/when secrets were updated or removed
• Centrailisation to store more than just several variable group secrets
• No need for repetition to store the secret in both Azure DevOps & Key Vault
• With Azure KeyV ault you can monitor when secrets were accessed

What are variable groups in Azure DevOps?

I really do like variable groups; they are awesome! Use a variable group to store any values that you want to control and possibly made available across multiple pipelines. You can even use variable groups to store secrets and other values that might need to be passed into the YAML pipeline.

Read more here about variable groups and their usage 

Initial Overview

I have a Key Vault: tamops-kv created already; with a number secrets created

Time to implement!

As mentioned above, I have an Azure Key Vault tamops-kv already created with a number of secrets created

I will now create a variable group within Azure DevOps called variable-group-keyvault

Pipelines -> Library -> Variable Groups -> + Variable Group

As below, create variable group and reference the azure subscription/service connection to where tamops-kv is located

I can then select, which variables to be created within my variable group

• variable name will be the name of each of the secrets selected

• username
• password
• tenantId

To reference a variable group & its associated variables within a pipeline – you reference as below

I will create a simple Bash@3 task to run az login –service-principal and use the variables from Key Vault to log in!

This is just for a test purpose – there is other task types I would use if I wanted to run az commands

Example of the Bash@thomasops

      - task: Bash@3
        displayName: 'az login and group list'
        inputs:
          targetType: inline
          script: |
            az login --service-principal --username  $(username) --password $(password) --tenant $(tenantId)
            az group list

Notice the reference to username, password & tenant? These are populated at run time with the variables from the specified variable group!

Full pipeline

name: "Azure DevOps Pipeline Variable Group KeyVault example"

variables:
  - group: variable-group-keyvault
  
# Don't run against PRs
pr: none

stages :
  - stage: az_group_list
    jobs:
    - job: az_group_list
      continueOnError: false
      steps:
      - task: Bash@3
        displayName: 'az login and group list'
        inputs:
          targetType: inline
          script: |
            az login --service-principal --username  $(username) --password $(password) --tenant $(tenantId)
            az group list

Time to run the pipeline!

Notice there is a task – Download secrets: tamops-kv, reviewing this – we can see the secrets are downloaded and ready to be used within the pipeline

Starting: Download secrets: tamops-kv
==============================================================================
Task         : Azure Key Vault
Description  : Download Azure Key Vault secrets
Version      : 1.182.1
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-key-vault
==============================================================================
SubscriptionId: XXXXXXXXXXXX.
Key vault name: tamops-kv.
Downloading secret value for: password.
Downloading secret value for: tenantId.
Downloading secret value for: username.
Finishing: Download secrets: tamops-kv

Checking the task az login and group list

It is logging into az cli using the specified service principal credentials successfully

[
  {
    "cloudName": "AzureCloud",
    "homeTenantId": "***",
    "id": "XXXXXXXXXXXX",
    "isDefault": true,
    "managedByTenants": [],
    "name": "thomasthorntoncloud",
    "state": "Enabled",
    "tenantId": "***",
    "user": {
      "name": "***",
      "type": "servicePrincipal"
    }
]

Followed by retrieving all resource groups successfully!

  {
    "id": "/subscriptions/XXXXXXXXXXXX/resourceGroups/thomasthorntoncloud1",
    "location": "uksouth",
    "managedBy": null,
    "name": "thomasthorntoncloud1",
    "properties": {
      "provisioningState": "Succeeded"
    },
    "tags": {},
    "type": "Microsoft.Resources/resourceGroups"
  },
  {
    "id": "/subscriptions/XXXXXXXXXXXX/resourceGroups/thomasthorntoncloud2",
    "location": "uksouth",
    "managedBy": null,
    "name": "thomasthorntoncloud2",
    "properties": {
      "provisioningState": "Succeeded"
    },
    "tags": {},
    "type": "Microsoft.Resources/resourceGroups"
  },

The end!

Hopefully you have enjoyed this blog and gained an insight into “Storing and retrieving secrets in Azure KeyVault with Variable Groups in Azure DevOps Pipelines”!

I hope this assists you and show how you can integrate variable groups into your pipelines!

Got any queries? As always – give me a shout!

GitHub repository