Thomas Thornton

Microsoft Azure Blog – Microsoft MVP

Using Azure Container Apps and KEDA to create self-hosted scalable Azure DevOps Agents deployed using Azure DevOps and Azure CLI

on by 8 Comments

You may have the requirement of creating self-hosted agents in Azure DevOps, there are multiple ways of doing this including Virtual Machine, Virtual Machine Scaleset & various Container-type approaches. In this blog I am going to to be using Azure Container Apps and KEDA to create self-hosted scalable Azure DevOps Agents deployed using Azure DevOps and Azure CLI.

In a previous blog post; I blogged Deploy to Azure Container App from Azure Container Registry using a CI/CD Azure DevOps Pipeline and Azure CLI – the setup for this post will be quite similar.

Not heard of Container apps? Watch this awesome video to see them in action!

How to Build and Deliver Apps Fast and Scalable with Azure Container Apps

Why a container for the self-hosted DevOps Agent?

A question that appears quite often; a container has so many benefits including:

  • No need to worry about the underlying host machine (patching/updating Windows/Linux – not required)
  • Usually a lot more cost effective
  • Deploy with ease; whether locally, Container Instance, Kubernetes cluster, Container App etc
  • Ability to deploy anywhere pretty much
  • Flexibility and scalability

Only a few advantages of why to use a container

Azure Container Apps and Scaling

As documented in the Azure documentation for Azure Container Apps; they can dynamically scale based on the following characteristics:

KEDA

Lets look at the feature we will be using in this blog post to scale the Azure DevOps agents, KEDA!

Current available KEDA scalers are listed here . We Will be using Azure Pipelines – its awesome! How does it scale you may be asking?

It scales based on the amount of pipeline runs pending in a given agent pool.

KEDA stands for Kubernetes Event-driven Autoscaler. It is built to be able to activate a Kubernetes deployment (i.e. no pods to a single pod) and subsequently to more pods based on events from various event sources. KEDA is a single-purpose and lightweight component that can be added into any Kubernetes cluster. KEDA works alongside standard Kubernetes components like the Horizontal Pod Autoscaler and can extend functionality without overwriting or duplication. With KEDA you can explicitly map the apps you want to use event-driven scale, with other apps continuing to function. This makes KEDA a flexible and safe option to run alongside any number of any other Kubernetes applications or frameworks.

https://keda.sh/docs/2.6/concepts/

Please note, this is a Proof-Of-Concept

This concept is cool ; but after further reading its not supported as such  Currently its a PoC concept and maybe never a complete reality. I was going with a Container Instance approach and then thought i’d try with Container Apps as they can use KEDA.

Due to the KEDA not having ScaledJobs – the agents can be removed at any time, I didn’t notice this during my testing/creation of running jobs (must have been lucky). Some discussion here if you are interested https://twitter.com/JorgeArteiro/status/1494324269371002882 

Time to Deploy

Pre-reqs

Lets start with the pre-reqs:

  • Variable group azure-container-app to be created

Check out this blog regarding Referencing Variable Groups in Azure DevOps Pipeline Templates and how to set one one

  • PAT Token is required , this will be used by the container to access Azure DevOps pipelines etc.

Documented here & save PAT token as pattoken in the created Variable Group azure-container-app

The Pipeline

Very similar to my last blog post “Deploy to Azure Container App from Azure Container Registry using a CI/CD Azure DevOps Pipeline and Azure CLI” the pipeline will be in 4 stages:

  • create_acr will create resource group , container registry to where the sample application will be uploaded to and also log analytics workspace that the container environment will connect to
  • build will build the sample application and push to the Container Registry created in create_acr stage (build will build Azure DevOps docker image, reference to docker files here)
  • create_container_environment Will create the container environment to host the container app that will be deployed in the final stage
  • deploy_container_app will deploy the container app from image created in build stage

Please follow that blog on the initial setup of the first 3 stages. The fourth stage deploy_container_app is slightly different & I will document this below:

Prior to the final stage of the pipeline, we need to create a new agent pool. This is where the container app agents will be situated

Select Agent Pools within Organisation settings

Add pool similar to the below, I will be using containerapp-pool

Now that the pool is created, select the pool in browser and make note of URL as below:

https://dev.azure.com/TerraformTraining/TamOpsTerraform/_settings/agentqueues?queueId=91&view=jobs

Make note of queueId, mine will be 91

Looking at the final stage

The final stage below deploys the Azure Container app as a group deployment with a template file.

The template file containerapp.json below will be referenced:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-08-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "location": {
            "defaultValue": "northeurope",
            "type": "String"
        },
        "environment_name": {
            "type": "String"
        },
        "azp_url": {
            "type": "securestring"
        },
        "azp_token": {
            "type": "securestring"
        },
        "azp_pool": {
            "type": "String"
        },
        "azp_poolId": {
            "type": "String"
        },
        "REGISTRY_LOGIN_SERVER": {
            "type": "String"
        },
        "REGISTRY_USERNAME": {
            "type": "String"
        },
        "REGISTRY_PASSWORD": {
            "type": "securestring"
        },
        "IMAGE": {
            "type": "String"
        },
        "MIN_REPLICAS": {
            "type": "String"
        },
        "MAX_REPLICAS": {
            "type": "String"
        },
        "CPU": {
            "type": "String"
        },
        "MEMORY": {
            "type": "String"
        },
        "CONTAINER_APP_NAME": {
            "type": "String"
        }
    },
    "variables": {},
    "resources": [
        {
            "name": "azuredevops-agent",
            "type": "Microsoft.Web/containerApps",
            "apiVersion": "2021-03-01",
            "kind": "containerapp",
            "location": "[parameters('location')]",
            "properties": {
                "kubeEnvironmentId": "[resourceId('Microsoft.Web/kubeEnvironments', parameters('environment_name'))]",
                "configuration": {
                    "secrets": [
                        {
                            "name": "azp-url",
                            "value": "[parameters('azp_url')]"
                        },
                        {
                            "name": "azp-token",
                            "value": "[parameters('azp_token')]"
                        },
                        {
                            "name": "azp-pool",
                            "value": "[parameters('azp_pool')]"
                        },
                        {
                            "name": "azp-poolid",
                            "value": "[parameters('azp_poolId')]"
                        },
                        {
                            "name": "registry-secret",
                            "value": "[parameters('REGISTRY_PASSWORD')]"
                        }
                    ],
                    "registries": [
                        {
                            "server": "[parameters('REGISTRY_LOGIN_SERVER')]",
                            "username": "[parameters('REGISTRY_USERNAME')]",
                            "passwordSecretRef": "registry-secret"
                        }
                    ]
                },
                "template": {
                    "containers": [
                        {
                            "image": "[parameters('IMAGE')]",
                            "name": "basedockeragent",
                            "resources": {
                                "cpu": 1.75,
                                "memory": "3.5Gi"
                            },
                            "env": [
                                {
                                    "name": "AZP_URL",
                                    "secretRef": "azp-url"
                                },
                                {
                                    "name": "AZP_TOKEN",
                                    "secretRef": "azp-token"
                                },
                                {
                                    "name": "AZP_POOL",
                                    "secretRef": "azp-pool"
                                },
                                {
                                    "name": "AZP_POOLID",
                                    "secretRef": "azp-poolid"
                                }
                            ]
                        }
                    ],
                    "scale": {
                        "minReplicas": "[parameters('MIN_REPLICAS')]",
                        "maxReplicas": "[parameters('MAX_REPLICAS')]",
                        "rules": [
                            {
                                "name": "[parameters('CONTAINER_APP_NAME')]",
                                "custom": {
                                    "type": "azure-pipelines",
                                    "metadata": {
                                        "poolID": "[parameters('azp_poolId')]",
                                        "targetPipelinesQueueLength": "[parameters('MIN_REPLICAS')]"
                                    },
                                    "auth": [
                                        {
                                            "secretRef": "azp-token",
                                            "triggerParameter": "personalAccessToken"
                                        },
                                        {
                                            "secretRef": "azp-url",
                                            "triggerParameter": "organizationURL"
                                        }
                                    ]
                                }
                            }
                        ]
                    }
                }
            }
        }
    ]
}

Lets look at the variables the script will be using

  • RESOURCE_GROUP: will be resource group to where the container app will be deployed
  • LOG_ANALYTICS_WORKSPACE: Name of the log analytics workspace to which the container apps metrics will be sent to
  • LOCATION: Location of the container app
  • AZP_URL: Azure DevOps URL
  • AZP_TOKEN: PAT Token saved in variable group at start of this blog post
  • AZP_POOL: Azure DevOps Agent Pool to which the container apps will use
  • AZP_POOL_ID: queueID of the Pool mentioned above
  • REGISTRY_USERNAME: Azure Container Registry Username
  • REGISTRY_PASSWORD: Azure Container Registry Password (Stored in variable group)
  • REGISTRY_LOGIN_SERVER: Azure Container Registry Login Server
  • IMAGE: Image name that has been built in above stage
  • CONTAINER_APP_NAME: Name of container app
  • MIN_REPLICAS: Minimum number of container apps to be deployed
  • MAX_REPLICAS: Maximum number of container apps that can be scaled with KEDA
  • CPU: CPU of container apps
  • MEMORY: MEMORY of container apps
  - stage: deploy_container_app
    dependsOn: [create_container_environment]
    jobs:
      - job: "deploy_app"
        steps:
          - task: AzureCLI@2
            displayName: 'Deploy app to Container App'
            inputs:
              azureSubscription: 'thomasthorntoncloud'
              scriptType: bash
              scriptLocation: inlineScript
              addSpnToEnvironment: true
              inlineScript: |
                #!/bin/bash
                az extension add \
                --source https://workerappscliextension.blob.core.windows.net/azure-cli-extension/containerapp-0.2.2-py2.py3-none-any.whl --yes

                az provider register --namespace Microsoft.Web

                RESOURCE_GROUP="azure-container-rg"
                LOG_ANALYTICS_WORKSPACE="tamopsacrcontainersla"
                CONTAINERAPPS_ENVIRONMENT="tamops-environment"
                LOCATION="westeurope"
                AZP_URL="https://dev.azure.com/TerraformTraining"
                AZP_TOKEN=$(pattoken)
                AZP_AGENT_NAME=devopsagentnew
                AZP_POOL=containerapp-pool
                AZP_POOL_ID=91
                REGISTRY_USERNAME=tamopsacrcontainers
                REGISTRY_PASSWORD=$(acrpassword)
                REGISTRY_LOGIN_SERVER=tamopsacrcontainers.azurecr.io
                IMAGE=tamopsacrcontainers.azurecr.io/adoagent:'$(Build.BuildId)'
                
                CONTAINER_APP_NAME="azuredevops-agent"
                MIN_REPLICAS="1"
                MAX_REPLICAS="3"
                CPU="1.75"
                MEMORY="3.5Gi"

                  az deployment group create \
                    --resource-group $RESOURCE_GROUP \
                    --template-file "containerapp/containerapp.json" \
                    --parameters \
                        environment_name="$CONTAINERAPPS_ENVIRONMENT" \
                        location="$LOCATION" \
                        azp_url="$AZP_URL" \
                        azp_token="$AZP_TOKEN" \
                        azp_pool="$AZP_POOL" \
                        azp_poolId="$AZP_POOL_ID" \
                        REGISTRY_LOGIN_SERVER=$REGISTRY_LOGIN_SERVER \
                        REGISTRY_USERNAME=$REGISTRY_USERNAME \
                        REGISTRY_PASSWORD=$REGISTRY_PASSWORD \
                        IMAGE=$IMAGE \
                        MIN_REPLICAS=$MIN_REPLICAS \
                        MAX_REPLICAS=$MAX_REPLICAS \
                        CPU=$CPU \
                        MEMORY=$MEMORY \
                        CONTAINER_APP_NAME=$CONTAINER_APP_NAME

Now lets run the pipeline for the full 4 stages!


Reviewing Azure Portal, we can see the resources have been deployed successfully

Reviewing the agent pool, we can see an active Azure DevOps agent that has been created with the pipeline & Azure Container app

Running some pipelines, we can see it scaling, notice some offline? When they run the completed job they restart and a new container app appears – this is setup in the KEDA configuration. Further details on this below

Awesome! We are scaling up and down, how is this determined? Lets look at KEDA documentation. It scales using:

targetPipelinesQueueLength – Target value for the amount of pending jobs in the queue to scale on. (Default: 1, Optional)

Example – If one pod can handle 10 jobs, set the queue length target to 10. If the actual number of jobs in the queue is 30, the scaler scales to 3 pods.

Feel free to review the KEDA documentation further in terms of parameter list etc, its pretty awesome 🙂

Code used in the above, including full pipeline

A recommended Microsoft Learn module to further your knowledge with KEDA – Scale container applications in Azure Kubernetes Services using KEDA

I look forward to seeing Container Apps develop further – they are awesome!!

8 comments

    1. Hi Sebastian,

      Great blog! Just followed you 🙂

      This concept is cool ; but after further reading its not supported as such 😦 Currently its a PoC concept and maybe never a complete reality. I was going with a Container Instance approach and then thought i’d try with Container Apps as they can use KEDA.

      Due to the KEDA not having ScaledJobs – the agents can be removed at any time, I didn’t notice this during my testing/creation of running jobs (must have been lucky). Some discussion here if you are interested https://twitter.com/JorgeArteiro/status/1494324269371002882 – did you notice it much?

      My next post will be similar but for Container Instance 🙂

      Thanks

      Thomas

      Reply
  2. Pingback: Top Stories from the Microsoft DevOps Community – 2022.02.18 - IT Skills You Need
  3. Pingback: Top Stories from the Microsoft DevOps Community – 2022.02.18 - Azure DevOps Blog

  5. Hi Thomas, thanks for this it is really helpful.
    By the way, where did you setup the KEDA part? I can’t see it and from my end it doesn’t restart the containers whether the job fails or succeeds.
    Thank you, looking forward for your answers.

    Reply

    1. Hi Jude,

      This was only ever a proof of concept and functionality of KEDA not fully supported! I recommend AKS with KEDA if you require

      Thanks

      Thomas

      Reply

Leave a Reply

Fill in your details below or click an icon to log in:

Gravatar
WordPress.com Logo

You are commenting using your WordPress.com account. ( Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. ( Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. ( Log Out /  Change )

Cancel

Connecting to %s