Creating Azure Kubernetes Service with Application Gateway Ingress using Terraform and deploying a sample app

In this blog post I am going to show how you can deploy Azure Kubernetes Service (AKS) with Application Gateway Ingress using Terraform; this include Virtual Network, Log Analytics and Azure Kubernetes Service, once created – will show how to deploy a sample application into the newly created AKS cluster

What is Azure Kubernetes Service (AKS)?

Azure Kubernetes Service (AKS) is a managed resource in Azure where when you deploy your Kubernetes cluster; Azure looks after the “hard-lifting” of the cluster, such as maintenance and health monitoring and even scaling! If you are familiar with Kubernetes, you may have heard of Kubernetes master and agent nodes, Azure looks after the master nodes – while you will look after the agent nodes!

What will be deployed using Terraform?

Azure Virtual Network (vNET)

  • Azure Virtual Network
  • Subnet: aks
  • Subnet: appgw

Log Analytics

  • Log Analytics Workspace
  • Log Analytics ContainerInsights solution

Azure Kubernetes Service (AKS)

  • Azure Kubernetes Service
  • Linux Nodepool
  • Azure Application Gateway Ingress Controller
  • Azure Active Directory integration for AKS control
  • SystemAssigned Managed Identity
  • Assign role assignment of Managed Identity to AKS nodepool resource group

Setup Storage Account for terraform remote state

In this blog post, I will be storing the Terraform state in remote Storage account for each of the Azure resource that I’ve mentioned above.

Using Azure CLI to create the Storage Account

The script will create

  • Azure Resource Group
  • Azure Storage Account
  • Azure Blob storage location within Azure Storage Account
#!/bin/sh

RESOURCE_GROUP_NAME="tamopsterraform-rg"
STORAGE_ACCOUNT_NAME="tamopsterraform"

# Create Resource Group
az group create -l uksouth -n $RESOURCE_GROUP_NAME

# Create Storage Account
az storage account create -n $STORAGE_ACCOUNT_NAME -g $RESOURCE_GROUP_NAME -l uksouth --sku Standard_LRS

# Create Storage Account blob
az storage container create  --name tfstate --account-name $STORAGE_ACCOUNT_NAME

Time to deploy Terraform

GitHub Repository for code used in this blog post

The Terraform will be deployed in the following order

  1. vNET
  2. Log Analytics
  3. AKS

Why the order? The terraform created for AKS has dependencies on 1 & 2

Each folder is setup as below

aks
 └──main.tf
 └──variables.tf
 └──terraform.tfvars

To deploy each, you will need to be in the folders, such as aks & run the following

  • terraform init
  • terraform plan (to review what is going to be deployed)
  • terraform apply

Azure Virtual Network (vNET) Terraform

main.tf

terraform {
  backend "azurerm" {
    resource_group_name  = "tamopsterraform-rg"
    storage_account_name = "tamopsterraform"
    container_name       = "tfstate"
    key                  = "vnet-terraform.tfstate"
  }
}

provider "azurerm" {
    version = "~> 2.0"
    features {}
}

resource "azurerm_resource_group" "resource_group" {
  name     = "${var.name}-rg"
  location = var.location
}

resource "azurerm_virtual_network" "virtual_network" {
  name =  "${var.name}-vnet"
  location = var.location
  resource_group_name = azurerm_resource_group.resource_group.name
  address_space = [var.network_address_space]
}

resource "azurerm_subnet" "aks_subnet" {
  name = var.aks_subnet_address_name
  resource_group_name  = azurerm_resource_group.resource_group.name
  virtual_network_name = azurerm_virtual_network.virtual_network.name
  address_prefixes = [var.aks_subnet_address_prefix]
}

resource "azurerm_subnet" "app_gwsubnet" {
  name = var.subnet_address_name
  resource_group_name  = azurerm_resource_group.resource_group.name
  virtual_network_name = azurerm_virtual_network.virtual_network.name
  address_prefixes = [var.subnet_address_prefix]
}

variables.tf

variable "name" {
  type        = string
  default     = "tamops"
  description = "Name for resources"
}

variable "location" {
  type        = string
  default     = "uksouth"
  description = "Azure Location of resources"
}

variable "network_address_space" {
  type        = string
  description = "Azure VNET Address Space"
}

variable "aks_subnet_address_name" {
  type        = string
  description = "AKS Subnet Address Name"
}

variable "aks_subnet_address_prefix" {
  type        = string
  description = "AKS Subnet Address Space"
}

variable "subnet_address_name" {
  type        = string
  description = "Subnet Address Name"
}

variable "subnet_address_prefix" {
  type        = string
  description = "Subnet Address Space"
}

terraform.tfvars

name     = "devopsthehardway"
location = "uksouth"
network_address_space = "192.168.0.0/16"
aks_subnet_address_name = "aks"
aks_subnet_address_prefix = "192.168.0.0/24"
subnet_address_name = "appgw"
subnet_address_prefix = "192.168.1.0/24"

Log Analytics Terraform

main.tf

terraform {
  backend "azurerm" {
    resource_group_name  = "tamopsterraform-rg"
    storage_account_name = "tamopsterraform"
    container_name       = "tfstate"
    key                  = "la-terraform.tfstate"
  }
}

provider "azurerm" {
    version = "~> 2.0"
    features {}
}

data "azurerm_resource_group" "resource_group" {
  name     = "${var.name}-rg"
}

resource "azurerm_log_analytics_workspace" "Log_Analytics_WorkSpace" {
    # The WorkSpace name has to be unique across the whole of azure, not just the current subscription/tenant.
    name                = "${var.name}-la"
    location            = var.location
    resource_group_name = data.azurerm_resource_group.resource_group.name
    sku                 = "PerGB2018"
}

resource "azurerm_log_analytics_solution" "Log_Analytics_Solution_ContainerInsights" {
    solution_name         = "ContainerInsights"
    location              = azurerm_log_analytics_workspace.Log_Analytics_WorkSpace.location
    resource_group_name   = data.azurerm_resource_group.resource_group.name
    workspace_resource_id = azurerm_log_analytics_workspace.Log_Analytics_WorkSpace.id
    workspace_name        = azurerm_log_analytics_workspace.Log_Analytics_WorkSpace.name

    plan {
        publisher = "Microsoft"
        product   = "OMSGallery/ContainerInsights"
    }
}

variables.tf

variable "name" {
  type        = string
  default     = "devopsthehardway"
  description = "Name for resources"
}

variable "location" {
  type        = string
  default     = "uksouth"
  description = "Azure Location of resources"
}

terraform.tfvars

name     = "tamops"
location = "uksouth"

Azure Kubernetes Service (AKS) Terraform

main.tf

terraform {
  backend "azurerm" {
    resource_group_name  = "tamopsterraform-rg"
    storage_account_name = "tamopsterraform"
    container_name       = "tfstate"
    key                  = "aks-terraform.tfstate"
  }
}

provider "azurerm" {
  version = "~> 2.0"
  features {}
}

data "azurerm_resource_group" "resource_group" {
  name = "${var.name}-rg"
}

data "azurerm_subnet" "akssubnet" {
  name                 = "aks"
  virtual_network_name = "${var.name}-vnet"
  resource_group_name  = data.azurerm_resource_group.resource_group.name
}

data "azurerm_subnet" "appgwsubnet" {
  name                 = "appgw"
  virtual_network_name = "${var.name}-vnet"
  resource_group_name  = data.azurerm_resource_group.resource_group.name
}

data "azurerm_log_analytics_workspace" "workspace" {
  name                = "${var.name}-la"
  resource_group_name = data.azurerm_resource_group.resource_group.name
}

resource "azurerm_kubernetes_cluster" "k8s" {
  name                = "${var.name}aks"
  location            = var.location
  resource_group_name = data.azurerm_resource_group.resource_group.name
  dns_prefix          = "${var.name}dns"
  kubernetes_version  = var.kubernetes_version

  node_resource_group = "${var.name}-node-rg"

  linux_profile {
    admin_username = "ubuntu"

    ssh_key {
      key_data = var.ssh_public_key
    }
  }

  default_node_pool {
    name                 = "agentpool"
    node_count           = var.agent_count
    vm_size              = var.vm_size
    vnet_subnet_id       = data.azurerm_subnet.akssubnet.id
    type                 = "VirtualMachineScaleSets"
    orchestrator_version = var.kubernetes_version
  }

  identity {
    type = "SystemAssigned"
  }

  addon_profile {
    oms_agent {
      enabled                    = var.addons.oms_agent
      log_analytics_workspace_id = data.azurerm_log_analytics_workspace.workspace.id
    }

    ingress_application_gateway {
      enabled   = var.addons.ingress_application_gateway
      subnet_id = data.azurerm_subnet.appgwsubnet.id
    }

  }

  network_profile {
    load_balancer_sku = "standard"
    network_plugin    = "azure"
  }

  role_based_access_control {
    enabled = var.kubernetes_cluster_rbac_enabled

    azure_active_directory {
      managed                = true
      admin_group_object_ids = [var.aks_admins_group_object_id]
    }
  }

}

data "azurerm_resource_group" "node_resource_group" {
  name = azurerm_kubernetes_cluster.k8s.node_resource_group
  depends_on = [
    azurerm_kubernetes_cluster.k8s
  ]
}

resource "azurerm_role_assignment" "node_infrastructure_update_scale_set" {
  principal_id         = azurerm_kubernetes_cluster.k8s.kubelet_identity[0].object_id
  scope                = data.azurerm_resource_group.node_resource_group.id
  role_definition_name = "Virtual Machine Contributor"
  depends_on = [
    azurerm_kubernetes_cluster.k8s
  ]
}

variables.tf

variable "name" {
  type        = string
  default     = "tamops"
  description = "Name for resources"
}

variable "location" {
  type        = string
  default     = "uksouth"
  description = "Azure Location of resources"
}

variable "addons" {
  description = "Defines which addons will be activated."
  type = object({
    oms_agent                   = bool
    ingress_application_gateway = bool
  })
}

variable "kubernetes_cluster_rbac_enabled" {
  default = "true"
}

variable "kubernetes_version" {
}

variable "agent_count" {
}

variable "vm_size" {
}

variable "ssh_public_key" {
}

variable "aks_admins_group_object_id" {
}

terraform.tfvars

  • Update ssh_public_key to your own ssh public key
  • Update aks_admins_group_object_id to an Azure AD group that you will use as “AKS admins”
name     = "tamops"
location = "uksouth"

kubernetes_version         = "1.19.11"
agent_count                = 3
vm_size                    = "Standard_DS2_v2"
ssh_public_key             = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrt/GYkYpuQYRxM3lgjOr3Wqx8g5nQIbrg6Mr53wZGb35+ft+PibDMqxXZ7xq7fC3YuLnnO022IPgEjkF9fP03ZmfUeLjJJvw8YcutN9DD/2cx93BpKFPNUsqEB+za1iJ16kMsCojy35c1R64O+rw20D6iP96rmDAyIc5FR03y00eyAzQ8vo7/u9+VPwpdGEI7QCokZROcj6iNVz1V/1t6G4AEufPLokdj8J0gla/dN+tvnSLRQVBTDiD4jmVGImpWFqqKaH6R9SSXmRzj0uhvJUmSiZAZCb1caPEYgPEvNITuGQFdykPoY/4Z/3B+x/ipEQbWy8yL7bDFSXZTYhVKlPVyPbUtN5QFt7QtCtg84xDAZ6GA6AnONTtMxX2jvdzB9yh1ZsteNrOZ/Jo3ecuie573syQfG23Tu6qTqak8O7ZTOLY9iPx2ego3KvTWH/Q3lIvjnlpfCQtFtSgkNxjalMBk+NwwEgZHWRREOHwJmQIKVN0gSitN1KXobrqwxNk= tamops@Synth"
aks_admins_group_object_id = "e97b6454-3fa1-499e-8e5c-5d631e9ca4d1"

addons = {
  oms_agent                   = true
  ingress_application_gateway = true
}

Deploying a test application to your AKS cluster

Deploy each of the above, as mentioned in order!

You can navigate to the newly created AKS cluster

Once deployed you are ready to begin deploying your applications to Kubernetes 🙂

Get the context of your newly created AKS cluster

az aks get-credentials --resource-group tamops-rg --name tamopsaks

Merged "tamopsaks" as current context in /Users/thomasthorntoncloud/.kube/config

Lets deploy a test application your AKS cluster, we will use azure-voting-app and i’ve added Application Gateway ingress as highlighted below

apiVersion: apps/v1
kind: Deployment
metadata:
  name: azure-vote-back
spec:
  replicas: 1
  selector:
    matchLabels:
      app: azure-vote-back
  template:
    metadata:
      labels:
        app: azure-vote-back
    spec:
      nodeSelector:
        "kubernetes.io/os": linux
      containers:
      - name: azure-vote-back
        image: mcr.microsoft.com/oss/bitnami/redis:6.0.8
        env:
        - name: ALLOW_EMPTY_PASSWORD
          value: "yes"
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 250m
            memory: 256Mi
        ports:
        - containerPort: 6379
          name: redis
---
apiVersion: v1
kind: Service
metadata:
  name: azure-vote-back
spec:
  ports:
  - port: 6379
  selector:
    app: azure-vote-back
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: azure-vote-front
spec:
  replicas: 1
  selector:
    matchLabels:
      app: azure-vote-front
  template:
    metadata:
      labels:
        app: azure-vote-front
    spec:
      nodeSelector:
        "kubernetes.io/os": linux
      containers:
      - name: azure-vote-front
        image: mcr.microsoft.com/azuredocs/azure-vote-front:v1
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 250m
            memory: 256Mi
        ports:
        - containerPort: 80
        env:
        - name: REDIS
          value: "azure-vote-back"
---
apiVersion: v1
kind: Service
metadata:
  name: azure-vote-front
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: azure-vote-front
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: azure-vote-front
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
spec:
  rules:
    - http:
        paths:
          - path: /
            backend:
              serviceName: azure-vote-front
              servicePort: 80

Navigate to the file deployment.yaml and run the following

kubectl apply -f deployment.yaml 

With a successful output:-

deployment.apps/azure-vote-back created
service/azure-vote-back created
deployment.apps/azure-vote-front created
service/azure-vote-front created

Now lets get the ingress IP , taken from the below

thomasthorntoncloud@Thomass-MBP scripts % kubectl get ingress
NAME               CLASS    HOSTS   ADDRESS        PORTS   AGE
azure-vote-front   <none>   *       20.90.224.49   80      50s

Accessing the address above on http:// will load the azure-vote-frontend 🙂

Awesome! You have now successfully deployed the required AKS Terraform and deployed a test application to the AKS Cluster!

In the coming blogs, I will go further into this journey, be sure to check them out!

4 comments

  1. great content as i am in the way of deploying AGIC. my first though was creating AGIC and bind it with terrafrom, turn out it can be call out directly from aks deployment

    1. Thank you for the message, yep I’d definitely bind using aks deployment rather than deploying AGIC separately

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s