Creating Azure Kubernetes Service with Application Gateway Ingress using Terraform and deploying a sample app

In this blog post I am going to show how you can deploy Azure Kubernetes Service (AKS) with Application Gateway Ingress using Terraform; this include Virtual Network, Log Analytics and Azure Kubernetes Service, once created – will show how to deploy a sample application into the newly created AKS cluster

What is Azure Kubernetes Service (AKS)?

Azure Kubernetes Service (AKS) is a managed resource in Azure where when you deploy your Kubernetes cluster; Azure looks after the “hard-lifting” of the cluster, such as maintenance and health monitoring and even scaling! If you are familiar with Kubernetes, you may have heard of Kubernetes master and agent nodes, Azure looks after the master nodes – while you will look after the agent nodes!

What will be deployed using Terraform?

Azure Virtual Network (vNET)

Azure Virtual Network
Subnet: aks
Subnet: appgw

Log Analytics

Log Analytics Workspace
Log Analytics ContainerInsights solution

Azure Kubernetes Service (AKS)

Azure Kubernetes Service
Linux Nodepool
Azure Application Gateway Ingress Controller
Azure Active Directory integration for AKS control
SystemAssigned Managed Identity
Assign role assignment of Managed Identity to AKS nodepool resource group

Setup Storage Account for terraform remote state

In this blog post, I will be storing the Terraform state in remote Storage account for each of the Azure resource that I’ve mentioned above.

Using Azure CLI to create the Storage Account

The script will create

Azure Resource Group
Azure Storage Account
Azure Blob storage location within Azure Storage Account

#!/bin/sh

RESOURCE_GROUP_NAME="tamopsterraform-rg"
STORAGE_ACCOUNT_NAME="tamopsterraform"

# Create Resource Group
az group create -l uksouth -n $RESOURCE_GROUP_NAME

# Create Storage Account
az storage account create -n $STORAGE_ACCOUNT_NAME -g $RESOURCE_GROUP_NAME -l uksouth --sku Standard_LRS

# Create Storage Account blob
az storage container create  --name tfstate --account-name $STORAGE_ACCOUNT_NAME

Time to deploy Terraform

GitHub Repository for code used in this blog post

The Terraform will be deployed in the following order

Why the order? The terraform created for AKS has dependencies on 1 & 2

Each folder is setup as below

aks
 └──main.tf
 └──variables.tf
 └──terraform.tfvars

To deploy each, you will need to be in the folders, such as aks & run the following

terraform init
terraform plan (to review what is going to be deployed)
terraform apply

Azure Virtual Network (vNET) Terraform

main.tf

terraform {
  backend "azurerm" {
    resource_group_name  = "tamopsterraform-rg"
    storage_account_name = "tamopsterraform"
    container_name       = "tfstate"
    key                  = "vnet-terraform.tfstate"
  }
}

provider "azurerm" {
    version = "~> 2.0"
    features {}
}

resource "azurerm_resource_group" "resource_group" {
  name     = "${var.name}-rg"
  location = var.location
}

resource "azurerm_virtual_network" "virtual_network" {
  name =  "${var.name}-vnet"
  location = var.location
  resource_group_name = azurerm_resource_group.resource_group.name
  address_space = [var.network_address_space]
}

resource "azurerm_subnet" "aks_subnet" {
  name = var.aks_subnet_address_name
  resource_group_name  = azurerm_resource_group.resource_group.name
  virtual_network_name = azurerm_virtual_network.virtual_network.name
  address_prefixes = [var.aks_subnet_address_prefix]
}

resource "azurerm_subnet" "app_gwsubnet" {
  name = var.subnet_address_name
  resource_group_name  = azurerm_resource_group.resource_group.name
  virtual_network_name = azurerm_virtual_network.virtual_network.name
  address_prefixes = [var.subnet_address_prefix]
}

variables.tf

variable "name" {
  type        = string
  default     = "tamops"
  description = "Name for resources"
}

variable "location" {
  type        = string
  default     = "uksouth"
  description = "Azure Location of resources"
}

variable "network_address_space" {
  type        = string
  description = "Azure VNET Address Space"
}

variable "aks_subnet_address_name" {
  type        = string
  description = "AKS Subnet Address Name"
}

variable "aks_subnet_address_prefix" {
  type        = string
  description = "AKS Subnet Address Space"
}

variable "subnet_address_name" {
  type        = string
  description = "Subnet Address Name"
}

variable "subnet_address_prefix" {
  type        = string
  description = "Subnet Address Space"
}

terraform.tfvars

name     = "devopsthehardway"
location = "uksouth"
network_address_space = "192.168.0.0/16"
aks_subnet_address_name = "aks"
aks_subnet_address_prefix = "192.168.0.0/24"
subnet_address_name = "appgw"
subnet_address_prefix = "192.168.1.0/24"

Log Analytics Terraform

main.tf

terraform {
  backend "azurerm" {
    resource_group_name  = "tamopsterraform-rg"
    storage_account_name = "tamopsterraform"
    container_name       = "tfstate"
    key                  = "la-terraform.tfstate"
  }
}

provider "azurerm" {
    version = "~> 2.0"
    features {}
}

data "azurerm_resource_group" "resource_group" {
  name     = "${var.name}-rg"
}

resource "azurerm_log_analytics_workspace" "Log_Analytics_WorkSpace" {
    # The WorkSpace name has to be unique across the whole of azure, not just the current subscription/tenant.
    name                = "${var.name}-la"
    location            = var.location
    resource_group_name = data.azurerm_resource_group.resource_group.name
    sku                 = "PerGB2018"
}

resource "azurerm_log_analytics_solution" "Log_Analytics_Solution_ContainerInsights" {
    solution_name         = "ContainerInsights"
    location              = azurerm_log_analytics_workspace.Log_Analytics_WorkSpace.location
    resource_group_name   = data.azurerm_resource_group.resource_group.name
    workspace_resource_id = azurerm_log_analytics_workspace.Log_Analytics_WorkSpace.id
    workspace_name        = azurerm_log_analytics_workspace.Log_Analytics_WorkSpace.name

    plan {
        publisher = "Microsoft"
        product   = "OMSGallery/ContainerInsights"
    }
}

variables.tf

variable "name" {
  type        = string
  default     = "devopsthehardway"
  description = "Name for resources"
}

variable "location" {
  type        = string
  default     = "uksouth"
  description = "Azure Location of resources"
}

terraform.tfvars

name     = "tamops"
location = "uksouth"

Azure Kubernetes Service (AKS) Terraform

main.tf

terraform {
  backend "azurerm" {
    resource_group_name  = "tamopsterraform-rg"
    storage_account_name = "tamopsterraform"
    container_name       = "tfstate"
    key                  = "aks-terraform.tfstate"
  }
}

provider "azurerm" {
  version = "~> 2.0"
  features {}
}

data "azurerm_resource_group" "resource_group" {
  name = "${var.name}-rg"
}

data "azurerm_subnet" "akssubnet" {
  name                 = "aks"
  virtual_network_name = "${var.name}-vnet"
  resource_group_name  = data.azurerm_resource_group.resource_group.name
}

data "azurerm_subnet" "appgwsubnet" {
  name                 = "appgw"
  virtual_network_name = "${var.name}-vnet"
  resource_group_name  = data.azurerm_resource_group.resource_group.name
}

data "azurerm_log_analytics_workspace" "workspace" {
  name                = "${var.name}-la"
  resource_group_name = data.azurerm_resource_group.resource_group.name
}

resource "azurerm_kubernetes_cluster" "k8s" {
  name                = "${var.name}aks"
  location            = var.location
  resource_group_name = data.azurerm_resource_group.resource_group.name
  dns_prefix          = "${var.name}dns"
  kubernetes_version  = var.kubernetes_version

  node_resource_group = "${var.name}-node-rg"

  linux_profile {
    admin_username = "ubuntu"

    ssh_key {
      key_data = var.ssh_public_key
    }
  }

  default_node_pool {
    name                 = "agentpool"
    node_count           = var.agent_count
    vm_size              = var.vm_size
    vnet_subnet_id       = data.azurerm_subnet.akssubnet.id
    type                 = "VirtualMachineScaleSets"
    orchestrator_version = var.kubernetes_version
  }

  identity {
    type = "SystemAssigned"
  }

  addon_profile {
    oms_agent {
      enabled                    = var.addons.oms_agent
      log_analytics_workspace_id = data.azurerm_log_analytics_workspace.workspace.id
    }

    ingress_application_gateway {
      enabled   = var.addons.ingress_application_gateway
      subnet_id = data.azurerm_subnet.appgwsubnet.id
    }

  }

  network_profile {
    load_balancer_sku = "standard"
    network_plugin    = "azure"
  }

  role_based_access_control {
    enabled = var.kubernetes_cluster_rbac_enabled

    azure_active_directory {
      managed                = true
      admin_group_object_ids = [var.aks_admins_group_object_id]
    }
  }

}

data "azurerm_resource_group" "node_resource_group" {
  name = azurerm_kubernetes_cluster.k8s.node_resource_group
  depends_on = [
    azurerm_kubernetes_cluster.k8s
  ]
}

resource "azurerm_role_assignment" "node_infrastructure_update_scale_set" {
  principal_id         = azurerm_kubernetes_cluster.k8s.kubelet_identity[0].object_id
  scope                = data.azurerm_resource_group.node_resource_group.id
  role_definition_name = "Virtual Machine Contributor"
  depends_on = [
    azurerm_kubernetes_cluster.k8s
  ]
}

variables.tf

variable "name" {
  type        = string
  default     = "tamops"
  description = "Name for resources"
}

variable "location" {
  type        = string
  default     = "uksouth"
  description = "Azure Location of resources"
}

variable "addons" {
  description = "Defines which addons will be activated."
  type = object({
    oms_agent                   = bool
    ingress_application_gateway = bool
  })
}

variable "kubernetes_cluster_rbac_enabled" {
  default = "true"
}

variable "kubernetes_version" {
}

variable "agent_count" {
}

variable "vm_size" {
}

variable "ssh_public_key" {
}

variable "aks_admins_group_object_id" {
}

terraform.tfvars

Update ssh_public_key to your own ssh public key
Update aks_admins_group_object_id to an Azure AD group that you will use as “AKS admins”

name     = "tamops"
location = "uksouth"

kubernetes_version         = "1.19.11"
agent_count                = 3
vm_size                    = "Standard_DS2_v2"
ssh_public_key             = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrt/GYkYpuQYRxM3lgjOr3Wqx8g5nQIbrg6Mr53wZGb35+ft+PibDMqxXZ7xq7fC3YuLnnO022IPgEjkF9fP03ZmfUeLjJJvw8YcutN9DD/2cx93BpKFPNUsqEB+za1iJ16kMsCojy35c1R64O+rw20D6iP96rmDAyIc5FR03y00eyAzQ8vo7/u9+VPwpdGEI7QCokZROcj6iNVz1V/1t6G4AEufPLokdj8J0gla/dN+tvnSLRQVBTDiD4jmVGImpWFqqKaH6R9SSXmRzj0uhvJUmSiZAZCb1caPEYgPEvNITuGQFdykPoY/4Z/3B+x/ipEQbWy8yL7bDFSXZTYhVKlPVyPbUtN5QFt7QtCtg84xDAZ6GA6AnONTtMxX2jvdzB9yh1ZsteNrOZ/Jo3ecuie573syQfG23Tu6qTqak8O7ZTOLY9iPx2ego3KvTWH/Q3lIvjnlpfCQtFtSgkNxjalMBk+NwwEgZHWRREOHwJmQIKVN0gSitN1KXobrqwxNk= tamops@Synth"
aks_admins_group_object_id = "e97b6454-3fa1-499e-8e5c-5d631e9ca4d1"

addons = {
  oms_agent                   = true
  ingress_application_gateway = true
}

Deploying a test application to your AKS cluster

Deploy each of the above, as mentioned in order!

You can navigate to the newly created AKS cluster

Once deployed you are ready to begin deploying your applications to Kubernetes 🙂

Get the context of your newly created AKS cluster

az aks get-credentials --resource-group tamops-rg --name tamopsaks

Merged "tamopsaks" as current context in /Users/thomasthorntoncloud/.kube/config

Lets deploy a test application your AKS cluster, we will use azure-voting-app and i’ve added Application Gateway ingress as highlighted below

apiVersion: apps/v1
kind: Deployment
metadata:
  name: azure-vote-back
spec:
  replicas: 1
  selector:
    matchLabels:
      app: azure-vote-back
  template:
    metadata:
      labels:
        app: azure-vote-back
    spec:
      nodeSelector:
        "kubernetes.io/os": linux
      containers:
      - name: azure-vote-back
        image: mcr.microsoft.com/oss/bitnami/redis:6.0.8
        env:
        - name: ALLOW_EMPTY_PASSWORD
          value: "yes"
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 250m
            memory: 256Mi
        ports:
        - containerPort: 6379
          name: redis
---
apiVersion: v1
kind: Service
metadata:
  name: azure-vote-back
spec:
  ports:
  - port: 6379
  selector:
    app: azure-vote-back
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: azure-vote-front
spec:
  replicas: 1
  selector:
    matchLabels:
      app: azure-vote-front
  template:
    metadata:
      labels:
        app: azure-vote-front
    spec:
      nodeSelector:
        "kubernetes.io/os": linux
      containers:
      - name: azure-vote-front
        image: mcr.microsoft.com/azuredocs/azure-vote-front:v1
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 250m
            memory: 256Mi
        ports:
        - containerPort: 80
        env:
        - name: REDIS
          value: "azure-vote-back"
---
apiVersion: v1
kind: Service
metadata:
  name: azure-vote-front
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: azure-vote-front
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: azure-vote-front
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
spec:
  rules:
    - http:
        paths:
          - path: /
            backend:
              serviceName: azure-vote-front
              servicePort: 80

Navigate to the file deployment.yaml and run the following

kubectl apply -f deployment.yaml

With a successful output:-

deployment.apps/azure-vote-back created
service/azure-vote-back created
deployment.apps/azure-vote-front created
service/azure-vote-front created

Now lets get the ingress IP , taken from the below

thomasthorntoncloud@Thomass-MBP scripts % kubectl get ingress
NAME               CLASS    HOSTS   ADDRESS        PORTS   AGE
azure-vote-front   <none>   *       20.90.224.49   80      50s

Accessing the address above on http:// will load the azure-vote-frontend 🙂