HolmesGPT + Azure AI Foundry: A Quick Kubernetes Troubleshooting Demo

I’ve been using HolmesGPT lately and wanted to put together a quick demo using Azure AI Foundry.

The setup is deliberately simple:

• a local kind cluster
• a pod stuck in pending
• HolmesGPT wired up to an Azure OpenAI deployment through Azure AI Foundry

The whole thing takes about five minutes, and you end up with an AI agent diagnosing a real Kubernetes scheduling problem instead of just summarising logs.

In this post, I’ll show how to:

• Create a local Kubernetes cluster with kind
• Deploy a deliberately broken pod
• Connect HolmesGPT to Azure AI Foundry
• Ask HolmesGPT to investigate the issue
• Use the output to understand, and optionally fix, the problem

Prerequisites

You’ll need:

• kind installed
• kubectl installed
• HolmesGPT CLI installed
• An Azure AI Foundry account with an Azure OpenAI deployment

I used an Azure OpenAI deployment called gpt-5.4, but the important part is that you use your own Azure deployment name when passing the model to HolmesGPT.

Install HolmesGPT

On macOS, Homebrew is the easiest path:

brew tap robusta-dev/homebrew-holmesgpt
brew install holmesgpt

Check its there:

holmes ask --help

Set Up for Azure AI Foundry

Grab your endpoint and key from the Azure portal under Endpoints and keys, then export them:

export AZURE_API_KEY="your-azure-api-key"
export AZURE_API_BASE="https://your-resource.cognitiveservices.azure.com/"
export AZURE_API_VERSION="2024-12-01-preview"
export AZURE_DEPLOYMENT_NAME="your-deployment-name"

Run the Demo

Save this as demo-holmes-azure.sh:

#!/usr/bin/env bash
set -euo pipefail

CLUSTER_NAME="${CLUSTER_NAME:-holmes-demo}"
DEMO_MANIFEST="https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/pending_pods/pending_pod_node_selector.yaml"

command -v kind >/dev/null 2>&1 || { echo "ERROR: kind is required"; exit 1; }
command -v kubectl >/dev/null 2>&1 || { echo "ERROR: kubectl is required"; exit 1; }

HOLMES_CMD="holmes"
if ! command -v holmes >/dev/null 2>&1; then
  echo "ERROR: holmes CLI is required. Install with: brew install holmesgpt"
  exit 1
fi

: "${AZURE_API_KEY:?AZURE_API_KEY is not set}"
: "${AZURE_API_BASE:?AZURE_API_BASE is not set}"
: "${AZURE_API_VERSION:?AZURE_API_VERSION is not set}"
: "${AZURE_DEPLOYMENT_NAME:?AZURE_DEPLOYMENT_NAME is not set}"

HOLMES_MODEL="${HOLMES_MODEL:-azure/$AZURE_DEPLOYMENT_NAME}"

cleanup() {
  echo
  echo "Cleaning up kind cluster '${CLUSTER_NAME}'..."
  kind delete cluster --name "${CLUSTER_NAME}" || true
}
trap cleanup EXIT

echo "Creating kind cluster '${CLUSTER_NAME}'..."
kind create cluster --name "${CLUSTER_NAME}"

echo "Waiting for default service account..."
for i in {1..30}; do
  if kubectl get serviceaccount default -n default >/dev/null 2>&1; then
    echo "Default service account ready."
    break
  fi
  sleep 1
done

echo "Deploying demo pod (it will stay Pending)..."
kubectl apply -f "${DEMO_MANIFEST}"

echo "Waiting for pod to appear..."
for i in {1..30}; do
  if kubectl get pod user-profile-import >/dev/null 2>&1; then
    echo "Pod found."
    break
  fi
  sleep 1
done

echo
echo "Pod status:"
kubectl get pod user-profile-import

echo
echo "Running HolmesGPT with model '${HOLMES_MODEL}'..."
export AZURE_API_KEY AZURE_API_BASE AZURE_API_VERSION

${HOLMES_CMD} ask "what is wrong with the user-profile-import pod?" --model="${HOLMES_MODEL}"

Run it:

chmod +x demo-holmes-azure.sh
./demo-holmes-azure.sh

The script creates the cluster, deploys a broken pod, waits for it to show up as Pending, and hands off to HolmesGPT. It also tears down the cluster on exit unless you set SKIP_CLEANUP=true.

HolmesGPT working through the investigation: locating the pod, checking events, fetching logs, reviewing related resources, and summarising the root cause.

What HolmesGPT Finds

The demo pod has a nodeSelector that doesn’t match any node:

nodeSelector:
  label: someLabel

So it sits in Pending forever. HolmesGPT reads the pod, checks events, looks at the nodes, and tells you the scheduling failed because no node has that label. You’ll get something like:

HolmesGPT identifying the scheduling failure and showing the exact pod selector and scheduler message.

No manual kubectl describe chain required. The agent figures out the symptom-to-cause path itself.

Fix Options

The useful part is that HolmesGPT does not just say “the pod is broken”. It explains why the pod is broken and gives you practical fix options.

For this demo, there are two obvious fixes.

You could remove the nodeSelector from the pod spec:

kubectl patch pod user-profile-import -n default --type=json \  -p='[{"op":"remove","path":"/spec/nodeSelector"}]'

Or you could label the node so it satisfies the selector:

kubectl label node holmes-demo-control-plane label=someLabel

In a real environment, the right answer depends on intent. If the selector was added by mistake, remove it. If the workload really is meant to run only on labelled nodes, then add the correct label to the correct node pool.

Applying the Fix

For this demo, I asked HolmesGPT to apply a suggested fix

After applying the node label, HolmesGPT confirms the pod is scheduled successfully and running.

Why Bother?

This is only a brief example, but the pattern is the useful part.

Kubernetes troubleshooting usually starts with a symptom:

• A pod is stuck in Pending
• A deployment will not roll out
• A service is not receiving traffic
• An alert fires, but the root cause is not obvious

Normally, you jump between kubectl get, kubectl describe, logs, events, dashboards, metrics, and runbooks until the problem becomes clear.

HolmesGPT gives you an assistant that can follow that same investigation path, gather the relevant context, and turn the raw Kubernetes state into a clear diagnosis.

In this example, it found:

• The pod was stuck in Pending
• The pod had a nodeSelector
• The only available node did not have the required label
• The scheduler event confirmed the node affinity / selector mismatch
• The pod had no logs because it had never been scheduled
• That is the useful bit. It does not just summarise logs; it reasons across the Kubernetes objects and events.

Going Further

Once the basic flow is working, try asking HolmesGPT broader questions against a real cluster:

holmes ask "what pods are unhealthy and why?"

You can also connect it to other data sources such as Prometheus, Grafana, Loki, or cloud provider tooling.

Operator mode is also worth looking at if you want HolmesGPT running continuously rather than using it for one-off investigations.

Final Thoughts

This demo only scratches the surface, but it is a nice way to understand the value of AI-assisted troubleshooting.

The important thing for me is that HolmesGPT is not just answering from a prompt. It is using tools, querying the cluster, inspecting the actual Kubernetes state, and then explaining what it found.

That is where this becomes useful for platform teams. Not replacing engineers, but reducing the time spent jumping between commands, dashboards, and runbooks to get to the first useful diagnosis.

Resources

• HolmesGPT docs
• Azure AI Foundry setup guide
• HolmesGPT on GitHub