Send GitHub Copilot CLI telemetry to Azure Application Insights

I wanted to send GitHub Copilot CLI telemetry to Azure Application Insights so I could query model requests, token usage and tool executions, including which skills were being used, with KQL. I used an OpenTelemetry Collector running on Azure Container Apps to receive telemetry from the CLI and forward it to Application Insights.

The Collector provides an authenticated OTLP endpoint for developer machines, with the Azure export configuration held centrally in the Container App. Application Insights stores the telemetry for queries and Azure Workbooks.

The implementation is in my github-copilot-otel-azure-containerapp repository, including the deployment scripts, Collector configuration and example KQL queries.

Use a Collector to receive Copilot CLI telemetry

GitHub Copilot CLI supports OpenTelemetry export, including traces and metrics for agent activity. I configured it to send OTLP over HTTPS to the Collector, which uses the Azure Monitor exporter to send the data to Application Insights.

Each part has a specific job:

I used the Collector so I could change the Azure destination and processing configuration centrally. The CLI needs the Collector URL and ingestion token, rather than the Application Insights connection string.

This uses a Collector container that I deploy and configure myself. It is separate from the managed OpenTelemetry agent available in Container Apps.

Create the Azure resources and configure Container Apps

The deployment script creates a resource group and these four resources:

• A Log Analytics workspace.
• A workspace-based Application Insights resource linked to that workspace.
• A Container Apps environment configured for application logs.
• A Container App running OpenTelemetry Collector Contrib.

Clone the repository and run the deployment from its root directory:

git clone https://github.com/thomast1906/github-copilot-otel-azure-containerapp.git
cd github-copilot-otel-azure-containerapp
az login
./scripts/deploy.sh

You need Azure CLI, Git, Bash, curl and openssl, plus permissions to create the resources in your subscription. The script defaults to UK South. To change resource names or the location, follow the .env instructions in the repository README.

The script generates an ingestion token if POC_TOKEN is not already set. It prints the Collector hostname and commands to set POC_FQDN and POC_TOKEN. Use those values in the same terminal for the verification and CLI steps below. The output includes the token, so keep it out of shared logs and screenshots.

Application Insights provides the application telemetry query scope, with its data stored in the linked workspace. Container Apps logs give me somewhere to check Collector startup and export errors.

Use the Collector Contrib distribution, which includes the authentication extension and Azure Monitor exporter used here. Pin the image to the version you validate with your configuration.

The Container App needs these settings:

The script stores the values as Container App secrets and references them in the container’s environment settings. The ingestion token must match the value supplied by the CLI.

The Application Insights connection string identifies the destination; it is not an authentication credential. The connection-string-only export shown here requires Application Insights local authentication to be enabled.

Configure the Collector receiver and Azure Monitor exporter

The repository’s otel-config.yaml configures OTLP/HTTP on port 4318 and requires a Bearer token through its authentication extension.

extensions:
  bearertokenauth:
    token: ${env:OTEL_INGEST_TOKEN}

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
        auth:
          authenticator: bearertokenauth

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 350
    spike_limit_mib: 100
  batch:
    timeout: 5s

exporters:
  azuremonitor:
    connection_string: ${env:APPLICATIONINSIGHTS_CONNECTION_STRING}
    spaneventsenabled: true

service:
  extensions: [bearertokenauth]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [azuremonitor]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [azuremonitor]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [azuremonitor]

The bearertokenauth extension reads the token from the container environment. Referencing it under the receiver’s auth setting makes authentication part of receiving telemetry.

The memory limiter runs before batching and applies backpressure when memory use reaches its configured threshold. Match the 350 MiB limit and spike allowance to the memory allocated to your container. The batch processor groups records before export, with a five-second timeout.

The Azure Monitor exporter reads the Application Insights connection string from the environment. I enabled spaneventsenabled so span events are exported as well.

The configuration includes traces, metrics and logs pipelines. The CLI’s documented exports are traces and metrics; the logs pipeline also accepts OTLP log records if a client sends them.

Configure GitHub Copilot CLI to send telemetry

Use the POC_FQDN and POC_TOKEN values from the deployment output in the shell that will launch Copilot CLI:

export COPILOT_OTEL_ENABLED=true
export COPILOT_OTEL_EXPORTER_TYPE=otlp-http
export OTEL_EXPORTER_OTLP_ENDPOINT="https://${POC_FQDN}"
export OTEL_EXPORTER_OTLP_PROTOCOL=http/json
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer ${POC_TOKEN}"
export OTEL_SERVICE_NAME=github-copilot
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=false

copilot

POC_FQDN is the Container App hostname without a scheme or path. The endpoint uses the base HTTPS URL; the OTLP exporter adds the signal paths such as /v1/traces and /v1/metrics.

OTEL_SERVICE_NAME gives the telemetry a consistent service name for filtering. The message content setting keeps full prompt and response capture disabled while retaining operational telemetry. Review the emitted metadata too, as identifiers and error details can still contain information you need to control.

Start a new Copilot process after setting the variables. If a session is already running, restart it so it inherits the configuration.

Check authentication and confirm ingestion into Azure

The repository includes a verification script. With POC_FQDN and POC_TOKEN set, run:

./scripts/verify.sh

It prints the HTTP status for requests without and with the token. The expected responses are 401 and 200 respectively.

Run a small task in Copilot CLI, then allow time for batching and Azure ingestion. If no records arrive, use the repository’s logging script to inspect Collector startup and export errors:

./scripts/logs.sh

Also check that the CLI started with the expected endpoint and headers, and that the service-name filter in your queries matches the records you sent.

Query GitHub Copilot CLI telemetry with KQL

Open Logs on the Application Insights resource. These examples use the Application Insights resource query schema, including dependencies and customMetrics. When querying the backing Log Analytics workspace directly, the corresponding tables are AppDependencies and AppMetrics, with different column names.

The repository includes KQL examples for metrics, tool activity and skill names. The examples below also use individual dependency spans to inspect model requests and calculate token usage for those requests.

Confirm which services are sending metrics

Start by checking the metric record count and latest timestamp for each service:

customMetrics
| where timestamp > ago(7d)
| summarize Records = count(), LastSeen = max(timestamp)
    by cloud_RoleName
| order by Records desc

Look for github-copilot in cloud_RoleName to confirm that CLI metrics are arriving. Records counts stored metric records, while LastSeen shows the most recent timestamp. This checks metric ingestion; dependency spans are queried separately below.

Find which agent skills are being used

In the PoC data, the invoked skill name appears on dependency records in customDimensions["gen_ai.skill.name"]:

dependencies
| where timestamp > ago(24h)
| where cloud_RoleName == "github-copilot"
| extend skill = tostring(customDimensions["gen_ai.skill.name"])
| where isnotempty(skill)
| project timestamp, skill, name, duration, success
| order by timestamp desc

To aggregate those records by skill:

dependencies
| where timestamp > ago(7d)
| where cloud_RoleName == "github-copilot"
| extend skill = tostring(customDimensions["gen_ai.skill.name"])
| where isnotempty(skill)
| summarize
    RecordedExecutions = count(),
    FailedExecutions = countif(success == false),
    LastSeen = max(timestamp)
    by skill
| order by RecordedExecutions desc

A generic tool count for skill does not identify the invoked skill. This query relies on the skill-name attribute ob

Show model request counts and latency

Filter to model requests and group by the returned model name, falling back to the requested model when the response attribute is absent:

dependencies
| where timestamp > ago(24h)
| where cloud_RoleName == "github-copilot"
| where tostring(customDimensions["gen_ai.operation.name"]) == "chat"
| extend model = coalesce(
    tostring(customDimensions["gen_ai.response.model"]),
    tostring(customDimensions["gen_ai.request.model"]),
    "unknown")
| extend durationMs = duration / 1ms
| summarize
    Requests = count(),
    FailedRequests = countif(success == false),
    AverageDurationMs = avg(durationMs),
    P95DurationMs = percentile(durationMs, 95)
    by model
| order by Requests desc

This measures the duration of the recorded model-call spans, rather than the duration of the whole agent run.

Calculate input and output tokens by model

The CLI can record token usage on model-call spans. Keep the chat filter when summing these attributes, because parent agent spans can also contain usage totals for the same work.

dependencies
| where timestamp > ago(24h)
| where cloud_RoleName == "github-copilot"
| where tostring(customDimensions["gen_ai.operation.name"]) == "chat"
| extend model = coalesce(
    tostring(customDimensions["gen_ai.response.model"]),
    tostring(customDimensions["gen_ai.request.model"]),
    "unknown")
| extend
    inputTokens = tolong(customDimensions["gen_ai.usage.input_tokens"]),
    outputTokens = tolong(customDimensions["gen_ai.usage.output_tokens"])
| where isnotnull(inputTokens) or isnotnull(outputTokens)
| summarize
    RequestsWithUsage = count(),
    InputTokens = sum(inputTokens),
    OutputTokens = sum(outputTokens)
    by model
| order by InputTokens desc

These are totals from the ingested model-call spans with usage attributes. They are not a billing reconciliation, and missing or sampled spans will affect the result.

The repository also includes a token query against customMetrics using the metric names observed in the PoC. To check which names your CLI version sends, run:

customMetrics
| where timestamp > ago(30m)
| where cloud_RoleName == "github-copilot"
| summarize Records = count() by name
| order by Records desc

Records counts stored metric records, not tokens. Check the metric’s aggregation and temporality before summing its values, particularly for cumulative counters or exported histograms.

Count tool executions and failures

Use the tool execution spans to see which tools ran and how long they took:

dependencies
| where timestamp > ago(2224h)
| where cloud_RoleName == "github-copilot"
| where tostring(customDimensions["gen_ai.operation.name"]) == "execute_tool"
| extend tool = tostring(customDimensions["gen_ai.tool.name"])
| where isnotempty(tool)
| summarize
    Executions = count(),
    FailedExecutions = countif(success == false),
    AverageDurationMs = avg(duration)
    by tool
| order by Executions desc

Failure counts include records explicitly marked unsuccessful. A missing success value is not counted as a confirmed failure.

A generic tool count for skill does not identify the invoked skill. This query relies on the skill-name attribute observed in the PoC; if your CLI version does not emit it, the query returns no rows.

These queries can feed Azure Workbooks once you have confirmed the fields in your own data. I’ve written before about measuring AI-assisted engineering outcomes, and I would use these execution records alongside workflow outcomes rather than treating usage counts as a productivity measure.

The limits of this PoC

I used one replica and a shared static Bearer token to keep the initial setup small. The token controls access to the Collector but does not authenticate an individual developer. It remains available to the client process, so storing the server-side copy as a Container App secret does not solve client credential distribution.

Before a wider rollout, I would define credential distribution and rotation, choose the appropriate network access and authentication model, and set ingestion limits. Availability and buffering also need a decision: this configuration has no persistent queue, so a restart can lose telemetry held in memory.

Start with the repository deployment instructions, run ./scripts/verify.sh, then complete a CLI task and check its records in Application Insights. Use those records to validate the query fields before building a Workbook.