APIM Policy Patterns for AI Governance: Part 1 – Rate Limits, Token Quotas & Observability

Putting APIM in front of a model endpoint does not give you useful runtime governance on its own. For shared AI access, I want request bursts controlled, token consumption bounded and enough telemetry to know which consumer is responsible for the traffic.

Those controls solve different problems, request rate tells you how often something is calling the API, while token usage tells you far more about how much model capacity it is consuming.

I’ve written before about building an Azure AI landing zone and the split between resource governance and runtime governance. Azure Policy controls what can be deployed and how those resources are configured. APIM sits on the consumption path, which is where rate limits, token quotas, usage attribution and other runtime controls belong. This is the APIM layer I would put in first.

Request rate and token consumption are different controls

Request rate is useful for protecting an API from bursts, but it is a poor proxy for AI consumption.

Two requests against the same model can have very different input context and output lengths. One might consume a few hundred tokens while another consumes thousands. Input and output tokens may also be priced differently, and available model capacity is not determined solely by the number of HTTP requests reaching the endpoint.

I therefore treat request and token limits separately.

Request limits protect the gateway and backend from bursts in call volume. Token limits control how much model consumption a caller can drive over a period of time. In a shared AI platform I usually want both.

There is also a difference in how early APIM can enforce them, a request rate limit can reject a call before it reaches the backend. Token consumption is less exact because APIM cannot know the final completion size until the model has generated it.

The llm-token-limit policy can estimate prompt tokens before forwarding a request, but the final count still depends on the response. Concurrent requests can temporarily take consumption beyond the configured limit before APIM has processed those responses and updated the counter.

That makes a token quota a useful runtime guardrail, but I would not treat it as an exact financial ledger.

Rate limit against a trusted consumer

For normal request throttling, I generally want the counter tied to a known consumer rather than an IP address.

What represents a consumer depends on the gateway design. It might be an APIM subscription, an application ID from an already validated Entra token, or another identity established earlier in the inbound policy.

For this example, each consuming application has an APIM subscription:

<inbound>
    <base />
    <set-variable
        name="consumerId"
        value="@(context.Subscription.Id)" />
    <rate-limit-by-key
        calls="100"
        renewal-period="60"
        counter-key='@("requests-" + context.Api.Id + "-" + (string)context.Variables["consumerId"])' />
</inbound>

<a href="https://learn.microsoft.com/en-us/azure/api-management/rate-limit-by-key-policy" target="_blank" rel="noopener">rate-limit-by-key</a> returns 429 Too Many Requests when the limit is exceeded and uses Retry-After by default.

Pulling consumerId into a variable keeps the rest of the policy readable and gives me one trusted identity that can be reused for token limits and telemetry later in the pipeline.

I would not use an arbitrary caller-supplied header such as client-id as the counter key unless something earlier in the policy has authenticated or validated it. Otherwise the caller can change the value and get another rate-limit bucket.

There are a couple of operational details I would account for as well.

APIM rate limiting is distributed, so the configured number is not guaranteed to be an exact boundary. Counters are also tracked independently by each gateway, which means a multi-region APIM deployment does not automatically have one global counter shared across every region.

rate-limit-by-key is not supported on the Consumption tier. It is supported on Developer, Basic, Basic v2, Standard, Standard v2, Premium and Premium v2.

Use llm-token-limit rather than building your own counter

For normal token rate limiting and quotas, I would use what APIM already provides before creating custom state.

<a href="https://learn.microsoft.com/en-us/azure/api-management/llm-token-limit-policy" target="_blank" rel="noopener">llm-token-limit</a> can enforce a short-term token rate and a longer token quota against the same consumer key:

<inbound>
    <base />
    <llm-token-limit
        counter-key='@((string)context.Variables["consumerId"])'
        tokens-per-minute="10000"
        token-quota="100000"
        token-quota-period="Monthly"
        estimate-prompt-tokens="true"
        remaining-tokens-variable-name="remainingTokens"
        remaining-quota-tokens-variable-name="remainingQuotaTokens" />
</inbound>

The numbers are illustrative and the useful part is that one policy gives the consumer a short-term token rate and a longer allocation without requiring another quota store. Quota periods can be hourly, daily, weekly, monthly or yearly.

One behaviour I would test explicitly is the response status. Exceeding tokens-per-minute returns 429 Too Many Requests, while exhausting token-quota returns 403 Forbidden.

If client applications have retry logic around the gateway, they need to understand those as two different conditions.

I would also make estimate-prompt-tokens a deliberate choice.

With it set to true, APIM estimates the prompt before forwarding it. That can prevent a request reaching the backend when the caller is already over its limit, although Microsoft notes there is a performance cost to the estimation.

With it set to false, APIM uses actual token usage reported in the model response. The request can therefore reach the model and consume tokens before APIM discovers that the limit has been crossed.

Streaming changes the behaviour again because APIM estimates prompt and completion tokens for streamed responses.

llm-token-limit has the same APIM tier constraint as rate-limit-by-key, so it is not available on Consumption.

I would not make the APIM cache a billing ledger

A token quota will not cover every cost-control requirement.

A team might want an allowance expressed in pounds rather than tokens, for example. That becomes more complicated because input and output prices differ, models have different prices and a request can potentially be routed to a different model.

APIM gives you cache-lookup-value and cache-store-value, so technically you can maintain arbitrary state and build custom counters around it.

I would be very careful about turning that into financial enforcement.

Microsoft describes the built-in cache as volatile and caching as best effort. In classic APIM tiers, the internal cache is also cleared progressively during service updates.

An external Redis-compatible cache gives you more control over persistence and availability, but by that point I would question whether a gateway cache should be the authoritative ledger for a financial budget.

For most platforms, I would use APIM token controls to constrain runtime consumption and send the usage data into the proper FinOps or reporting path for showback and chargeback.

Emit useful token telemetry before choosing quotas

llm-emit-token-metric is one of the first policies I would add to a shared AI gateway.

It belongs in the inbound section even though APIM can use usage information returned later in the model response.

A basic configuration can stay small:

<inbound>
    <base />
    <llm-emit-token-metric namespace="llm-usage">
        <dimension name="API ID" />
        <dimension name="Product ID" />
        <dimension name="Subscription ID" />
    </llm-emit-token-metric>
</inbound>

The policy sends token metrics to Application Insights. Depending on the model and provider, those metrics can include prompt, completion and total token counts.

If an APIM subscription maps cleanly to a consuming application or team, Subscription ID gives you a useful attribution boundary without introducing another custom identifier. Product ID is useful where APIM products represent service tiers, teams or defined access profiles.

Custom dimensions need more restraint.

APIM currently allows up to five custom dimensions on the policy, while Azure Monitor also imposes active time-series limits. Putting user ID, application ID, repository, team, model and environment into every metric quickly creates a cardinality problem.

I would start with the dimensions somebody will actually use to make an operational or cost decision and add more only when there is a question the existing telemetry cannot answer.

Metrics are good for questions such as which consumers are using tokens, where usage is growing and whether limits are regularly being approached. They are less useful when somebody has one failed request and wants to know what happened to it.

For that I would keep request-level telemetry alongside the metrics:

<outbound>
    <base />
    <trace source="ai-gateway" severity="information">
        <message>AI gateway response</message>
        <metadata
            name="requestId"
            value="@(context.RequestId.ToString())" />
        <metadata
            name="consumer"
            value="@(context.Subscription.Id)" />
        <metadata
            name="api"
            value="@(context.Api.Name)" />
        <metadata
            name="status"
            value="@(context.Response.StatusCode.ToString())" />
        <metadata
            name="latencyMs"
            value="@(context.Elapsed.TotalMilliseconds.ToString())" />
    </trace>
</outbound>

context.Elapsed is the time elapsed since APIM received the request, so I would use that instead of stamping another start time purely to calculate gateway latency.

There is a cost to tracing everything, though. Trace policy telemetry is not affected by Application Insights sampling, and tracing can expose sensitive information if request data starts finding its way into the metadata.

For a shared AI gateway I would keep prompts and completions out of generic operational traces unless there is a deliberate requirement, retention policy and access model for storing them.

Backend throttling needs a separate plan

Controlling your own consumers does not stop the model backend from returning a 429 or failing with a server-side error.

For a simple primary and fallback arrangement, APIM’s retry policy can switch backend entities after a failed response.

The child policies inside retry execute once before the retry condition is evaluated, so I would keep the backend selection explicit:

<backend>
    <retry
        condition="@(context.Response != null && (context.Response.StatusCode == 429 || context.Response.StatusCode >= 500))"
        count="1"
        interval="1"
        first-fast-retry="true">
        <set-variable
            name="attemptCount"
            value='@(context.Variables.GetValueOrDefault<int>("attemptCount", 0) + 1)' />
        <set-backend-service
            backend-id='@(context.Variables.GetValueOrDefault<int>("attemptCount", 0) < 2
                ? "primary-openai"
                : "fallback-openai")' />
        <forward-request />
    </retry>
</backend>

The first execution increments attemptCount to 1 and uses primary-openai. If the backend returns a 429 or 5xx, the retry executes immediately, increments the counter to 2 and switches to fallback-openai.

Buffering the request body matters here because the request needs to be sent again on retry.

Both backend names are APIM backend entities, rather than URLs embedded directly into the policy.

I would keep the scope of this example clear. It handles HTTP 429 and 5xx responses. It is not generic failover for every possible backend failure because the condition deliberately requires context.Response to exist.

If fallback becomes a permanent part of the platform rather than a narrow retry path, I would move towards APIM backend pools, priority-based routing and circuit breakers rather than growing the retry XML.

Backend pools also give you somewhere sensible to model several deployments without teaching each API policy about every endpoint.

The details I would test before sharing the policy

These are the things most likely to get missed when a policy moves from an example into a shared gateway:

• llm-emit-token-metric belongs in inbound.
• llm-token-limit returns 429 for the token rate but 403 when the longer quota is exhausted.
• rate-limit-by-key and llm-token-limit are not available on the Consumption tier.
• Caller-controlled values are not trustworthy consumer keys unless they have already been authenticated or validated.
• Reusing the same counter key across policy scopes shares the counter, so include the scope in the key when separate limits are required.
• Rate and token counters are tracked per gateway rather than aggregated into one global multi-region counter.
• Concurrent requests can briefly take actual token consumption beyond the configured token limit.
• The built-in APIM cache is volatile and should not be the only source of truth for a financial budget.
• Custom metric dimensions need to be kept under control because high-cardinality values multiply the number of active time series.
• A retried POST needs its request body available for the next attempt.
• XML quoting and C# policy-expression quoting are easy to get wrong, particularly when dictionary access appears inside an attribute.

I would validate those behaviours in a non-production APIM instance rather than assuming that policy XML compiling means the design behaves the way the client expects.

I would start with telemetry, then enforce limits

I would put llm-emit-token-metric in before choosing a token quota.

Get real traffic through the gateway and confirm that the dimensions map consumption back to the consumer boundary you care about. If a spike appears on the dashboard, you should be able to identify the application or team that owns it without joining three other datasets first.

Once that attribution works, use the observed traffic to set the controls.

Add rate-limit-by-key where burst traffic can put pressure on the gateway or backend. Add tokens-per-minute where one consumer can put pressure on shared model capacity. Add a longer token quota where the platform needs a boundary around overall consumption.

Then test the failure paths as deliberately as the successful request:

1. Push a consumer over its request rate and confirm the client handles the 429 correctly.
2. Exhaust the token-per-minute allowance and check the retry behaviour.
3. Exhaust the longer token quota and make sure the client treats the 403 as quota exhaustion rather than an authentication failure.
4. Trigger backend throttling and confirm the request reaches the expected fallback backend.
5. Verify the same request can be traced from its consumer identity through token telemetry and request-level diagnostics.
6. Test the behaviour in every APIM region you operate rather than assuming counters are shared globally.

I would not call the AI gateway ready until those paths are as predictable as the successful one.