From Azure Policy to APIM: Implementing Azure AI Guardrails

I blogged recently about building an Azure AI Landing Zone without slowing teams down and the split I use between resource governance and runtime governance.

Azure Policy controls what can be deployed and how Azure resources are configured while API Management controls what happens when applications start consuming those resources.

I deliberately kept that post at the architecture level. I referred to controls such as llm-token-limit, model allow-lists and content safety without getting into the policy definitions or XML behind them.

This is the next layer down: the Azure Policy definitions I would look at first, the APIM policies behind some of the runtime controls, and a few details that are easy to miss when you move from an architecture diagram to an implementation.

This post sits between the architecture and the deeper APIM implementation. I’ll follow it with two posts that go further into the APIM side, covering rate limits, token quotas and observability first, then content safety and model control. Here, I want to focus on how the Azure Policy and APIM layers fit together and what the controls actually look like.

The resource layer, in Azure Policy terms

Microsoft’s current built-in policy reference groups these under Foundry Tools, although the individual definitions still use a mixture of Azure AI Services and Cognitive Services in their names.

For an AI platform baseline, these are the definitions I would look at first:

• Azure AI Services resources should restrict network access supports Audit or Deny. This is one of the controls I would rather get right when the resource is created than retrofit later
• Azure AI Services resources should use Azure Private Link is an Audit policy rather than a Deny policy. If you want enforcement around public access, Configure Cognitive Services accounts to disable public network access uses Modify and can remediate existing resources
• Azure AI Services resources should have key access disabled (disable local authentication) supports Audit or Deny. There are also remediation definitions for disabling local authentication. I would use this where the consuming path supports Microsoft Entra ID rather than leaving API keys as another authentication route
• Cognitive Services accounts should use a managed identity is useful as a separate resource control. It governs the identity the service itself can use when accessing other Azure services; it is not the same thing as forcing callers to authenticate through Entra ID
• Diagnostic logs in Azure AI services resources should be enabled is AuditIfNotExists. This is one of those controls that is much easier to establish as a baseline than discover during an incident that a resource has never been sending the logs you expected
• Foundry model deployments should only use approved models is generally available and controls model deployment by identity. It accepts allowedPublishers and allowedAssetIds
• Foundry model deployments should meet eligibility requirements is still Preview. It controls attributes rather than an explicit model allow-list, currently including onlyAllowDirectFromAzure and denyPreviewModels

There is an important detail in the approved-model policy around asset IDs.

Microsoft uses prefix matching. If you specify:

azureml://registries/azure-openai/models/gpt-5/

the trailing slash restricts the match to that model while allowing its versions.

If instead you want to approve only one version, use the full asset ID:

azureml://registries/azure-openai/models/gpt-5.2/versions/1

That gives you a useful choice in how tightly you operate the allow-list. Production may justify pinning an exact version, while another environment may allow any version of an approved model family.

Be careful with prefixes without the trailing slash, Microsoft documents that an entry ending in models/gpt-5 can also match models whose names begin with the same characters, such as GPT-5.2 or GPT-5.4. That is probably broader than most people expect from an allow-list.

The eligibility policy is interesting for rules such as “no Preview models” or “only models Direct from Azure”, but I would not make a production control depend on it yet. Microsoft still marks the definition as Preview and advises against using Preview features for production workloads.

For the established controls, I would package the baseline into an initiative and assign it at the highest sensible scope rather than managing individual assignments per workload. I would also be deliberate about effects.

Deny works well for things that are cheap to get right during provisioning. Existing estates are different. Network access, local authentication and private connectivity can have consumers behind them already, so Audit gives you a much better starting point than discovering the dependency when a deployment or remediation breaks something.

Above image is also available in this repo

The runtime layer, in APIM XML

Once an application starts calling the model, Azure Policy is no longer the main control point. This is where API Management’s AI gateway capabilities come in.

The core set now includes token limits and quotas, token metrics, content safety, semantic caching, backend pools and circuit breakers. Capability support still varies by APIM tier, so I would check the individual policy reference rather than assuming that “AI gateway” means every feature is available on every SKU.

llm-token-limit is the policy I would start with for shared model access.

It lets you apply either tokens-per-minute, a longer-period token quota, or both, against a counter key that you define. If APIM subscriptions represent consuming applications, the subscription ID is a reasonable counter key:

xml

<policies>
    <inbound>
        <base />
        <llm-token-limit
            counter-key="@(context.Subscription.Id)"
            token-quota="100000"
            token-quota-period="Monthly"
            estimate-prompt-tokens="true"
            remaining-quota-tokens-variable-name="remainingQuotaTokens" />
    </inbound>
    <outbound>
        <base />
    </outbound>
</policies>

It only works on Developer, Basic, Standard and Premium tiers (and their v2 equivalents), not Consumption, which is worth checking before you plan an APIM SKU around it.

llm-content-safety routes prompts and, optionally, completions through an Azure AI Content Safety resource configured as an APIM backend:

xml

<inbound>
    <llm-content-safety backend-id="content-safety-backend" shield-prompt="true">
        <categories output-type="EightSeverityLevels">
            <category name="Hate" threshold="4" />
            <category name="Violence" threshold="4" />
        </categories>
    </llm-content-safety>
</inbound>

As written, that checks the incoming prompt, enables prompt shielding and blocks Hate or Violence content at severity 4 or above.

If you also want the policy to validate completions, enforce-on-completions="true" can be set when the policy is applied inbound.

The part I would not skip is the backend configuration.

backend-id is the name of an APIM backend entity configured for the Azure AI Content Safety endpoint. APIM’s managed identity also needs the appropriate access to the Content Safety resource, and the backend authentication needs to use the Cognitive Services resource identifier documented by Microsoft.

Pointing the XML at the name of an Azure Content Safety resource is not enough. The APIM backend is part of the configuration.

Then there is <a href="https://learn.microsoft.com/en-us/azure/api-management/llm-emit-token-metric-policy" target="_blank" rel="noopener">llm-emit-token-metric</a>.

I would describe this as the foundation for usage attribution and showback rather than saying the policy itself provides chargeback. It sends token metrics to Application Insights and lets you add dimensions such as API, product, subscription, client or another consumer identifier.

Above image is also available in this repo

That means the same gateway enforcing a limit can also tell you which consumer is driving the usage.

There is a practical limit here: custom metrics have cardinality constraints. API Management allows up to five custom dimensions on the policy, limits the number of unique values and time series it tracks, and can stop tracking new series after those limits are reached.

I therefore would not put a user ID, application ID, repository, team, model, environment and every other interesting attribute into a metric just because the policy lets me add dimensions. Pick the dimensions you will actually operate against and keep higher-cardinality detail in logs where appropriate.

Semantic caching with llm-semantic-cache-lookup and llm-semantic-cache-store, plus backend pools and circuit breakers, sit alongside these controls. I see those more as performance, resiliency and cost controls than access governance, but operationally they end up in the same APIM policy path.

Why the two layers change at different speeds

The resource layer is the part I would be most careful about retrofitting. Moving an existing model endpoint behind private connectivity, disabling public network access or removing local-key authentication changes assumptions that applications may already depend on. None of those changes inherently requires a big migration, but applying them before consumers are ready is an easy way to cause an outage.

That is why I prefer to establish those controls when the resource is provisioned and use Audit to expose gaps in anything that already exists.

The APIM layer is easier to tune as usage develops and changing a token quota, adjusting a rate limit or adding another telemetry dimension changes gateway behaviour without changing the AI resource behind it. The policies can also be applied at global, product, API or operation scope, depending on how you have chosen to divide responsibility between the platform and its consumers.

That gives the two layers different release cadences and resource governance should change relatively slowly because it defines the security and deployment boundary of the platform. Runtime controls should be expected to move as you learn what teams are consuming, which limits they regularly hit and where the defaults are too loose or too restrictive.

If I were implementing this against an existing Azure estate, I would start with visibility.

Assign the established network, authentication, diagnostics and approved-model policies in Audit where enforcing them immediately could affect existing workloads. Inspect the compliance results, understand which consumers would be affected, and fix the resource path before turning those controls into enforcement.

Then put a real consumer through APIM with an identifiable counter key, a token limit and token telemetry. Add content safety where the workload needs it.

At that point the two-layer model stops being an architecture diagram. Azure Policy controls what is allowed to exist, APIM controls how it is consumed, and you have enough evidence from both layers to tighten the controls without guessing.