Deploying a model endpoint in Microsoft Foundry or Azure OpenAI is easy enough. The harder part starts when several teams want to use it, with different applications, budgets, data boundaries, security requirements and opinions about which model they need.
Without a shared platform pattern, direct integrations spread quickly. Identity gets handled differently between applications, network controls vary, ownership becomes vague, and cost usually becomes interesting after the consumption has already happened.
Centralising every decision creates a different problem because teams then wait for platform or security approval whenever they want to test a model, change a quota or try a different integration.
For me, a useful Azure AI Landing Zone sits between those two extremes. It gives teams a pre-approved route for building AI workloads while keeping the controls that genuinely need to be consistent across the organisation. The governed route also needs to be the easiest route, because a platform that is regularly slower than bypassing it will eventually be bypassed.
I’ve written before about the platform controls AI engineering needs, particularly ownership, cost visibility, observability and sensible defaults. This is the Azure architecture underneath that: what I would put into an AI Landing Zone, where Azure API Management fits, and which decisions should stay with workload teams.
An AI Landing Zone should remove repeated platform decisions
An Azure AI Landing Zone is not one Azure resource. I think of it as a repeatable platform foundation that removes decisions teams should not need to redesign for every AI workload.
That usually includes:
- Microsoft Entra ID, managed identities and role-based access control
- private endpoints and controlled network paths
- Azure Policy for resource-level governance
- Azure API Management for governed runtime access
- Application Insights, Log Analytics, alerts and dashboards
- budgets, quotas, tagging and usage attribution
- reusable Terraform or Bicep modules
- clear ownership between platform and workload teams
The value comes from how those controls work together. A private endpoint without a usable access pattern will be worked around and a budget alert without attribution tells you money was spent but not who needs to act. An API gateway without resource governance does not stop inconsistent resources appearing behind it.
A landing zone should resolve those platform decisions once and package the result into something teams can consume repeatedly. Product teams can then spend their time on application behaviour, prompts, evaluations, data and user experience instead of rebuilding the surrounding Azure platform.
Microsoft’s Azure AI Landing Zones design checklist covers identity, networking, governance, cost, monitoring, reliability and model deployment, so it is a useful starting point rather than inventing the whole pattern yourself.
Keep platform and workload ownership separate
I would normally start with a centrally managed access layer and workload-owned applications:
The central platform can own:
- APIM and shared API contracts
- common authentication and authorisation policies
- baseline rate and token limits
- approved backend access patterns
- shared telemetry and usage attribution
- reusable infrastructure modules and policy fragments
- organisation-wide Azure Policy assignments
Workload teams should retain responsibility for:
- application and domain logic
- prompts and system instructions
- evaluations and quality thresholds
- workload-specific data and indexes
- model selection within agreed boundaries
- application-level observability
- the business outcome the feature is meant to deliver
APIM should govern access to AI services without making the platform team the owner of every prompt, index, model deployment or orchestration flow.
A rule I find useful is:
Gateway the API contract, govern the resources where the data and runtime live. Central controls are useful, but centralising every implementation decision creates a queue.
APIM should govern the runtime contract
Azure API Management gives you a controlled point in the request path rather than relying entirely on reporting after the traffic has reached the model.
For AI APIs, that can include:
- validating Microsoft Entra tokens
- identifying the calling application, user or team
- enforcing request and token limits
- applying quotas by consumer
- controlling which approved backends are exposed
- applying content-safety checks where required
- routing or load balancing across approved backends
- using circuit breakers when a backend is unhealthy or throttled
- emitting token and latency telemetry
- applying consistent error handling and headers
Microsoft’s AI gateway capabilities in API Management now cover much of this directly. The llm-token-limit policy can enforce token rate limits or quotas by a calculated key, while llm-emit-token-metric can send token consumption metrics and custom dimensions to Application Insights. Microsoft also provides llm-content-safety for checking prompts and responses through Azure AI Content Safety. Capability availability varies between APIM tiers, so I would check the policy support matrix when choosing the gateway tier rather than assuming every feature exists everywhere.
This is where the gateway becomes particularly useful for shared AI access. A budget alert can tell you that consumption increased, but a runtime token policy can prevent one consumer from exhausting an agreed allowance before someone sees the alert.
I would not turn every governance concern into a hard deny, though. Some controls should block, some should warn, and some only need to create evidence. If every experiment requires an exception before it can start, the gateway has become an approval system rather than a platform capability.
Azure Policy and APIM policies solve different problems
Azure Policy is an important part of an AI Landing Zone, but its job is different from the runtime controls in APIM.
At the resource layer, Azure Policy can govern things such as:
- approved regions and deployment types
- allowed resource configuration
- required tags
- diagnostic settings
- public network access
- private networking requirements
- which models may be deployed
Microsoft now provides built-in Azure Policy definitions specifically for controlling model deployment in Microsoft Foundry.
APIM operates once something is being consumed. It can decide whether a caller is allowed through the gateway, how much traffic or how many tokens that consumer may use, which backend receives the request and what telemetry is emitted.
The boundary looks roughly like this:
| Risk or requirement | Azure Policy | APIM policies |
|---|---|---|
| Unsupported region or SKU deployed | Yes | No |
| Model endpoint publicly accessible | Yes | No |
| Ownership tags missing | Yes | No |
| Application floods a model endpoint | No | Yes |
| Team requests an unapproved model | No | Yes |
| Consumer exceeds its token allowance | No | Yes |
| Usage cannot be attributed to a team | Indirectly | Yes |
| Requests need consistent routing or fallback | No | Yes |
I would therefore separate governance into three layers:
- Azure Policy controls the Azure resources
- APIM controls runtime consumption and the application controls workload-specific behaviour.
Trying to make one of those layers do all three jobs usually leaves a gap somewhere else.
AI cost control belongs in the architecture
AI cost discussions can become binary quite quickly. Teams either get very little restriction and spend becomes difficult to predict, or higher-cost models and quota changes end up behind an approval process that slows experimentation.
I would rather make consumption visible, attributable and proportionate to the environment and workload.
Development workloads should not inherit production-sized limits by default. A small summarisation experiment does not need the same allowance as a production service with known traffic. Teams should still be able to request more capacity, but the increase needs an owner, a reason and, where it is temporary, an expiry date.
Runtime controls are useful alongside Azure budgets because they solve different parts of the problem:
| Layer | Typical control | Purpose |
| FinOps | Budgets and alerts | Visibility of unusual spend |
| Platform governance | Azure Policy and mandatory tagging | Ownership and resource consistency |
| Runtime governance | APIM rate, token and model policies | Actively control consumption |
| Observability | Token, latency and outcome telemetry | Evidence for optimisation |
| Workload engineering | Prompt, caching and model design | Better economics per useful result |
The FinOps Foundation’s FinOps for AI guidance makes a similar distinction between variable consumption and provisioned capacity, and puts workload profiling and business value alongside the underlying unit price.
This matters because platform controls cannot fix an inefficient application design on their own. If a workload repeatedly sends large amounts of unnecessary context, moving it to a cheaper model may reduce the unit price while leaving the waste untouched.
Use different defaults in each environment
Applying the same limits everywhere is simple to configure but difficult to justify operationally.
I would package these as reusable profiles rather than asking every team to build its own APIM policy configuration. Once each workload carries a slightly different copy of the same XML, it becomes much harder to understand which controls are actually in force.
Exceptions are unavoidable, but they should be changes to a known baseline rather than the normal way a workload gets onboarded.
Give teams a sensible model starting point
An allow-list tells a team what it may use, but it does not help much with where to start.
I prefer a small number of internal model profiles, for example:
- a default tier for common workloads and experimentation
- a higher-capability tier where evaluations show a material improvement
- specialist models for workloads with particular technical or domain requirements
The names matter less than having a default and a clear path away from it.
Model selection should then be backed by evaluation rather than preference. Saying one model is “better” is not particularly useful unless you can explain which task it performs better, by how much, what the latency difference is and whether that improvement justifies the additional cost.
Forcing everything onto the cheapest model can also be false economy if the result creates retries, manual correction or downstream rework. I would look at model cost alongside whether the workload produced an acceptable outcome, which is the same reason I favour measuring AI-assisted engineering outcomes rather than activity.
Quotas need something identifiable behind them
A large organisation-wide quota is easy to create and difficult to operate.
I would key quotas to something that has an owner, such as an application, team, APIM product, subscription, environment or defined use case. That gives you a way to answer operational questions when consumption changes:
- which application is consuming the tokens
- whether the traffic is development or production
- which model is driving the increase
- whether throttling is isolated to one consumer
- whether growth is expected
- who can decide whether the limit should change
I would also start with showback before building a complicated chargeback model. Giving teams a dashboard with their own consumption, models and throttling events is useful immediately, while internal billing introduces another set of organisational decisions that may not be justified early on.
Temporary quota increases should be easy to make but should carry an owner, reason and expiry. Otherwise temporary exceptions quietly become part of the permanent platform design.
Collect useful telemetry without collecting everythingThe platform should be able to attribute AI usage to enough context that someone can act on it.
At a minimum I would want:
- team or product
- application or client identity
- environment
- requested model
- prompt tokens
- completion tokens
- total tokens
- latency
- throttling and quota events
- backend route
- request outcome
APIM’s llm-emit-token-metric policy supports custom dimensions, which makes it possible to attach much of that consumer context to token telemetry rather than producing one organisation-wide consumption number.
The interesting part is then looking for patterns: unexpected development consumption, consumers repeatedly hitting limits, workloads with unusually large prompts, model-specific latency differences, or applications regularly asking for quota increases.
I would be much more cautious about central prompt and response logging.
APIM can log prompts and completions, but the fact that the capability exists does not mean it should be enabled as the platform default. Prompts and responses can contain personal information, commercially sensitive data or security-relevant content, and a central store of that data creates its own access, retention and operational requirements.
Token counts, model names, consumer identities and correlation IDs give the platform useful operational evidence without requiring every prompt to be retained. Workloads that genuinely need content logging can enable it deliberately with suitable retention, redaction and access controls.
Be careful with central prompt and response logging – Content may contain personal data, commercially sensitive material or security-relevant information.
Do not turn APIM into the application
Once APIM becomes the approved route for AI traffic, it is tempting to keep moving more AI behaviour into policies.
Authentication, rate limiting, token controls, routing, content-safety enforcement and common telemetry are sensible gateway responsibilities.
Prompt construction, retrieval logic, conversation memory, agent orchestration and domain decisions generally belong with the workload.
Moving too much of that behaviour into APIM couples gateway releases to application behaviour, makes policies harder to test and eventually leaves the platform team troubleshooting business logic written in XML.
I would keep the gateway contract stable and push workload decisions back to the workload.
The same judgement applies to shared AI resources. A central resource can work well where several applications genuinely have the same data boundaries, capacity needs, availability requirements and release model. Separate workload-owned resources are often easier once those requirements start to diverge.
An AI Landing Zone should support both patterns rather than treating centralisation as the goal in itself.
Start with one complete governed path
The first version does not need every possible AI governance feature. It needs one complete route that a real team can use.
I would build it in this order:
- Establish the identity pattern, expose one AI API through APIM, apply baseline request and token limits, require ownership metadata and emit enough telemetry to attribute usage.
- Once real traffic exists, introduce model profiles, team-level quotas, dashboards and reusable policy fragments based on what consumers are actually doing.
- Use that evidence to tune limits, add semantic caching or backend failover where it is justified, close unintended direct-access paths and strengthen controls that are proving useful.
APIM supports semantic caching for compatible LLM APIs and backend pools with circuit breakers, but I would add either because there is an observed need rather than because the feature exists.
The order matters more than an arbitrary 30, 60 or 90-day plan. Visibility should exist before elaborate allocation models, and reusable defaults should exist before the exception list becomes longer than the platform documentation.
Where Azure AI Landing Zones tend to go wrong
Allowing direct model access to become the normal route weakens the platform quickly. Exceptions will exist, but if most workloads can bypass the gateway, runtime governance becomes optional and usage telemetry will always be incomplete.
Another common problem is expecting Azure Policy to provide runtime AI governance. Azure Policy can stop or audit an unapproved model deployment, but it does not manage how many tokens an application is consuming through an API once that model is running. Those are separate control points.
The opposite problem appears when APIM starts owning too much. The gateway should provide common runtime controls without making the platform team responsible for every prompt, index, retrieval flow and application failure.
Central prompt and response logging can create another problem by quietly building a sensitive data store that nobody planned to operate. I would collect the metadata the platform needs and make deeper content logging a workload decision.
Finally, the approval path has to remain usable. If trying a different approved model or requesting a temporary quota increase requires several meetings, the platform is creating an incentive to work around it. The control may technically exist, but the operating model around it is failing.
What are legitimate exceptions to routing through APIM?
I would treat exceptions as architecture decisions rather than informal team preferences.
Examples might include a protocol or network path that APIM cannot support appropriately, platform control-plane operations that are not application API traffic, or a workload with latency or throughput requirements that have been measured and shown to be incompatible with the gateway path.
The exception should still have an owner and an alternative way to provide the identity, cost and operational evidence the central gateway would otherwise have supplied.
Should every team share one Microsoft Foundry or Azure OpenAI resource?
I would not make that a blanket rule.
Shared resources can work well where teams have similar data boundaries, regional requirements, capacity needs and operational expectations. Workload-owned resources become easier to reason about when those requirements differ, particularly when teams need independent scaling, deployment lifecycles or stronger isolation.
The landing zone should define how both patterns are governed rather than forcing everything through one central AI resource.
If you’re evaluating where to start, the design checklist and the two solution accelerators below are worth reading side by side, since they take slightly different positions on how much of this the gateway should own versus what belongs in workload-owned resources:
- Azure AI Landing Zone design checklist
- Accelerating Enterprise AI Adoption with Azure AI Landing Zone
- AI Hub Gateway Solution Accelerator
- AI Hub Gateway full deployment guide
- Citadel Governance Hub component architecture
If I were building this again now, I would start by getting one real consumer through Entra ID, APIM, an approved model backend and attributable token telemetry. Once that path is easier than direct access and you can see exactly who is consuming what, you have something worth scaling.