One front door for every model: building an Azure AI gateway with API Management
How to put Azure API Management in front of Azure OpenAI and Microsoft Foundry models so that identity, private networking, token budgets, failover and telemetry are properties of the platform instead of something each team has to remember.
- Enterprise AI
- Azure
- API Management
- Architecture
Why models need a gateway
Most organisations start using generative AI the same way. A team creates an Azure OpenAI resource, copies a key into an app setting, and ships a prototype. The second team does the same thing, then the third. Six months later there are a dozen model deployments, a few dozen keys, no shared view of who is spending what, and nobody who can say with confidence that a prompt containing client data has never left the private network.
None of those teams did anything wrong. Calling the endpoint directly is the fastest way to get started. The trouble is that a direct connection gives you nowhere to put the controls an enterprise needs once usage grows:
- Identity. Who is calling, and are they allowed to use this model?
- Capacity. How do you stop one noisy workload consuming the tokens-per-minute (TPM) quota that five other workloads depend on?
- Resilience. What happens when a region is throttled or a deployment is unavailable?
- Safety. Where do content checks and prompt-attack detection run, and can a team switch them off?
- Cost. Which team, product or customer consumed those tokens?
An AI gateway answers all of these in one place. On Azure, that gateway is Azure API Management (APIM). Microsoft describes the AI gateway as an extension of APIM’s existing API gateway, so the same product most enterprises already use for REST APIs now understands tokens, models, MCP tools and agents.
The rest of this post covers how I would design it, with the policies that do the work.
The same gateway for Azure OpenAI and Microsoft Foundry
A common question is whether you need a different design for Azure OpenAI models and for the wider catalogue in Microsoft Foundry: Llama, Mistral, DeepSeek, Claude and the rest. You don’t.
APIM’s LLM policies work with any model API that follows the OpenAI Chat
Completions or Responses schema. In the v2 tiers they also support the
Anthropic Messages API, and they support the Google Vertex AI API.
The capabilities documentation
lists models in Microsoft Foundry and non-Microsoft providers such as Amazon
Bedrock as valid backends. The llm-* policy family (llm-token-limit,
llm-emit-token-metric, llm-content-safety, llm-semantic-cache-*) applies
to all of them. The older azure-openai-* policies still exist, but for new
work the llm-* versions are the ones to standardise on, because they don’t tie
your policy library to a single provider.
There is also a unified model API (in preview at the time of writing). It exposes several backends from different providers through one OpenAI-compatible endpoint, translates request formats, and lets you apply governance once. If you expect to run more than one model family, it is worth evaluating before you build a separate API per provider.
The practical result is that the application targets one stable endpoint and a stable model alias. Whether that alias resolves to a GPT deployment in Azure OpenAI or an open-weight model in Foundry is a platform decision, made at the gateway, without a code change in the consuming app.
Reference architecture
The diagram at the top of this post shows the path every request takes. The caller authenticates to APIM with an Entra ID token. APIM validates it, checks the prompt with Azure AI Content Safety, tries the semantic cache, charges the caller’s token budget, and forwards the request to a pool of model backends using its own managed identity. On the way back it records token usage against the caller.
The dashed lines are the two exception paths. A cache hit returns to the caller
before any model capacity is spent. When the PTU deployment returns 429, the
pool spills the request over to pay-as-you-go capacity in another region.
Security: making the governed path the only path
No keys anywhere in the application estate
There are two hops, and neither of them should use an API key.
Caller to gateway. Applications use their own Entra workload identity (managed identity on App Service, AKS workload identity, and so on) to get a token for the gateway’s app registration. APIM validates it:
<validate-azure-ad-token tenant-id="{{tenant-id}}"
header-name="Authorization"
failed-validation-httpcode="401"
failed-validation-error-message="Unauthorised">
<audiences>
<audience>api://ai-gateway</audience>
</audiences>
<client-application-ids>
<application-id>{{claims-triage-app-id}}</application-id>
<application-id>{{support-copilot-app-id}}</application-id>
</client-application-ids>
</validate-azure-ad-token>
APIM subscription keys are still useful as a product and quota boundary, but they shouldn’t be the only thing standing between the internet and your models.
Gateway to model. APIM authenticates to Azure OpenAI and Foundry with its
own managed identity, holding the Cognitive Services User role on the
target resources:
<authentication-managed-identity resource="https://cognitiveservices.azure.com/" />
Once this is in place, disable local (key) authentication on the model resources. That is the step that closes the bypass. If keys still work, someone will eventually use one.
Keep traffic private
Put the model resources behind private endpoints and turn off their public network access. APIM then needs to reach them privately. The Foundry AI Gateway documentation is explicit about the options: a Standard v2 or Premium v2 instance with a private endpoint, or a Premium v2 instance injected into a virtual network.
For most regulated estates, I would run Premium v2 with VNet injection in the hub, so that both inbound traffic from spokes and outbound traffic to models stay on the private network. If the gateway also serves external consumers, put Application Gateway with WAF in front of it.
Content safety as policy, not configuration
Content filters on the model deployment are a good baseline, but they belong to whoever owns that deployment. Applying Azure AI Content Safety at the gateway means the check runs for every model, including non-Microsoft ones, and a workload team cannot opt out of it:
<llm-content-safety backend-id="content-safety" shield-prompt="true">
<categories output-type="EightSeverityLevels">
<category name="Hate" threshold="4" />
<category name="Violence" threshold="4" />
<category name="SelfHarm" threshold="4" />
<category name="Sexual" threshold="4" />
</categories>
</llm-content-safety>
shield-prompt="true" turns on Prompt Shields, which detects jailbreak and
prompt-injection attempts. A blocked request returns 403.
One behaviour catches teams out. With streaming responses, the
policy reference
notes that APIM checks the stream in a sliding window and stops forwarding
events when it finds a violation. It does not return a 403 in that case, so
the client sees a truncated stream. Client code has to handle a stream that
ends early, and the platform team has to make that visible in telemetry.
Scalability: capacity is tokens, not requests
Traditional API gateways throttle requests. That is the wrong unit for language models. One request might cost 200 tokens and the next 60,000. The quota you buy from Azure is expressed in TPM, so the gateway has to manage tokens.
Token budgets per consumer
llm-token-limit enforces a per-minute rate, a quota over a period, or both,
against any key you choose. Keying on the calling application’s identity means
the budget follows the workload, not the subscription key it happens to hold:
<llm-token-limit
counter-key="@(context.Request.Headers.GetValueOrDefault("Authorization","").AsJwt()?.Claims.GetValueOrDefault("appid", "unknown"))"
tokens-per-minute="20000"
token-quota="50000000"
token-quota-period="Monthly"
estimate-prompt-tokens="true"
remaining-tokens-header-name="x-ratelimit-remaining-tokens"
tokens-consumed-header-name="x-tokens-consumed" />
The two limits fail differently, and clients should treat them differently.
Exceeding the rate returns 429 with a Retry-After header, which means “back
off and try again”. Exceeding the quota returns 403, which means “this
workload has used its allocation for the period”. A client that retries a 403
is wasting its time.
Three details from the policy documentation matter at scale:
- Streaming is always estimated. With
stream: true, prompt and completion tokens are estimated rather than read from the response. - Concurrent requests can overshoot. The real token count is only known when the response arrives, so a burst of parallel requests can briefly exceed the limit before blocking starts.
- Counters are per gateway. In a multi-region APIM deployment, each regional gateway counts independently. A 20,000 TPM limit across three regions is, in practice, up to 60,000. Size limits for that, or route each consumer to a home region.
Backend pools: spend PTU first, spill to pay-as-you-go
Provisioned Throughput Units (PTU) are paid for whether you use them or not, so you want them saturated before you pay per token anywhere else. APIM backend pools support round-robin, weighted, priority-based and session-aware balancing. Priority is the one that matches PTU economics:
resource ptuUkSouth 'Microsoft.ApiManagement/service/backends@2024-06-01-preview' = {
parent: apim
name: 'aoai-ptu-uksouth'
properties: {
url: 'https://aoai-ptu-uksouth.openai.azure.com/openai'
protocol: 'http'
circuitBreaker: {
rules: [
{
name: 'throttled'
failureCondition: {
count: 1
interval: 'PT1M'
statusCodeRanges: [ { min: 429, max: 429 } ]
}
tripDuration: 'PT1M'
acceptRetryAfter: true
}
]
}
}
}
resource modelPool 'Microsoft.ApiManagement/service/backends@2024-06-01-preview' = {
parent: apim
name: 'gpt-pool'
properties: {
type: 'Pool'
pool: {
services: [
{ id: '/backends/aoai-ptu-uksouth', priority: 1 }
{ id: '/backends/aoai-paygo-swedencentral', priority: 2, weight: 1 }
{ id: '/backends/aoai-paygo-francecentral', priority: 2, weight: 1 }
]
}
}
}
When the PTU deployment is saturated it returns 429. The circuit breaker
trips, and with acceptRetryAfter: true it stays open for exactly as long as
the backend asked, then closes again. While it is open, traffic flows to the
priority 2 backends. Pair this with a retry in the policy so that the request
which caused the trip is retried against the next backend instead of failing
back to the caller:
<backend>
<retry condition="@(context.Response.StatusCode == 429 || context.Response.StatusCode >= 500)"
count="2" interval="0" first-fast-retry="true">
<forward-request buffer-request-body="true" />
</retry>
</backend>
The same pattern gives you regional resilience. A pool spread across regions keeps answering when one region is throttled or degraded, and the application does not need to know any of this happened. Check data residency before you add a region to a pool: failover to a region your contracts don’t allow is a compliance incident, not a resilience win.
Semantic caching
A lot of enterprise traffic repeats itself: the same policy question asked hundreds of times a day in slightly different words. Semantic caching embeds each prompt, compares it with recent prompts in Azure Managed Redis, and returns the stored completion when they are close enough. The lookup goes in the inbound section and the store goes in the outbound section:
<llm-semantic-cache-lookup
score-threshold="0.10"
embeddings-backend-id="embeddings"
embeddings-backend-auth="system-assigned"
ignore-system-messages="true"
max-message-count="10">
<vary-by>@(context.Request.Headers.GetValueOrDefault("Authorization","").AsJwt()?.Claims.GetValueOrDefault("appid", "unknown"))</vary-by>
</llm-semantic-cache-lookup>
<llm-semantic-cache-store duration="3600" />
The vary-by element is the security control here. Without it, one
application can be served a completion generated for another. Partition the
cache by application at minimum, and by tenant or user where answers depend on
who is asking. For RAG workloads with document-level permissions, a shared
cache can return content the second user isn’t allowed to see. Either key the
cache on the user or leave those routes uncached.
Treat score-threshold as something you tune against real traffic. A lower
value demands closer matches. Start strict, measure the hit rate, and loosen
only while answer quality holds.
Scaling the gateway itself
APIM adds scale units manually or automatically depending on tier, and Premium tiers support additional regional gateways. Put model backends in the same regions as the gateways that call them. A gateway in UK South forwarding everything to Sweden Central adds latency to every token and creates a cross-region dependency you didn’t intend.
Observability: the record that makes cost and audit possible
Every request should leave behind one structured record: which workload called,
which model answered, prompt and completion tokens, latency, and whether a
policy intervened. llm-emit-token-metric sends token counts to Application
Insights with whatever dimensions you need for chargeback:
<llm-emit-token-metric namespace="ai-gateway">
<dimension name="Workload" value="@(context.Request.Headers.GetValueOrDefault("Authorization","").AsJwt()?.Claims.GetValueOrDefault("appid", "unknown"))" />
<dimension name="API ID" />
<dimension name="Cost Centre" value="@(context.Request.Headers.GetValueOrDefault("x-cost-centre", "unassigned"))" />
</llm-emit-token-metric>
With that metric in place, cost becomes something you can query by team, by product and by model, and you can alert before a workload overspends instead of finding out from the invoice.
APIM can also log prompts and completions to Azure Monitor. Decide this deliberately. Full payload logging is valuable for audit and evaluation, but it creates a second copy of potentially sensitive data with its own retention, access and residency obligations. For most workloads I log token counts and metadata everywhere, and full payloads only where a specific requirement justifies it.
Putting the policy together
Order matters. Reject unauthenticated and unsafe requests before they cost anything, serve from cache before charging the budget, and only then spend model capacity:
<policies>
<inbound>
<base />
<validate-azure-ad-token tenant-id="{{tenant-id}}" header-name="Authorization">
<audiences><audience>api://ai-gateway</audience></audiences>
</validate-azure-ad-token>
<llm-content-safety backend-id="content-safety" shield-prompt="true">
<categories output-type="EightSeverityLevels">
<category name="Hate" threshold="4" />
<category name="Violence" threshold="4" />
</categories>
</llm-content-safety>
<llm-semantic-cache-lookup score-threshold="0.10"
embeddings-backend-id="embeddings" embeddings-backend-auth="system-assigned">
<vary-by>@(context.Request.Headers.GetValueOrDefault("Authorization","").AsJwt()?.Claims.GetValueOrDefault("appid", "unknown"))</vary-by>
</llm-semantic-cache-lookup>
<llm-token-limit
counter-key="@(context.Request.Headers.GetValueOrDefault("Authorization","").AsJwt()?.Claims.GetValueOrDefault("appid", "unknown"))"
tokens-per-minute="20000" estimate-prompt-tokens="true" />
<llm-emit-token-metric namespace="ai-gateway">
<dimension name="API ID" />
</llm-emit-token-metric>
<authentication-managed-identity resource="https://cognitiveservices.azure.com/" />
<set-backend-service backend-id="gpt-pool" />
</inbound>
<backend>
<retry condition="@(context.Response.StatusCode == 429 || context.Response.StatusCode >= 500)"
count="2" interval="0" first-fast-retry="true">
<forward-request buffer-request-body="true" />
</retry>
</backend>
<outbound>
<base />
<llm-semantic-cache-store duration="3600" />
</outbound>
</policies>
Keep this in source control and deploy it through the same pipeline as the rest of the platform, using Bicep or Terraform with named values for the environment-specific parts. Onboarding a new workload should be a pull request that adds an app ID, a budget and a cost centre, not a ticket that waits for someone to click through the portal.
Choosing how to deploy it
There are now three ways to get an APIM-based AI gateway on Azure, and they suit different organisations.
A regular APIM instance (v2 tiers). This is the full product: every policy, custom expressions, enterprise networking, multi-region, and your existing REST APIs alongside AI traffic. If you already run APIM as a platform service, this is where the AI gateway belongs.
AI Gateway from the Foundry portal. Under Manage > AI Gateway, Foundry can
create a new APIM instance (Basic v2, aimed at development and testing) or
associate an existing v2 instance in the same subscription. Token limits and
quotas are then set per Foundry project, and all projects in the Foundry
resource share the gateway. Projects over the TPM limit get 429, and projects
over their quota get 403. This suits model and platform owners who mostly
live in Foundry. For production, Microsoft’s own guidance is to associate an
existing Standard v2 or Premium v2 instance rather than use the Basic v2 one it
creates. If you need hard isolation between groups of projects, use separate
Foundry resources, each with its own gateway.
The dedicated API Management AI Gateway tier. This is a newer resource type organised around providers, models, MCP ToolServers and runtime keys instead of general-purpose APIs. It is in public preview, and a recent walkthrough lists the caveats: no SLA, East US 2 and Sweden Central only, no announced pricing, and runtime callers using an access key instead of managed identity. It is worth exploring for agent platforms, but not yet something I would put a regulated workload on.
These options share the same data path. A Foundry-associated gateway is an APIM instance you can open in the Azure portal and extend with custom policy, so starting in Foundry doesn’t lock you out of the full product later.
Agents and tools go through the same door
Agents call tools as well as models, and an ungoverned MCP server is as much of a risk as an ungoverned model endpoint. APIM can expose existing REST APIs as MCP servers, pass through to remote MCP servers, and import A2A agent APIs. Foundry can register MCP tools and custom agents hosted anywhere into its control plane through the same gateway.
The principle doesn’t change. The agent authenticates to the gateway, the
gateway holds the credential for the backend tool, and the tool rejects anything
that didn’t come through the gateway. llm-content-safety also works on MCP
and A2A traffic, which matters once tool outputs start flowing back into
prompts.
What to get right on day one
If I had to reduce this to a checklist for a new platform:
- Disable key authentication on every model resource once APIM’s managed identity has access.
- Authenticate callers with Entra workload identities, and key budgets and metrics on the application identity.
- Put models behind private endpoints and keep the gateway on the private network.
- Apply content safety and Prompt Shields at the gateway so no workload can opt out.
- Set both a TPM limit and a monthly quota per workload, and teach clients that
429means retry and403means stop. - Use priority backend pools with circuit breakers so PTU is used first and failover is automatic.
- Partition the semantic cache by application, and by user where permissions differ.
- Emit token metrics with chargeback dimensions from the first request.
- Keep all of it in infrastructure as code.
The real benefit
The individual policies are useful, but the bigger change is organisational. Once every model call goes through one gateway, adding a new model, moving a workload from pay-as-you-go to PTU, adding a region or tightening a safety threshold are platform changes made in one place. Application teams keep calling the same endpoint with the same identity.
That is what lets generative AI scale safely in an enterprise. The governed path has to be the easiest one for teams to use, and once it is, they stop looking for ways around it.