AI-103 Manage, Monitor and Secure Foundry AI Systems: Quotas, Cost, Evaluation, Search Health and Keyless Access

Model quota in Foundry is a pool of tokens per minute (TPM) held per subscription, region, model and deployment type. When you create a deployment you assign it part of that pool, and the TPM you assign sets both its token limit and a proportional requests per minute (RPM) limit. The pool belongs to the subscription in that region, so creating another Foundry resource or using another resource group in the same region does not give you new quota, and spare TPM for one model cannot be moved to a different model.

When the pool is used up you have two choices:

• Rebalance — lower the TPM on an existing deployment of the same model, region and deployment type, and give the freed capacity to the new deployment. No approval is needed, and the change propagates within minutes (Microsoft says to allow up to 15).
• Request an increase — submit a quota request, which takes time to approve.

To see quota allocation and usage across regions without broad read access, assign Cognitive Services Usages Reader at the subscription scope. The usages API only honours a subscription-scope assignment, so the same role on a resource group or resource doesn't work, and the general Reader role works but exposes every resource in the subscription.

Why calls get HTTP 429

The rate limiter decides whether to admit a request when it arrives, before any tokens are generated. It estimates the request's cost as prompt tokens plus max_tokens. A deployment can therefore throttle while its processed-token metrics look low: if every request sets max_tokens far above the real output size, each one is counted as much larger than it is. Setting max_tokens close to the expected output length is the first fix, before buying more quota.

RPM is not checked once per minute. It is enforced over short windows of roughly 1 to 10 seconds, so a burst at the start of each batch can be throttled even when the per-minute total is under the limit. Two causes are on the platform side rather than yours: system-wide capacity throttling, and a temporary reduction of the rate limit on shared Standard capacity, visible when the x-ratelimit-limit-tokens response header comes back lower than the TPM you configured — retry with backoff covers both. The fixes for bursty traffic are client-side:

• Pace the requests — smooth the send rate so no few-second window gets a burst, rather than firing each batch at once.
• Retry properly — wait for the retry-after-ms (or retry-after) header, then retry with exponential backoff and jitter. The OpenAI and Azure SDKs retry automatically, controlled by max_retries; setting it to 0 turns every throttle into a failure, while a very high value just re-sends requests that still count against the limit.

The deployment type is the biggest lever on a workload's cost footprint and throughput. Standard types share capacity and bill per token; provisioned types reserve capacity for a fixed hourly price; batch types run asynchronously at a discount.

Two consequences matter. A second Standard deployment of the same model draws from the same TPM pool as the first, so it doesn't isolate a large job from online traffic — a Batch deployment does, because batch quota is separate. And sizing PTUs for an occasional peak means paying for that capacity all month.

Spillover for provisioned deployments

Spillover lets a provisioned deployment send requests it can't serve (429s when the PTUs are exhausted, plus some 400 errors for over-length context and 500/503 server errors) to a standard deployment, so only the overflow is billed per token. It can be enabled two ways:

• On the deployment — set the spilloverDeploymentName property on the provisioned deployment. It applies to all requests and needs no client changes.
• Per request — send the x-ms-spillover-deployment header, which requires changing the calling code.

If both are set, the deployment property wins over the header.

The target must be a pay-as-you-go (standard) deployment in the same Foundry resource, running the same model and model version as the provisioned deployment. A deployment in another resource or at an older version is rejected; the header has the same rules as the property. To measure how much traffic spilled over, chart the Azure OpenAI Requests metric on the standard deployment split by the IsSpillover dimension — spilled requests are recorded there, and they are not counted as 429s on the provisioned deployment.

Microsoft Cost Management is where model and agent spend is reported. Cost analysis can be grouped by resource and by meter; meters separate cost by model and by input versus output tokens, but not by who generated it.

When several teams share one Foundry resource, each in its own project, Foundry automatically tags usage with the project, and filtering cost analysis by that tag gives per-project spend for chargeback without anyone changing how they deploy or call models. This project-level attribution is in preview and covers models sold directly by Azure, such as Azure OpenAI models. Adding your own tags to deployments also works, but changes how teams create them.

Budgets watch spend for a scope (subscription, resource group and so on) and send alerts on actual or forecasted cost. They notify — they never stop, throttle or cap resources, and Azure OpenAI has no hard spending limit. If you need a spend ceiling, the only built-in enforcement lever is capacity (a budget alert can trigger your own automation through an action group, but that is custom work): lowering TPM quota caps throughput, which shows up to users as 429s. A Cost Management scheduled alert is different again: it emails a saved cost view on a schedule, with no threshold.

Performance monitoring in Foundry rests on Application Insights: connect an Application Insights resource to the project, and agent telemetry, traces and evaluation results flow into it. Without that connection the Agent Monitoring Dashboard (the agent's Monitor tab) has no data source. The dashboard shows token usage, latency, run success rate, evaluation scores and red-team results, and its settings let you configure alerts, including alerts on evaluation scores.

Two kinds of evaluation feed it:

• Continuous evaluation — a rule that samples live agent responses as they complete and scores them with chosen evaluators. The rule runs as the project's managed identity, which needs the Foundry User role on the Foundry resource; if telemetry shows but evaluation results never appear, check that role first. The rule's max_hourly_runs (default 100) only limits how many runs are evaluated per hour.
• Scheduled evaluation — runs on a fixed recurrence and can use a dataset as its source, so every run scores the same curated questions. This is how you detect slow quality drift: the inputs stay fixed, so a falling score reflects a changed system, not changed traffic.

For one slow or failing run, use tracing: traces record each span (model calls, tool calls, their inputs and outputs, and durations) in the connected Application Insights. The dashboard and metrics show trends across many runs; tracing explains a single run.

Model version upgrades as a source of drift

A deployment's model version can change underneath an app, which alters tone, format and quality. Each deployment has a version upgrade policy:

The api-version in client code is the request contract, not the model version, so pinning it doesn't stop a model upgrade. Pair the upgrade policy with a scheduled evaluation on a golden dataset so a version change is caught by a score, not by users.

Safety events show up in two places: in the responses your app receives when the content filter acts, and in safety evaluation scores computed over traffic.

In code, the content filter acts differently on the two sides of a non-streaming chat completion call:

To log every filter action, detect the first two conditions. To monitor safety in production at the level of content, add a safety evaluator (for example the violence evaluator, builtin.violence) to a continuous evaluation rule, which scores sampled live responses, and configure an alert on evaluation scores so the team hears when they worsen. A red-team scan probes the agent with synthetic adversarial prompts and a scheduled evaluation scores a fixed dataset — neither scores real production responses.

Monitoring doesn't change what is blocked. Configuring guardrails, filter categories and severity thresholds is part of implementing responsible AI (the next topic in this domain).

Grounding quality is measured with Foundry's RAG evaluators. The deciding question is whether you have ground truth — expected answers or relevance labels. Production traces usually don't, so monitoring relies on the evaluators that need none.

Read the pairs carefully. Groundedness and Relevance both judge the response, but only Groundedness checks it against the context. Retrieval and Document Retrieval both judge search, but Retrieval uses an LLM judge with no labels while Document Retrieval computes ranking metrics from labels. Response Completeness and Similarity both use the expected answer, but only Completeness asks whether every required fact is present. Coherence and fluency score how the answer reads, not whether it is grounded.

For an Azure AI Search index that grounds an app or agent, ingestion quality is monitored through the indexer, and service health through Azure Monitor.

Indexer status and execution history

Each indexer keeps an execution history of recent runs (up to 50), and each run lists its errors and warnings per document, with the document key and message, plus counts such as items processed and failed. A run can report Success while individual documents failed, as long as the failures stayed within the allowed limit — so a "successful" run with missing documents is diagnosed from that run's errors. The activity log records control-plane operations, and the documents-processed metric counts documents, so neither names the failed files; Search explorer queries the index, where failed documents never arrived.

The maxFailedItems parameter sets how many document failures a run tolerates. The default is 0, so the first unreadable file fails the whole run. -1 means no limit: the indexer processes every readable file and still records the failures in the execution history. Resetting an indexer clears change tracking so everything is reprocessed; it doesn't change failure tolerance.

Service health: metrics, logs and capacity

• Platform metrics — such as search latency, throttled search queries percentage and storage usage — are collected automatically, appear in Metrics explorer and can drive alerts.
• Resource logs (per-request operation name, duration and so on) are only stored and queryable with KQL after you create a diagnostic setting that routes them to a destination such as a Log Analytics workspace.

Vector index quota is allocated per partition, and the Usage tab reports it separately from storage. Adding partitions raises it but adds cost. To keep the bill flat, compress the vectors: scalar quantization (int8) cuts the in-memory size of float32 vectors about fourfold, applied by defining a new compressed vector field and reindexing. Exhaustive KNN fields use no vector quota, but they scan every vector per query, so they don't stay fast at large scale.

Relevance is diagnosed from the scores Azure AI Search returns with each result, and from the subscores you can ask it to expose.

• @search.score — in a hybrid query this is the Reciprocal Rank Fusion (RRF) score that merges the keyword and vector result lists. Its range depends on how many queries are fused, so a fixed cutoff on it shifts when you add or remove a vector query.
• @search.rerankerScore — returned when the semantic ranker reranks results. It sits on a fixed 0 to 4 scale, so it is the stable value for a "drop weak chunks" threshold before calling the model.
• @search.captions and @search.answers — text extracted by the semantic ranker (per-result passages and query-level answers), not numeric relevance values.

To tune a hybrid query, add the debug parameter (vector, semantic or all). It returns the subscores behind each result — for example how each vector query contributed — which is what you need to set vector weights or thresholds. Other parameters change results rather than explain them: queryType: semantic turns on reranking, vectorFilterMode decides whether filters run before or after the vector search, and count only returns the number of matches.

Relevance should also be tracked over time with the Retrieval evaluator (no labels) or the Document Retrieval evaluator (with labels), covered above.

Keyless authentication means the app proves its identity with a Microsoft Entra ID token instead of an API key, and the Foundry resource authorizes it through Azure RBAC. It needs three things: an identity for the app (normally a managed identity), a data-plane role on the Foundry resource or project, and — to reject key-based calls outright — disableLocalAuth set to true on the resource. Storing a key in Key Vault, rotating keys or restricting the network all leave key authentication enabled. Entra ID authentication also needs the resource's custom subdomain endpoint, which Foundry resources have by default.

Tokens for Foundry data-plane calls are requested for the scope https://ai.azure.com/.default. Other scopes belong to other services: https://management.azure.com/.default is Azure Resource Manager, https://search.azure.com/.default is Azure AI Search, https://vault.azure.net/.default is Key Vault.

from openai import OpenAI
from azure.identity import DefaultAzureCredential, get_bearer_token_provider

token_provider = get_bearer_token_provider(
    DefaultAzureCredential(), "https://ai.azure.com/.default"
)
client = OpenAI(
    base_url="https://<resource>.openai.azure.com/openai/v1/",
    api_key=token_provider,
)

The Foundry SDK takes a token credential directly: AIProjectClient(endpoint=..., credential=DefaultAzureCredential()). DefaultAzureCredential tries a chain of sources, so the same code uses the managed identity in Azure and the developer's Azure CLI login on a laptop. ManagedIdentityCredential works only in Azure, AzureCliCredential only where someone is signed in to the CLI, and AzureKeyCredential is a key.

System-assigned or user-assigned identity

When an app has both, DefaultAzureCredential uses the system-assigned identity unless the AZURE_CLIENT_ID setting holds the user-assigned identity's client ID (not its principal/object ID). A PermissionDenied error when only the user-assigned identity holds the role is the classic symptom.

Least-privilege roles

Microsoft advises using the Foundry roles rather than the older Cognitive Services roles (such as Cognitive Services OpenAI User) for Foundry work; the Usages Reader role for quota is the exception. Search data-plane roles only apply when role-based access is enabled on the search service.

Private networking keeps traffic to and from AI services off the public internet. Inbound and outbound paths are configured separately, and each service has its own mechanism.

Inbound to Foundry: private endpoints and DNS

Disable public network access on the Foundry resource and create a private endpoint in a virtual network. Clients then reach the private IP only if DNS resolves the endpoint name to it, through the private DNS zones privatelink.cognitiveservices.azure.com, privatelink.openai.azure.com and privatelink.services.ai.azure.com, linked to the virtual network. Azure-hosted clients that use Azure DNS resolve correctly once the zones are linked to their network. On-premises clients can't query Azure DNS (168.63.129.16) directly, so their DNS server needs a conditional forwarder for the Foundry zones (cognitiveservices.azure.com, openai.azure.com, services.ai.azure.com) to an Azure DNS Private Resolver inbound endpoint in a virtual network linked to the private zones. A request that resolves to a public IP is a DNS problem, not a firewall one.

Outbound from agents: virtual network injection

For agents whose tools (an OpenAPI tool, an MCP server) are reachable only inside a company network, the Foundry Agent Service supports virtual network injection, so agent traffic leaves through your virtual network. It needs a dedicated subnet delegated to Microsoft.App/environments, sized /27 or larger, and it is chosen when the Foundry resource is created — Microsoft documents adding it to an existing account as unsupported, so plan it up front. /27 is the minimum; Microsoft recommends /24 for production, because a /27 can run out of addresses as agents scale. A private endpoint only secures inbound access and gives agents no outbound path.

Azure AI Search indexers reaching protected data

A private endpoint for the search service is inbound traffic to search; it doesn't give the indexer a route to storage. The managed identity still needs a data role such as Storage Blob Data Reader — but more permissions never fix a network path.