AI-200 Container Apps and AKS: Revisions, KEDA Scaling, Manifests and Troubleshooting

An Azure Container Apps environment is the secure boundary that a group of container apps runs in: every app in it shares the same virtual network, writes to the same log destination (the environment's appLogsConfiguration, usually a Log Analytics workspace) and can use the same Dapr configuration. Put apps in one environment when they belong to one solution and call each other; put them in separate environments when they must be isolated.

Microsoft recommends separate environments when apps must never share the same compute resources, must not communicate through Dapr service invocation, or belong to different teams or stages such as test and production. Because the log destination (appLogsConfiguration) is also set once per environment, apps that must send logs to different workspaces end up in different environments too. Settings that live on the app, such as its resource group, workload profile or revision settings, never change what the environment shares.

Custom virtual networks are chosen at creation

An environment created with default networking uses a VNet that Azure manages for you. To reach resources behind private endpoints (an Azure Database for PostgreSQL flexible server, a storage account, a Key Vault) or to keep inbound traffic private, the environment must run on a subnet of your own virtual network. For workload profiles environments that subnet is delegated to Microsoft.App/environments. The VNet and the accessibility level are set when the environment is created and cannot be added to an existing environment later, so the supported path is to create a new environment and redeploy the apps into it.

• External environment: the environment's endpoint has a public IP address, so apps with external ingress are reachable from the internet.
• Internal environment: the environment has only an internal load balancer with a private IP in your VNet, so nothing in it is reachable from the internet, only from the VNet, peered networks and connected on-premises networks.

Ingress decides whether, and from where, an app accepts HTTP or TCP requests. External ingress accepts requests from outside the environment; internal ingress accepts them only from other apps in the same environment; disabled ingress creates no endpoint at all. A background worker that only pulls messages from a queue and makes outbound calls needs no endpoint, so disabling ingress gives it the smallest exposure.

Apps in the same environment call each other by app name: a request to http://orders-api resolves to that app's internal FQDN, so no IP addresses are hard-coded. Turning ingress off removes this path too, because there is nothing to call. External ingress with IP restrictions still leaves a public endpoint.

Two separate settings decide reachability, and they combine:

The general rule: ingress describes the boundary of the environment, accessibility describes the boundary of the network. In an internal environment, "external" ingress means "outside the environment, inside the network", never "public", while internal ingress always stops at the environment boundary.

Ingress also has a target port: the port the container listens on. Ingress forwards requests (and runs its default probes) on that port, so it must match what the app actually binds to.

A revision is an immutable snapshot of a container app version. Whether a change creates a new revision depends on where it lives in the app's resource definition:

Application-scope changes apply to every revision at once. That has a consequence for secrets: updating a secret value (for example with az containerapp secret set) creates no revision, and running containers don't see the new value. Microsoft notes that revisions must be restarted before containers recognise new secret values, so after rotating a password referenced through secretref, run az containerapp revision restart on the active revision. As a rule, a running container only sees a new secret value when it is restarted or replaced; other application-scope edits leave the existing replicas running as they were.

Key Vault references and automatic rotation

A secret can be a reference to Azure Key Vault, read with the app's managed identity. The URI decides which version the app gets:

• A URI that ends with a version ID (.../secrets/search-key/4b9e...) always resolves to that one version, so rotations in Key Vault never reach the app.
• A versionless URI (.../secrets/search-key) resolves to the latest version. Container Apps retrieves new versions automatically within 30 minutes and restarts the active revisions that use the secret in an environment variable, so rotations need no manual step.

Which identity reads the reference (system- or user-assigned) doesn't change the version a URI returns.

Container Apps has two revision modes. In single revision mode (the default) a new revision receives all traffic as soon as it is ready and the previous revision is deprovisioned. In multiple revision mode several revisions stay active and ingress traffic weights decide each one's share of the app's URL. Weighted splits for canary or blue-green releases are only possible in multiple revision mode, so switching the mode is the prerequisite.

az containerapp revision set-mode -n api -g rg-ai --mode multiple
az containerapp ingress traffic set -n api -g rg-ai \
  --revision-weight api--v1=90 api--v2=10

Rollback in multiple mode is a weight change: because the old revision is still active, giving it 100 percent restores the previous behaviour within seconds, with nothing to build or deploy. Fast rollback depends on the previous revision still being active, which is exactly what multiple revision mode provides.

The "latest revision" trap

Traffic can be assigned to a revision by name or to latestRevision. If the weight points at the latest revision, any newly deployed revision becomes the latest and takes that traffic as soon as it is ready. To hold production on a known revision during a release, assign the weight to it by name first.

Labels vs per-revision FQDNs

For testers who need one URL that always reaches the release candidate, assign a label (for example green) to the candidate and move it each release, while production weights stay 100 percent on the current revision. Keep the distinction general: a traffic weight always routes a share of real users, while a label routes only the callers who use the label URL.

Container Apps scales each revision between minReplicas and maxReplicas (defaults 0 and 10) using scale rules. There are three rule types: HTTP (concurrent requests), TCP (concurrent connections) and custom rules, which are built on KEDA scalers such as azure-servicebus, azure-eventhub, azure-queue, kafka, cpu and memory. Event scalers like Service Bus can scale an app from zero and back to zero; CPU and memory rules cannot scale to zero.

How the replica count is calculated

KEDA polls the event source (every 30 seconds by default) and computes desiredReplicas = ceil(currentMetricValue / targetMetricValue), clamped between the minimum and maximum. With 180 waiting messages and messageCount: 20, the target is 9 replicas, provided maxReplicas is at least 9. Scale-out happens in steps (1, 4, 8, 16, 32 ...) until the calculated count is reached, so an intermediate number is not where scaling settles. When the load drops, scale-in is governed by a scale-down stabilization window (300 seconds); the 300-second cool-down period applies to the final step, from the last replica down to 0.

Two traps with defaults

• No scale rule means the default HTTP rule applies. An app with ingress disabled never receives HTTP traffic, so it scales to zero and never starts again. Microsoft warns about this: a queue worker needs a custom rule for its queue.
• Cold starts: with a minimum of 0, an idle HTTP app scales to zero and the next request waits for a replica to start. Setting minReplicas to 1 keeps one replica warm (it can be billed at the lower idle rate while it isn't processing) while the HTTP rule still scales out under load. If the requirement is "cost nothing while idle", keep the minimum at 0.

Authenticating a custom scale rule

A scaler that reads an event source needs credentials. Two options:

• Managed identity (supported for Azure Service Bus, Event Hubs and Queue Storage scalers): pass --scale-rule-identity with the user-assigned identity's resource ID, or system, in az containerapp create/update; in ARM or Bicep the rule's custom block takes an identity property. No connection string is stored. The rule itself must name the identity it authenticates with; an identity attached to the app for another purpose is not used by the scaler unless the rule references it.
• Secrets: the rule's auth array maps a triggerParameter (the scaler setting to fill) to a secretRef (the name of a Container Apps secret); in the CLI that is --scale-rule-auth "connection=sb-connection".

When converting a KEDA TriggerAuthentication, each secretTargetRef becomes one auth entry: its parameter becomes triggerParameter, and its key is the value to store in a Container Apps secret that secretRef names. The Kubernetes Secret object's name has no equivalent in Container Apps.

"rules": [{
  "name": "rabbit",
  "custom": {
    "type": "rabbitmq",
    "metadata": { "queueName": "jobs", "mode": "QueueLength", "value": "25" },
    "auth": [{
      "secretRef": "rabbit-conn",
      "triggerParameter": "host"
    }]
  }
}]

On Azure Kubernetes Service you declare workloads in YAML manifests and apply them with kubectl apply -f. The first decision is the workload kind:

In apps/v1, a Deployment's spec.selector is required and must match spec.template.metadata.labels; otherwise the API server rejects the object with "selector does not match template labels". The Deployment's own metadata.labels label only the object and are not part of that check.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rerank-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: rerank-api
  template:
    metadata:
      labels:
        app: rerank-api       # must match the selector
    spec:
      containers:
      - name: api
        image: contoso.azurecr.io/rerank-api:2.1
        ports:
        - containerPort: 9000
        resources:
          requests: { cpu: "500m", memory: "512Mi" }
          limits:   { memory: "1Gi" }

Requests reserve capacity for scheduling; limits cap what the container may use. A container that exceeds its memory limit is killed (OOMKilled, exit code 137) even when the node has free memory, so the fix is a higher limit or lower memory use, not more nodes or a higher request.

Consuming ConfigMaps and Secrets

A common alternative is a pod-template annotation holding a hash of the ConfigMap, but changing it starts a new rollout, which restarts the pods.

Pulling images from Azure Container Registry

The kubelet on each node pulls images, using the cluster's kubelet (agent pool) managed identity. az aks update --attach-acr <registry> (or az aks create --attach-acr) assigns that identity the AcrPull role on the registry in one step, with no stored password. AcrPull is the least-privilege role: AcrPush adds push and Contributor manages the registry resource. The general rule is to grant the role to the identity that actually performs the pull, which on AKS is the kubelet identity, and to prefer it over any approach that stores a registry password. If pods still report ImagePullBackOff, az aks check-acr --acr <name>.azurecr.io runs a check from the cluster's nodes and reports whether DNS resolution, network access or authentication fails. (az acr check-health only checks the machine you run it on.)

A Kubernetes Service gives a set of pods one stable address. It routes only to pods that are Ready and whose labels match its selector; those pods appear as its endpoints (EndpointSlices).

• port is what clients connect to on the Service; targetPort is the port on the pods traffic is forwarded to. It must match the port the container listens on (or name a containerPort).
• Empty endpoints (kubectl get endpoints shows <none>) while pods are Running and Ready means the selector matches no pod labels: fix the selector or the labels.
• Endpoints present but "connection refused" means the selector works but nothing listens on targetPort inside the containers: fix targetPort.

Kubernetes has three probes, and each controls a different thing:

A startup probe with periodSeconds: 5 and failureThreshold: 36 allows up to 3 minutes of startup (period × threshold), while the liveness probe keeps tight timings for hangs afterwards. Stretching the liveness probe's initialDelaySeconds or failureThreshold instead weakens hang detection. Never put a shared dependency in a liveness check: when the dependency is slow, every pod restarts at once. Check it in readiness so pods stop taking traffic but keep running.

Managing rollouts

• kubectl rollout status deployment/summarize-api --timeout=3m waits until the new pods are updated and available and exits non-zero on timeout, so a pipeline stage can depend on it.
• kubectl rollout undo deployment/translate-api returns to the previous revision (ReplicaSet), or one named with --to-revision, without editing the manifest. Fix the manifest afterwards, or the next apply reintroduces the bad image.
• kubectl rollout restart starts a fresh rollout of the same template; kubectl rollout history lists revisions.

The default RollingUpdate strategy uses maxSurge (pods allowed above the desired count) and maxUnavailable (desired pods allowed to be unavailable). maxUnavailable: 0 with maxSurge: 1 keeps full capacity and adds one new pod at a time; both set to 0 is rejected by the API. Recreate stops all old pods first, which takes the service down. A readiness probe is what makes rolling updates wait for new pods to be able to serve.

Start from the pod's state, because it tells you where to look:

Once a container has crashed, its output survives only in the logs of the previous instance (or in Container insights), so tools that work against the running container answer a different question.

Container insights tables

With Container insights sending data to a Log Analytics workspace, query these tables with KQL:

KubeEvents
| where TimeGenerated > ago(6h)
| where Namespace == "inference" and Reason == "FailedScheduling"
| project TimeGenerated, Name, Reason, Message

Container Apps produces two log streams. Console logs are the app's stdout and stderr; system logs are messages from the platform about the app and its revisions, such as image pull errors, probe failures and revision provisioning errors. When a container never starts its console output is empty, so the answer is in the system logs.

The console and system tables are written whenever the environment's log destination is a Log Analytics workspace. ContainerAppHTTPLogs is different: HTTP logs are only collected after you enable them through Azure Monitor diagnostic settings on the environment.

The two _CL tables are custom-log tables, which is why their columns carry suffixes such as ContainerAppName_s, RevisionName_s and Log_s. AKS tables (ContainerLogV2, KubeEvents) don't hold Container Apps data.

ContainerAppSystemLogs_CL
| where ContainerAppName_s == "ingest-worker"
| project TimeGenerated, RevisionName_s, Log_s

End-to-end connectivity

Trace a failing request from ingress to the container. In ContainerAppHTTPLogs, ResponseFlags value UH means ingress found no healthy upstream for the revision. If the console logs show the server listening on one port (say 5000) while ingress targetPort is another (8080), connections and the default probes fail, so set the target port to the port the app listens on. The general rule: fix the setting that mismatches, because anything that redeploys the same configuration reproduces the failure.

For outbound problems, test from inside a replica: az containerapp exec --command /bin/sh opens a shell in a running replica, where nslookup and curl use the same VNet and DNS as the app. If a private endpoint's name resolves to a public IP, the private DNS zone (for example privatelink.blob.core.windows.net) is not linked to the VNet the environment uses. Link the zone to that VNet (peering then carries the traffic). A private DNS zone only answers lookups from VNets it is linked to, and the goal is always to resolve the private IP rather than to open a public path.