AI-200 OpenTelemetry Tracing and KQL: Monitor and Troubleshoot Azure Solutions

A Python app sends traces, logs and metrics to Application Insights by installing the Microsoft OpenTelemetry Distro (pip install microsoft-opentelemetry) and calling use_microsoft_opentelemetry(enable_azure_monitor=True) once, before the app starts handling work. Microsoft directs new Python apps to this distro; apps already on the older Azure Monitor OpenTelemetry Distro (configure_azure_monitor()) need no change, but you never initialize both distros in one app.

import logging
from microsoft.opentelemetry import use_microsoft_opentelemetry

use_microsoft_opentelemetry(
    enable_azure_monitor=True,   # export to Application Insights
    logger_name="billing",       # collect logs from this logger tree
)

logger = logging.getLogger("billing")
logger.setLevel(logging.INFO)

The destination is the resource's connection string. Microsoft recommends the APPLICATIONINSIGHTS_CONNECTION_STRING environment variable for production: the same image or package then reports to a different Application Insights resource in each environment just by changing the variable. Passing azure_monitor_connection_string=... in code works but hard-codes one resource into the build, so keep it for local testing. A bare instrumentation key (APPINSIGHTS_INSTRUMENTATIONKEY) is the legacy SDK mechanism, not how the OpenTelemetry exporter is configured.

Bundled instrumentation: don't add it twice

The distro bundles and enables instrumentation for common libraries (web frameworks such as Flask, Django and FastAPI, HTTP clients such as requests and urllib3, database drivers and others). Microsoft warns against also instrumenting a bundled library by hand, for example calling a framework's Instrumentor().instrument_app(...) yourself: duplicate instrumentation isn't supported and shows up as doubled or inconsistent telemetry. The fix is to remove the manual call, not to filter or sample the duplicates away. Libraries the distro doesn't cover can still be added with their own OpenTelemetry instrumentation package.

Offline storage and retries

When ingestion is briefly unavailable, the exporter writes unsent telemetry to local disk and retries later. This is on by default (azure_monitor_exporter_disable_offline_storage defaults to False) and uses a folder under the temp directory. If the process can't write there, as in a locked-down container running as a non-root user, set azure_monitor_exporter_storage_directory to a writable path such as a mounted volume. Disabling offline storage silences the write errors but throws the data away.

Telemetry ingestion can require a Microsoft Entra token instead of the connection string alone. Once the security team sets local authentication to disabled on the Application Insights resource (DisableLocalAuth), only authenticated telemetry is accepted, and any app still sending with just its connection string is rejected. The connection string is still needed to find the endpoint; it simply stops being enough on its own.

Setting it up takes three things:

1. An identity for the app: a managed identity (preferred) or a service principal.
2. The Monitoring Metrics Publisher role assigned to that identity with the Application Insights resource as the scope. Despite its name, the role publishes all telemetry types. Reader or contributor roles (Monitoring Reader, Application Insights Component Contributor, Log Analytics Contributor) manage or read resources and don't grant the ingestion data action. A role on the Log Analytics workspace behind the resource doesn't count either, because the workspace is a different resource from the Application Insights component.
3. A credential passed to the distro with azure_monitor_exporter_credential.

from azure.identity import ManagedIdentityCredential
from microsoft.opentelemetry import use_microsoft_opentelemetry

use_microsoft_opentelemetry(
    enable_azure_monitor=True,
    # user-assigned identity: name it by client ID
    azure_monitor_exporter_credential=ManagedIdentityCredential(
        client_id="<client-id-of-the-identity>"
    ),
)

ManagedIdentityCredential() with no arguments uses the system-assigned identity, so an app that has only a user-assigned identity must pass that identity's client_id. A ClientSecretCredential also authenticates, but it means storing a secret, which is exactly what managed identities avoid. Some Microsoft samples still show the older distro's configure_azure_monitor(credential=...); with the Microsoft OpenTelemetry Distro the parameter is azure_monitor_exporter_credential.

Application Insights labels each service by its cloud role name, which the exporter builds from OpenTelemetry resource attributes, the attributes that describe the process emitting telemetry rather than any one operation. Application Map draws one node per role name, so when several services share one Application Insights resource Microsoft requires each to set its own role name; otherwise they collapse into a single node. You don't need separate resources to separate them.

Two rules decide the final value. First, the role name is service.namespace + . + service.name, or just service.name when no namespace is set. Second, under the OpenTelemetry SDK configuration rules, OTEL_SERVICE_NAME takes precedence over a service.name given inside OTEL_RESOURCE_ATTRIBUTES.

OTEL_SERVICE_NAME=invoicer
OTEL_RESOURCE_ATTRIBUTES=service.namespace=billing,service.instance.id=replica-3
# AppRoleName     -> billing.invoicer
# AppRoleInstance -> replica-3

In workspace tables these values appear as AppRoleName and AppRoleInstance, which is what KQL filters on. Two things don't change the role: a service.name span attribute (span attributes describe one operation and become custom properties) and other custom resource attributes, which the exporter doesn't copy onto every row (it reports them separately, in a metric named _OTELRESOURCE_). For a per-row property, use a span processor (next section).

A span's kind decides which Application Insights table it lands in: spans for incoming work become requests, and spans for outgoing or in-process work become dependencies. Requests drive request rates, failure rates and the operation-based Performance and Failures views, so the kind matters for any span you create yourself.

A span created with no kind is INTERNAL. That is right for a sub-step, but wrong for the top-level unit of work in a background worker that autoinstrumentation doesn't capture (a custom queue loop, a scheduled job): Microsoft recommends kind=SpanKind.SERVER there so the work shows up as an operation. A call to a remote system over a protocol no library instruments should be CLIENT, so it appears as a remote dependency rather than in-process work.

Creating spans

Parent-child links come from the active context. tracer.start_as_current_span(name) starts a span and makes it current, so spans created inside the block, including auto-instrumented HTTP or database calls, become its children. Used as a context manager it ends the span on exit (end_on_exit=True by default), which is what gets it exported.

from opentelemetry import trace
from opentelemetry.trace import SpanKind

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("rerank", kind=SpanKind.INTERNAL) as span:
    span.set_attribute("candidates", len(items))
    ranked = rerank(items)      # calls inside are children of "rerank"

Attributes set with span.set_attribute become custom properties (Properties in the workspace tables).

Span processors: enrich every span without touching handlers

A span processor is called by the SDK for every span, auto-instrumented ones included, so it's how you add a property to all requests and dependencies without editing route handlers. Register it with span_processors=[...] in the distro call. Set attributes in on_start, which receives the live, writable span. on_end receives a read-only ReadableSpan that has no set_attribute method, so writes there fail. A request middleware only reaches the request span, not its dependencies.

from opentelemetry.sdk.trace import SpanProcessor

class TenantTagger(SpanProcessor):
    def on_start(self, span, parent_context=None):
        span.set_attribute("tenantTier", TIER)   # value known up front

use_microsoft_opentelemetry(enable_azure_monitor=True,
                            span_processors=[TenantTagger()])

A distributed trace survives a hop between services only if the caller sends its trace context and the receiver continues from it. OpenTelemetry Python's default propagators are W3C Trace Context (the traceparent and tracestate headers) plus baggage. Instrumented HTTP clients and servers inject and extract these headers automatically; a proxy, gateway or other hop that strips them breaks the trace, and the fix is to forward them unchanged. The legacy Request-Id header isn't what OpenTelemetry sends.

Reading a traceparent header

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
             |  |                                |                |
          version  trace ID (32 hex)        parent span ID (16 hex)  flags (01 = sampled)

In Application Insights the trace ID is the OperationId shared by every item in the trace. Each span has its own Id (span ID), and its ParentId is the span ID of its direct parent. So the server span that handled an incoming call has the caller's trace ID as its OperationId and the caller's span ID (the third field) as its ParentId, while getting a new Id of its own.

Custom transports: inject and extract

When work travels through something no instrumentation covers, such as jobs pushed onto a Redis list or a custom socket protocol, carry the context yourself. The producer calls opentelemetry.propagate.inject(carrier) to write traceparent/tracestate into a dict that travels with the payload; the consumer calls extract(carrier) and passes the result as the context of its span.

from opentelemetry.propagate import inject, extract

# producer
message = {"body": payload, "meta": {}}
inject(message["meta"])                 # adds traceparent / tracestate
queue.push(json.dumps(message))

# consumer
message = json.loads(queue.pop())
ctx = extract(message["meta"])
with tracer.start_as_current_span("handle-message", context=ctx,
                                  kind=SpanKind.CONSUMER):
    handle(message["body"])

Setting a span kind, sharing a role name or logging the trace ID doesn't link spans: only a propagated context makes both sides share one OperationId.

An exception that escapes a start_as_current_span block is handled for you: by default (record_exception=True, set_status_on_exception=True) the SDK records it as an exception event, which becomes an AppExceptions row linked to the span, and sets the span's status to ERROR, which marks the request or dependency failed. The two flags are independent: one controls the exception record, the other the failed status.

A handled exception (caught inside the block, perhaps with a fallback) triggers neither. To keep the failure visible while still recovering, record it explicitly:

from opentelemetry.trace import Status, StatusCode

with tracer.start_as_current_span("fetch-profile") as span:
    try:
        profile = primary.get(user_id)
    except Exception as ex:
        span.record_exception(ex)                       # AppExceptions row
        span.set_status(Status(StatusCode.ERROR, str(ex)))  # span failed
        profile = replica.get(user_id)                  # fallback continues

Logging the error at a level below the collected threshold produces nothing, and re-raising to get automatic recording also cancels the fallback.

Sampling in the Python distro keeps a share of whole traces and is configured with two environment variables set before the distro initializes: OTEL_TRACES_SAMPLER picks the sampler and OTEL_TRACES_SAMPLER_ARG its argument. Microsoft says not to carry over the older distro's sampling_ratio and traces_per_second arguments.

Because the default is rate-limited, a busy service with no sampling settings already keeps only about five traces a second. A ratio of 10 isn't "10%": the fixed-percentage argument is a fraction. Generic OpenTelemetry samplers such as parentbased_always_on keep everything.

Sampling decides per trace, so a kept trace is complete. It doesn't apply to everything: metrics are never sampled, and in Python trace-based sampling for logs is off by default, so log volume doesn't drop when spans are sampled. Passing enable_trace_based_sampling_for_logs=True to use_microsoft_opentelemetry drops log records that belong to unsampled traces and keeps the others as they are. Kept items carry an ItemCount (how many original items each represents), which KQL must use to count correctly (see below).

Filtering: choose by where and what you drop

Lowering the sampling rate cuts all traces rather than one endpoint, and a logger level controls log records only, never request telemetry. Deleting rows after ingestion doesn't prevent them from being ingested.

The distro collects Python logging records from the logger named in logger_name and from every logger below it in the dot-separated hierarchy, and sends them to traces (AppTraces) with the active trace's IDs attached, which correlates each log line with its request. Microsoft suggests a named application logger so the SDK's own log messages aren't collected.

• Hierarchy. With logger_name="billing", records from billing.invoices and billing.invoices.pdf propagate up to it. Names that merely contain the text (billingapi, billing_jobs) or put it later (jobs.billing) are separate branches and aren't collected.
• Levels. A logger with no level of its own inherits from its ancestors; with nothing set, the root's WARNING applies, so info and debug calls are discarded before they reach the exporter. Set the level on the collected logger (setLevel(logging.INFO)); child loggers without a level inherit it. Microsoft's advice is to control log volume with levels and filters.

Custom events

A log call becomes a custom event (customEvents, AppEvents in the workspace) when its extra dict carries the key microsoft.custom_event.name, logged on a collected logger at a collected level:

logger.info("invoice sent",
            extra={"microsoft.custom_event.name": "invoice-sent"})

Custom metrics: pick the instrument by the shape of the data

Attributes passed with a measurement, such as {"region": r}, become dimensions, so one metric can be split per value; putting the value in the metric name instead loses that shared dimension. An observable gauge's callback receives an options argument and yields one Observation(value, attributes) per series:

from opentelemetry import metrics
from opentelemetry.metrics import Observation

meter = metrics.get_meter(__name__)

def read_backlog(options):
    for pool, size in pool_sizes().items():
        yield Observation(size, {"pool": pool})

meter.create_observable_gauge("pool_backlog", [read_backlog])

If you only know the current value, not each change, a counter or up-down counter is wrong: they add whatever you pass, so repeatedly adding the current value inflates the total.

KQL (Kusto Query Language) queries Application Insights data in a Log Analytics workspace as a pipeline of operators separated by |, each working on the rows the previous one produced. Workspace-based Application Insights stores telemetry in AppRequests, AppDependencies, AppTraces, AppExceptions, AppEvents and AppMetrics (the classic requests, dependencies, traces and so on), with columns such as TimeGenerated, Name, DurationMs, Success, AppRoleName, OperationId and ItemCount.

AppDependencies
| where TimeGenerated > ago(6h) and DependencyType == "SQL"
| summarize p90 = percentile(DurationMs, 90), calls = sum(ItemCount)
    by bin(TimeGenerated, 15m)
| render timechart

• where with ago() limits the time range; between (a .. b) and in (...) test ranges and lists.
• summarize ... by bin(TimeGenerated, 5m) creates fixed-size time buckets. Grouping by raw TimeGenerated gives one group per timestamp. Aggregates include count(), countif(), avg(), percentile()/percentiles(), dcount(), arg_max().
• Order of operators matters. To rank groups, aggregate first and then top N by <aggregate>; top before summarize ranks individual rows, and take returns arbitrary rows with no sort.
• render timechart draws the result over time.

Gap-free time series with make-series

summarize never invents rows: a bin with no data is simply absent, even when countif would have been 0. Series functions such as series_decompose_anomalies() need evenly spaced arrays, which make-series builds, filling empty steps with default=. It fills only inside its range, and without from ... to ... the range runs from the first to the last row present, so set both:

AppDependencies
| make-series Errors = countif(Success == false) default=0
    on TimeGenerated from ago(7d) to now() step 1h
| extend anomalies = series_decompose_anomalies(Errors)

String operators

Kusto indexes string columns by terms, runs of letters and digits split at other characters (so cache-v2 holds the terms cache and v2). Term operators can use that index; substring operators must scan.

Choose has for a whole word in any case on large tables, contains when the text can sit inside a longer word (has can't find sync inside AsyncWorker), and =~ for an exact value in any case; has or contains would also match longer values that include the term.

Counting sampled telemetry

When telemetry is sampled, each stored row stands for ItemCount original items (about 1 ÷ the sampling ratio, or more under rate-limited sampling as traffic grows). count() and dcount() count only the stored rows, so they understate traffic and, under rate limiting, flatten at the cap. sum(ItemCount) estimates the real total, which is what the portal's own charts do. avg(ItemCount) shows the effective sampling factor, useful for checking configuration but not a total. With unsampled data every ItemCount is 1, so the two agree.

Telemetry for one distributed operation is correlated in KQL through three columns: OperationId (the trace ID every item shares), Id (this item's span ID) and ParentId (its direct parent's span ID).

• Everything in one operation: union AppRequests, AppDependencies, AppTraces, AppExceptions | where OperationId == id | order by TimeGenerated asc. A join of two tables leaves out logs and exceptions, and filtering on Id or ParentId returns one item or one item's children.
• Direct children of a request: join the request's Id to the dependency's ParentId. Joining on OperationId returns every dependency anywhere in the trace, including other services' calls.

Join kinds

lookup extends a fact table with columns from a smaller dimension table and defaults to leftouter behaviour, so unmatched rows stay. The default join is easy to miss: A | join (B) on OperationId without a kind is innerunique, which silently de-duplicates the left side.

AppDependencies
| where Success == false
| join kind=leftanti (AppTraces | where SeverityLevel >= 3) on OperationId
// failed dependencies whose operation logged no error

Platform metrics versus application metrics

Azure resources' platform metrics (for example a Service Bus namespace's server errors or a storage account's transactions) live in the Azure Monitor metrics store, where metrics explorer reads them. They reach a Log Analytics workspace, and so KQL and the workspace's retention, only after you create a diagnostic setting on the resource that sends AllMetrics to the workspace. They then land in the AzureMetrics table (Resource, MetricName, Total, Average, Minimum, Maximum, Count, TimeGrain), where they can be joined or unioned with App* data. AppMetrics holds application telemetry (custom and SDK metrics), not platform metrics, and an app can't measure a broker's internal errors itself.

AzureMetrics
| where MetricName == "UsedCapacity"
| summarize arg_max(TimeGenerated, Average) by Resource

arg_max(TimeGenerated, cols) returns, per group, the columns from the same row as the latest timestamp: the latest value per resource. Separate max(TimeGenerated) and max(Average) aggregates can mix values from different rows, and top 1 has no grouping.

One limitation catches people: the diagnostic-setting export flattens dimensions, summing or averaging across them, so per-entity values that metrics explorer can split (such as a Service Bus EntityName) aren't in AzureMetrics, and no diagnostic-setting option adds them. Microsoft's workaround is to read dimensional values from the Azure Monitor Metrics REST API and, if they're needed in the workspace, ingest them with the Logs Ingestion API.