AI-103: Optimize and Operationalize Generative AI — Parameters, Reflection, Tracing and Model Routing

On non-reasoning models such as gpt-4.1 and gpt-4.1-mini, temperature and top_p are the two controls for randomness: lower values make output focused and repeatable, higher values make it varied. Temperature ranges from 0 to 2. Around 0.2 suits factual, legal or extraction work where two runs should agree; 0.7 and above suits brainstorming, taglines and other creative tasks where sameness is the complaint.

top_p (nucleus sampling) is the alternative control: each token is sampled only from the most likely candidates whose cumulative probability reaches the top_p value, so top_p=0.1 means only the top 10% of probability mass is considered. Lowering it trims odd, low-probability word choices; raising it toward 1.0 widens the pool.

Microsoft's guidance is to change one of the two, not both. Both reshape the same token distribution, so moving them together confounds any experiment — you can't tell which change caused the effect. Leave one at its default, tune the other against a fixed set of prompts, and compare.

These sampling controls apply to non-reasoning models only. Reasoning models reject most of them, which the next section covers.

Reasoning models — the o-series (o1, o3, o3-mini, o4-mini) and the GPT-5 family — do not support temperature, top_p, presence_penalty, frequency_penalty, logprobs, top_logprobs, logit_bias or max_tokens (GPT-6 Astra is the documented exception: it accepts all of them). Sending them returns HTTP 400 errors about unsupported parameters, so a migration from gpt-4o or gpt-4.1 usually means deleting the sampling arguments, not swapping one for another — top_p is just as unsupported as temperature.

• The output cap is renamed per API. Chat Completions uses max_completion_tokens; the Responses API uses max_output_tokens. Each name belongs to its own API.
• System and developer messages. On o4-mini, o3, o3-mini and o1 a system message is treated as a developer message; the two are functionally the same. Don't send both in one request — merge persona and formatting rules into one developer (or system) message.
• Tools on Chat Completions with gpt-5.6. gpt-5.6 models reject Chat Completions requests that send function tools while reasoning is active. Because these models default to medium effort, the error appears even if the code never sets reasoning_effort. Two documented fixes: move tool-calling requests to the Responses API (recommended — keeps reasoning and planning quality), or set reasoning_effort to none on each Chat Completions request that sends tools (works, but the model then calls tools without reasoning). Setting a low effort or omitting the field leaves reasoning on.

Reasoning tokens and the output budget

Reasoning models think in hidden reasoning tokens before writing the answer. They are never returned as text, but they occupy the context window, are billed as output tokens, and count toward max_output_tokens / max_completion_tokens together with the visible answer. Two consequences matter in practice:

• A small cap meant to keep answers short can be consumed entirely by reasoning. On the Responses API the response returns status: incomplete with reason max_output_tokens, possibly with no visible text; on Chat Completions the message content is empty and finish_reason is length, with reasoning_tokens equal to the cap. You still pay for the reasoning.
• Microsoft recommends reserving at least 25,000 tokens for reasoning and output while you learn a workload, then tuning the cap from logged reasoning_tokens. Control the length of the visible answer with verbosity or instructions, not with a tiny cap.

Reasoning effort decides how much the model thinks before answering; verbosity decides how long the visible answer is. Because they are independent, you can ask for deep reasoning and a short answer at the same time — a contract-review assistant, for instance, can weigh every clause carefully and still reply in two sentences. Changing one dial never fixes a problem that belongs to the other.

Model-specific details are where errors hide:

• none skips reasoning and is available from gpt-5.1 onward. Microsoft positions it for latency-critical work such as voice assistants, fast retrieval and classification, where reasoning adds no quality.
• minimal exists only on the original GPT-5 models, not gpt-5.1 or later.
• Defaults differ. gpt-5.1 defaults to none, while earlier GPT-5 reasoning models reasoned by default. Code that never set an effort therefore stops reasoning after an upgrade to gpt-5.1 — reasoning_tokens drops to 0 and multistep answers get worse. Pass an effort level explicitly. gpt-5.6 models default to medium.
• Lower effort is the latency lever when traces show most of the wait happening before the first output token and usage shows reasoning tokens far outnumbering answer tokens — provided an evaluation shows quality holds.

Prompt engineering is the cheapest way to change behaviour: no training, no new deployment, effect on the current request only. Microsoft's techniques for chat models are:

• System message. Defines the assistant's role, rules, output format and boundaries.
• Few-shot examples. In Chat Completions, add examples as user/assistant message pairs after the system message. They show the exact labels and the exact format (for example a bare category name with no explanation) — far lighter than fine-tuning, which costs training and a custom deployment.
• Clear syntax and delimiters. Mark where pasted content starts and stops with headings, --- or triple quotes, so the model never confuses an email or document with your instructions.
• Repeat instructions at the end. Models show recency bias: what comes last can weigh more than what comes first. When instructions precede a large block of content, restate the key ones (length, language, format) after it.
• Prime the output. End the prompt with the start of the expected answer, such as a heading or the first word of a list.
• Break the task down. Ask for smaller steps in sequence instead of one compound request.
• Give the model an out. Tell it what to do when the provided text doesn't contain the answer (for example, return a fixed "not found" message). Without that permission the model tends to invent a plausible answer. Few-shot examples that always contain an answer reinforce answering.

Chain-of-thought prompting vs reasoning summaries

Chain-of-thought (CoT) prompting — "work through this step by step, then give the final answer" — reduces multistep and arithmetic errors on non-reasoning models. Pair it with a fixed final-answer format (a last line that begins with a fixed label) so code can still parse the result.

It does not belong on reasoning models, which already reason internally; for them, state the goal, constraints and output format instead of prescribing steps. When people need to see how a reasoning model reached an answer, request a reasoning summary (reasoning.summary on the Responses API: auto, concise or detailed, model-dependent). Microsoft states that extracting raw reasoning by any other method — such as decoding encrypted reasoning items — isn't supported and may violate the Acceptable Use Policy. Turning effort up or enabling tracing does not expose reasoning to reviewers.

A reflection (self-critique) loop has one call generate a draft and another critique it against explicit criteria, then revises when the critique finds problems. It raises quality at the cost of extra model calls, so the design work is in making the critique useful and the loop cheap.

• Give the critic a basis for judgement. A critic that sees only the draft and a yes/no question approves almost everything — it cannot know that a cited date differs from the source document. Pass it the same retrieved passages (a grounding check) and ask it to score set criteria and list each failing claim, so the generator gets actionable fixes. The fix is evidence, not a stronger judge: any critic that can't see the source shares the generator's blind spot.
• Bound the loop. Set a maximum number of revision rounds and a rule for what to return when the limit is reached (for example the best-scored draft so far). An unbounded generator–critic loop can run until the request times out.
• Revise conditionally. When failures are rare, run a fast verifier (a groundedness check or evaluator) on every output and call critique-and-revise only for the outputs that fail. Always-on revision and best-of-n generation both work but tax every request with extra latency and cost.
• Check before delivery. The critique must run before the answer reaches users, not after.

Evaluating chain-of-thought and output quality

Chain-of-thought evaluation means using a judge — an LLM following a rubric, or code — to score outputs and explain each judgement, so you can see why answers fail before changing the prompt. Foundry's evaluators provide this. (Choosing among the built-in quality and safety evaluators and running cloud evaluations is covered in the lesson on building generative applications; this section covers the judge's output and custom evaluators.)

A built-in quality evaluator such as coherence or fluency returns:

• score — for example 1–5 on a Likert scale;
• reason — the judge model's explanation; read it across low-scoring items to learn what to fix;
• threshold — the pass cut, 3 by default and configurable (raise it when your quality bar sits above the middle of the scale);
• label / passed — the pass/fail outcome from comparing score with threshold.

When no built-in evaluator fits, create a custom evaluator of the type that matches the rule:

A requirement often splits across types: schema validity in a code-based evaluator, empathy in a prompt-based one. Asking the system under test to rate its own reply is not an evaluation.

Tracing in Foundry records each model call, agent run and tool call as OpenTelemetry spans stored in the Application Insights resource connected to the project, where they appear in the Foundry portal's Traces view and in Application Insights.

• Server-side (no code). Once an Application Insights resource is connected to the project, Foundry automatically records traces for prompt agents, hosted agents and workflows — including tool calls and token usage — without changing or redeploying application code.
• Access. Viewing traces needs Log Analytics Reader on the connected Application Insights resource. Roles for calling agents or using the project don't grant it, and Foundry diagnostic settings don't deliver run traces.
• Client-side (Python). Set AZURE_EXPERIMENTAL_ENABLE_GENAI_TRACING=true before calling AIProjectInstrumentor().instrument(); if it isn't set at that moment, a warning is logged and no GenAI spans are produced. Export with configure_azure_monitor(), passing the project's Application Insights connection string (the samples read it with project.telemetry.get_application_insights_connection_string()) — not the project endpoint, a workspace ID or a deployment key.

import os
os.environ["AZURE_EXPERIMENTAL_ENABLE_GENAI_TRACING"] = "true"   # first

from azure.ai.projects.telemetry import AIProjectInstrumentor
from azure.monitor.opentelemetry import configure_azure_monitor

from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential

project = AIProjectClient(endpoint=os.environ["PROJECT_ENDPOINT"],
                          credential=DefaultAzureCredential())
configure_azure_monitor(
    connection_string=project.telemetry.get_application_insights_connection_string())
AIProjectInstrumentor().instrument()
openai_client = project.get_openai_client()   # after instrumenting

Extending traces

• Custom functions. Decorate a step such as a reranker with @trace_function (from azure.ai.projects.telemetry) to give it its own span and duration. It records every parameter as code.function.parameter.<name> and the return value as code.function.return.value regardless of the content-recording setting — so anything sensitive in its signature, such as a patient's date of birth or an access token, lands in telemetry. Keep sensitive values out of a decorated function's arguments and return value (pass an opaque reference the step resolves itself), rather than giving up the step's span.
• Attributes on every span. A span processor registered on the tracer provider is called for every span, including those the GenAI instrumentation creates automatically. Setting an attribute (tenant ID, feature name) in its on_start tags all spans without recording content. OTEL_SERVICE_NAME is fixed per process, and OpenTelemetry baggage carries context but does not become a span attribute on its own.
• Trace context propagation. With GenAI tracing enabled, the SDK injects traceparent/tracestate headers so client spans link to server-side spans. It is on by default but applies only to OpenAI clients obtained from get_openai_client() after instrumentation; clients acquired earlier, or built directly with OpenAI(), aren't injected. Baggage propagation is a separate opt-in.
• Content recording. Prompt and response text are captured only when content recording is on (OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT or enable_content_recording); these settings control the built-in GenAI spans, not decorated functions.

Token analytics means reading the usage object of every response (or the GenAI span attributes) to see where tokens — and cost — go. The field names differ by API:

The detail fields are subsets of their totals, not additions: 4,000 input tokens with 1,536 cached means 2,464 uncached plus 1,536 cached; 900 output tokens with 700 reasoning means 200 visible answer tokens, not 1,600 in total. Reasoning tokens are billed at the output rate; cached input is billed at a discount. A bill that rises while visible answers stay the same length points at reasoning_tokens.

GenAI spans carry gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. When several features share one deployment, platform metrics can't separate them; add a feature attribute to the spans (for example with a span processor) and sum the usage attributes by it in Application Insights — no extra deployments needed.

Prompt caching

Prompt caching reuses the processed prefix of a recent identical prompt, reducing cost and latency. It is automatic on supported models, on both Standard and Provisioned deployments — there is nothing to turn on — but only when:

• the request is at least 1,024 tokens long, and
• its first 1,024 tokens are identical to a recent request. A single differing character in that range — a request ID or the user's name placed near the start of the prompt — is a miss.

So put static content (instructions, policies, few-shot examples, tool definitions) first and dynamic content (user question, timestamps, IDs) last. Hits appear as cached_tokens. On GPT-5.5 and earlier models, hits are counted in 128-token increments beyond the first 1,024; on GPT-5.6 and later, writing a prefix to the cache can itself be charged, reported as cache_write_tokens.

• Retention. Cached prefixes are evicted after inactivity. On gpt-4.1, prompt_cache_retention can be set to 24h (instead of the in-memory default) so prefixes survive idle gaps without warm-up requests.
• GPT-5.6 and later. prompt_cache_retention doesn't apply; prompt_cache_options.ttl supports only 30m. prompt_cache_key improves matching for requests that share a long common prefix — use one key per distinct prefix (per tenant, say), not one key across unrelated prefixes. If one prefix-and-key combination exceeds roughly 15 requests per minute, some requests can miss, so spread busy prefixes across a few keys with a stable mapping.

A latency breakdown reads the nested spans of one request to find which step consumes the time, then fixes that step first. In an agent trace, invoke_agent spans the whole run, with child chat spans for model calls and execute_tool spans for tools; each records start and end times.

Raising a deployment's TPM quota changes how much throughput is allowed before 429 throttling; it doesn't make a single request generate faster. Quotas and the Monitor dashboard are covered in the lesson on managing and monitoring AI systems.

Safety signals are the content-filter outcomes returned with each call; logging them next to traces shows how often filtering blocks input, withholds output, or fails to run. With Chat Completions they look different for prompts and completions:

Keeping sensitive content out of the wrong hands

With content recording on, prompts, responses, system instructions and tool-call arguments and results are sensitive. Application Insights routes these GenAI attributes to a dedicated AppGenAIContent table. Setting that table's protection level to Protected applies deny-by-default to standard read and custom roles: engineers with Log Analytics Reader keep seeing spans, timings and token counts, but not content. Grant Privileged Monitoring Data Reader on the Application Insights resource linked to the project to the group that must read the content (for example a compliance review group) — it works in both Foundry and Application Insights. Turning content recording off instead removes content that auditors may need.

Orchestration means sending each request to the cheapest component that can handle it well — a managed router, your own classifier, a model cascade, or deterministic rules — instead of sending everything to one large model.

Model router (managed)

Model router is a single Foundry deployment that analyses each prompt's complexity, reasoning needs and task type in real time and routes it to an eligible underlying model, using smaller, cheaper models when they suffice. You call it like any other deployment and write no routing logic.

• Routing modes: Balanced (default — weighs cost and quality), Quality (favours the most capable models, trading higher cost for answer quality), Cost (favours cheaper models).
• Model subset: narrows the pool to models that meet your requirements; routing mode then applies within the subset, and failover stays within it.
• Context window: a large prompt succeeds only if it happens to land on a model with a big enough window, so long-context traffic fails apparently at random. Make it reliable by limiting the pool to a subset of models whose windows fit the largest prompts you send.
• Which model answered: the response's model field names the underlying model that served the request. Log it to correlate weak answers with models; the deployment name is the same for every call.
• Pool limits: the router chooses only among the supported models for its version (Claude models must be deployed separately). A subset is chosen from that supported list, so it can't include your own deployments, such as a fine-tuned model.

App-owned routing and cascades

Running both models on every request, or retrying a failure on the same small model with a higher temperature, pays more or fixes nothing.

Hybrid LLM and rules engines

Use the model where flexibility helps and deterministic code where outcomes must be identical every time and auditable:

• Model: reads unstructured, variable input (claim letters, free-text tickets, invoices) and returns structured data — ideally with structured outputs and a strict JSON schema — or a flag such as "mentions a suspected data breach".
• Rules: apply thresholds and policies, enforce mandatory outcomes (every message flagged as a suspected breach goes straight to the security queue), and validate model output (do the extracted line items sum to the total?). Failures route to review or re-extraction.
• Rules first for structured traffic: requests that arrive in a fixed format with scripted handling (order-status lookups, for example) can skip the model entirely — lowest cost, fully predictable.

Putting a rule in the system message, adding few-shot examples or fine-tuning all improve the odds but leave the decision probabilistic and hard to audit; a second model call or prompt-based judge isn't deterministic either.