AI-103: Choose Foundry Models, Retrieval Methods and Agent Tools

Pick the model by what goes in and what must come out, then by how hard the reasoning is. Microsoft Foundry's model catalog offers several families, and each answers a different kind of request:

Two direction-of-travel distinctions decide many scenarios. Image in, text out (a customer uploads a photo and asks about it) needs a multimodal chat model; text in, image out (a brief becomes a catalog photo) needs an image generation model. And an embedding model never chats: it only turns text into vectors, while a chat model never produces the vectors stored in an index.

When a reasoning model is the right fix

A reasoning model spends extra compute "thinking" before it answers, and many expose a reasoning_effort setting (low, medium, high). It helps when the model already has the right information — the rules are in the prompt, tracing shows them on every request — but still reaches wrong conclusions when several dependent steps must be chained. It is slower and costs more per request, so it is the wrong choice for short, simple, high-volume classification. If the problem is missing or stale facts, a reasoning model won't help: that is a grounding problem (see below).

When several models are accurate enough, choose the cheapest and fastest one that meets the requirement, and check where it must run.

• Small language models for narrow, high-volume work. Routing messages into a handful of queues, tagging intents or extracting a few values are tasks a compact model often does well. Measure it: if a pilot shows the small model meets the accuracy bar, it wins on cost per request and latency.
• A fine-tuned small model can beat a large prompted one on running cost. When a base small model falls short on a narrow task but plenty of labelled examples exist, fine-tuning it can lift accuracy to the required level. At millions of requests a day the lower per-token price of the small model dominates, so a fine-tuned small model that meets the bar is cheaper to run than a large model with a few-shot prompt. Compare measured accuracy against the requirement, then cost.
• Foundry Local for offline and on-device inference. Foundry Local runs models — typically small language models such as the Phi family — directly on a Windows, macOS or Linux device, using its CPU, GPU or NPU; the app calls the model through the Foundry Local SDK or an optional OpenAI-compatible local server. Choose it when the app must work with no network connection, keep data on the device, or avoid per-token cloud charges. Any cloud deployment type (Global Standard, Data Zone, provisioned throughput) or a model router still needs the network.

Model router: per-request model choice behind one deployment

Model router is a trained model that you deploy like any other Foundry model. For each prompt it picks an underlying chat model — smaller and cheaper models when they are sufficient, larger or reasoning models when the prompt needs them — so the application calls one deployment and you write no model-selection logic of your own.

• Routing mode (custom deployment): Balanced (the default) picks the most cost-effective model within a small quality band of the best one; Cost widens that band for more savings; Quality picks the highest-quality model and ignores cost.
• Model subset: restricts which underlying models the router may use, for cost, compliance or capability reasons. The subset also acts as the failover set.
• Context window: the router's effective context limit is that of the smallest underlying model. A long prompt succeeds only if it happens to be routed to a model that can take it, so intermittent context-length errors on long inputs are fixed by a model subset of long-context models (or by shortening the prompt). Routing modes only trade cost against quality, and every mode can still pick a smaller model.

Use a prebuilt Foundry Tool when one already does the whole job, because it needs no prompt engineering, returns a predictable output shape and is maintained by Microsoft. Prompting a general model is the fallback for open-ended work no prebuilt service covers.

Content Understanding vs a multimodal chat model

The two are complements, not rivals. Azure Content Understanding fits repeatable extraction: every document (or image, audio or video file) is turned into the same schema of fields, with confidence and grounding for each, ready for an automated pipeline. A multimodal chat model fits open-ended conversation about an image: a user asks free-form follow-up questions that no schema could anticipate. A solution can use both — a chat model for the conversational feature and Content Understanding for the batch extraction. Note that Content Understanding analyzes audio and video files in batch; live, streaming speech-to-text is Azure Speech.

Use retrieval-augmented generation (RAG) when the model must answer from private or frequently changing knowledge and show its sources; use fine-tuning when the facts are already right but the model's format, tone or style isn't; use structured outputs when the answer must match a JSON schema the base model can already follow.

Diagnose before you choose. If evaluations show retrieval already finds the right passages but drafts ignore the required structure, more retrieval or a better ranker won't help — the gap is style, which points to fine-tuning. If the format is fine but facts are wrong or outdated, the gap is knowledge, which points to RAG. Putting an entire document set into the system prompt doesn't scale, adds cost on every call and still lacks reliable citations.

In Azure AI Search, use hybrid search by default for RAG, and add the semantic ranker when the right passages are found but ranked too low.

A request is hybrid when it carries both search_text and vector_queries. In the Python SDK:

results = search_client.search(
    search_text=user_query,            # keyword half
    vector_queries=[VectorizableTextQuery(
        text=user_query, k_nearest_neighbors=20,
        fields="chunk_embedding")],    # vector half
    select=["title", "chunk"],
    top=3)

The keyword half is what matches exact identifiers such as part or policy numbers. Settings on the vector query tune only the vector half — how many neighbours it returns and how they are found — so keyword matching exists only while the request carries search text.

Turning on the semantic ranker

The semantic ranker needs two things: a semantic configuration on the index that names the title, content and keyword fields to read, and queries sent with query_type="semantic" and that configuration's name. Adding a semantic configuration is an index update — no rebuild and no re-embedding. It is the fix when the best passage is found but sits below the few results passed to the model. A scoring profile, by contrast, boosts fields or freshness in the first-stage score; it isn't a meaning-based reranker.

Chunk documents along their meaning, embed every chunk and every query with the same embedding model, and let Azure AI Search do both with integrated vectorization when you don't want to write embedding code.

Integrated vectorization: two halves

The skillset alone embeds documents but leaves the app to embed queries; the vectorizer alone embeds queries but leaves the app to chunk and embed documents. "No custom embedding code" needs both. Pushing pre-split, pre-embedded chunks from app code (the push model) is the opposite of integrated vectorization.

Chunking choices

• Text Split skill — fixed-size chunks: textSplitMode (pages or sentences), maximumPageLength and pageOverlapLength. For plain text with no structure, the smallest fix when an explanation is cut between two chunks is to add overlap, so neighbouring chunks share some text. Making chunks much larger dilutes relevance and wastes tokens.
• Document Layout skill — structure-aware chunking: it reads headings, paragraphs and tables (using the Document Intelligence layout model) and chunks by section, so tables aren't cut in half and paragraphs keep the headings that give them meaning. Use it for structured documents such as manuals and reports.
• OCR skill — extracts text from images; it doesn't decide chunk boundaries.

Changing the embedding model means re-embedding

Vectors from different embedding models live in different vector spaces, even when their dimension counts match. Requesting a shorter vector from a larger model (the dimensions parameter) makes the query fit the field but not comparable with it, so results look random. To move to a new model, re-embed every document with it — typically into a new vector field — and point the vectorizer at the same model.

Classic RAG sends one (usually hybrid, semantically ranked) query per question from your own orchestration code; agentic retrieval uses a language model to break a complex question into several subqueries, runs them in parallel across sources, reranks and merges the results. Choose classic RAG for short, simple questions where latency and fine-grained control of each pipeline step matter most; choose agentic retrieval for complex, multi-part questions and agent scenarios.

Foundry IQ is the managed knowledge layer for agents, built on Azure AI Search. A knowledge base groups one or more knowledge sources — indexed or remote content such as Azure Blob Storage, SharePoint, OneLake, existing search indexes and public web data — and uses agentic retrieval to answer. It:

• automates chunking, embedding and metadata extraction for indexed sources, with scheduled refresh;
• enforces permissions at query time (synchronised ACLs, Microsoft Purview sensitivity labels, queries run under the caller's Microsoft Entra identity), so users only get answers from content they can open;
• returns grounded results with citations;
• can be connected to many agents, so several agents share one curated knowledge base instead of each wiring its own tools.

Match the service to where the information comes from: curated company content → a Foundry IQ knowledge base; an existing index you must use as-is → the Azure AI Search tool; documents a user hands the agent during a chat → the file search tool; things learned about a user over time → Memory.

Memory in Foundry Agent Service

Memory (preview) is managed long-term memory. You create a memory store (it needs compatible chat and embedding model deployments) and attach the memory search tool to a prompt agent, which then reads and writes memories during conversations; low-level memory store APIs exist for finer control. The system extracts information from conversations, consolidates duplicates and conflicts, and retrieves relevant memories later. A memory store holds three types:

• User profile memory — durable preferences and personal context (language, seat preference, dietary needs, exam date).
• Chat summary memory — distilled summaries of earlier conversations, which is what lets an agent pick up an earlier conversation days or weeks later.
• Procedural memory — reusable routines inferred from repeated tasks.

Memories are partitioned by scope. Setting the memory search tool's scope to {{$userId}} resolves it from the caller's identity, so each user's memories stay separate. A conversation's own history is different: it keeps the messages of one thread, whereas memory distils knowledge across many sessions.

Choose the tool by who executes the call and what it needs to reach. Foundry Agent Service tools can also be grouped into a reusable toolbox exposed through one managed MCP-compatible endpoint.

The key near-twin is OpenAPI tool vs function calling: both can describe the same operations, but with the OpenAPI tool the service makes the HTTP call, while with function calling your client code executes the function. Function calling is therefore the only option when policy forbids exposing the logic on any endpoint — an Azure Function or an OpenAPI endpoint would expose it.

Scoping and approving MCP tools

The MCP tool is configured with a server label and URL plus two controls:

• allowed_tools — the list of the server's tools the agent may call. The service enforces this list; agent instructions only guide the model and cannot restrict which tools it can reach.
• require_approval — "always" makes the run pause with an approval request before each call, so a person (via your app) approves or rejects it; "never" lets calls run automatically. Approval can also be set per tool.

Combine them when an agent should reach only a few of a server's tools and a human must sign off before calls that change data.

Choose by how much you build versus how much Foundry manages: a prompt agent is pure configuration, a hosted agent runs your own code, and the Responses API lets code that already runs elsewhere use Foundry models and tools.

Two contrasts decide most scenarios. Code-first prompt agents can be defined through the SDK in a pipeline, but they are still a persisted Foundry resource that must stay in sync with the app; an ephemeral agent through the Responses API has no such resource. And moving existing framework code into Foundry-managed hosting is a hosted agent, whereas keeping it in your own app and only borrowing Foundry models and tools is the Responses API.