AI-103: Build Generative Apps with Foundry — Models, RAG, Tools and Evaluation

An application connects to a Microsoft Foundry project in one of two ways: through AIProjectClient (the azure-ai-projects SDK) pointed at the project endpoint, or through the standard OpenAI() client pointed at the resource's v1 endpoint. Both are keyless when you use Microsoft Entra ID, and you choose by what the code needs to reach.

The resource name is the host and the project name is the last path segment; swapping them gives a URL of the right shape that reaches nothing. DefaultAzureCredential tries a chain of credential sources in turn (environment variables, workload identity, managed identity, then developer tools such as the Azure CLI), so it typically resolves to a managed identity in Azure and to your az login session locally; the same code works in both places once the identity holds a suitable role on the project.

from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient

project = AIProjectClient(
    endpoint="https://fabrikam-ml.services.ai.azure.com/api/projects/triage-app",
    credential=DefaultAzureCredential(),
)
client = project.get_openai_client()   # OpenAI-compatible client, same sign-in
response = client.responses.create(model="assistant-main", input="Hello")

If the code already has an AIProjectClient, get_openai_client() returns an OpenAI-compatible client that reuses the project's credential, so you don't configure a second endpoint or credential. For async frameworks such as FastAPI, import AIProjectClient from azure.ai.projects.aio and DefaultAzureCredential from azure.identity.aio — the async client needs an async credential.

The v1 API with the plain OpenAI client

The v1 API lets you use the standard OpenAI() class instead of AzureOpenAI, and new features arrive without bumping a dated api_version. The base_url must end in /openai/v1/, because the SDK appends paths such as /responses to it.

from openai import OpenAI
from azure.identity import DefaultAzureCredential, get_bearer_token_provider

token_provider = get_bearer_token_provider(
    DefaultAzureCredential(), "https://ai.azure.com/.default"
)
client = OpenAI(
    base_url="https://fabrikam-ml.openai.azure.com/openai/v1/",
    api_key=token_provider,
)

Read errors by status code: a 404 on every call usually means the path is wrong (for example a base_url without /openai/v1/, or a model value that is not a deployment), while a 401 points at the credential or token scope.

Using connections and deployments from the SDK

A Foundry project stores connections to other services (Azure AI Search, Storage, Application Insights, other AI resources), and AIProjectClient.connections lets the app read them at run time instead of hard-coding endpoints and keys per environment — this is how the SDK acts as a connector between your generative workflow and the services it depends on.

from azure.ai.projects.models import ConnectionType

search_conn = project.connections.get_default(ConnectionType.AZURE_AI_SEARCH)
search_endpoint = search_conn.target          # endpoint resolved at run time

vendor_conn = project.connections.get("vendor-catalog", include_credentials=True)
api_key = vendor_conn.credentials.api_key         # only present with include_credentials

Without include_credentials=True the key is never returned, so code that reads it gets nothing. When the app reaches the target with Entra ID (a managed identity holding a role on the search service), it doesn't need credentials at all — read the endpoint and pass a token credential to that service's SDK. Model deployments are discovered the same way through project.deployments.list(), described in the next section.

You call a Foundry model through a deployment, and the model parameter of every request carries the deployment name you chose, not the catalog model name. A deployment can be called anything — digest-writer, helpdesk-ptu, docs-embedder — so the service resolves model= against deployment names, and passing a catalog name such as gpt-4o when no deployment has that name returns HTTP 404 with DeploymentNotFound. The same rule applies to chat, Responses and embeddings calls.

• Same model, different capacity. Two deployments of one model (for example provisioned throughput and Global Standard) are told apart only by name; the deployment name is how a request picks the capacity it uses.
• Names that differ per environment. When dev and prod use different deployment names, avoid hard-coding: project.deployments.list(), filtered by model_name or model_publisher (for example model_name="Phi-4"), returns the deployments in the project with their names, so the app can discover the right one at startup. deployments.get expects a name you already know.

Which API a model speaks

Not every model supports every API surface or feature, so check the model's catalog card before you design the call:

One OpenAI() client on one v1 endpoint can serve all of these — you pick the method (responses.create, chat.completions.create, embeddings.create) per model. Request settings such as tool_choice only steer a model that already supports the feature; they never add a capability the model card doesn't list. (Choosing which model type fits a requirement is covered in the AI-103 lesson on choosing Foundry models, retrieval methods and agent tools.)

In the Responses API you send an image as an input_image content item and a PDF as an input_file item, placed in the same input message as the input_text question. Text that merely contains a URL or base64 string is only text — the model never sees a picture.

response = client.responses.create(
    model="inspect-vision",
    input=[{
        "role": "user",
        "content": [
            {"type": "input_text", "text": "List every reading shown on this gauge panel."},
            {"type": "input_image",
             "image_url": "https://example.com/plant/panel-07.png",
             "detail": "high"},
        ],
    }],
)

Supported image formats are PNG, JPEG, WebP and non-animated GIF. A local path or file:// URL is never reachable by the service, and a private blob URL can't be fetched; a SAS URL works technically but grants access outside the app, which some security rules forbid. The detail setting (low, high, default auto) controls processing resolution: low uses the fewest tokens and responds fastest, which suits coarse yes/no checks where fine detail doesn't matter, while reading small text or fine visual detail (like the gauge readings above) needs high.

Each Responses API call is independent unless you link it, so multi-turn apps either pass previous_response_id or carry the earlier output items into the next request's input themselves. Sending only the newest user message without either loses the earlier turns.

• Stored responses (the default). Responses are stored and retained for 30 days by default. previous_response_id=response.id links the next call to the stored one; responses.retrieve reads one back and responses.delete removes a specific stored response early — for example to honour a privacy request. Turning storage off later doesn't delete what already exists.
• Stateless (store=False). Nothing is kept on the service, so previous_response_id has nothing to point at: the app resends the previous output items as input. For reasoning models, add "reasoning.encrypted_content" to the request's include list so reasoning items come back in encrypted form and can be passed into the next turn, preserving the model's earlier reasoning without service-side storage.

Streaming versus background mode

stream = client.responses.create(model="assistant-main", input=prompt, stream=True)
for event in stream:
    if event.type == "response.output_text.delta":
        websocket_push(event.delta)

Other stream events mark structure rather than carry text: the …added events announce new output items and parts, and response.output_text.done arrives once the text is complete. Background mode needs the response stored so it can be polled, so a wrapper that forces store=False has to be overridden for those calls.

Retrieval-augmented generation (RAG) grounds a model's answer in your own content by doing three steps in order for every question: retrieve relevant chunks from an index such as Azure AI Search, augment the model input with those chunks, then generate the answer. Grounding only works if the retrieved text is in the input when the answer is generated — searching after generating, or adding results to a later prompt, leaves the current answer ungrounded.

Choosing the implementation path

The Azure OpenAI On Your Data feature — which grounded chat completions by adding an Azure AI Search entry to the request's data_sources field — has been retired and is no longer supported. Older samples still show it, so recognise it as retired. Current options:

• A Foundry IQ knowledge base called directly — Microsoft's recommendation for question answering over a document corpus.
• Foundry Agent Service with Foundry IQ — when the app also needs tools, actions or workflows (agents are covered in the AI-103 lesson on building agents).
• Classic app-side RAG — generally available: query the index yourself with the Azure AI Search SDK, then add the results to the messages you send to chat completions or the Responses API. This keeps an existing index and existing chat code, and needs no agents.

Fine-tuning is not a grounding mechanism: it changes style and behaviour, not which current documents the answer draws on.

Writing a grounded prompt

Retrieval supplies the right chunks; the prompt decides whether the model uses them faithfully. A grounded RAG prompt:

• Labels each chunk with a short source ID and requires the ID after each claim, so every statement can be traced.
• Restricts the answer to the supplied sources and gives an explicit "not found" path ("say that the sources don't contain the answer"), which curbs additions from the model's general knowledge.
• Keeps URLs out of the model's output. If the model writes link text, it can invent or mangle URLs. Have it cite chunk IDs only, and let the app map each cited ID to the stored source URL from the retrieved results — every rendered link is then real.

Retrieving more chunks, switching to exhaustive search or lowering temperature can reduce problems but don't guarantee grounded, cited answers; the instructions and the ID-to-URL mapping do.

The retrieval call sends either text or a vector to Azure AI Search, and which query class you use depends on whether the index can vectorize text itself:

from azure.search.documents.models import VectorizedQuery

vector = client.embeddings.create(model="docs-embedder", input=user_text).data[0].embedding
results = search_client.search(
    search_text=None,
    vector_queries=[VectorizedQuery(vector=vector, k_nearest_neighbors=5, fields="contentVector")],
)

Two behaviours catch people out. First, matching dimensions are not enough: two different embedding models can return vectors of the same length, so mixing them raises no error, but the distances become meaningless and results look random. Fix it by embedding queries with the index's model or re-embedding the index. Second, a vector query always returns the k nearest neighbours when at least k documents exist — even for an off-topic question — because it returns the closest vectors whatever their similarity. Relevance has to be handled downstream: a grounding instruction that lets the model decline, a hybrid query, or a minimum similarity threshold (preview) on pure vector queries. (Choosing between keyword, vector, hybrid and semantic ranking is covered in the AI-103 lesson on choosing Foundry models, retrieval methods and agent tools.)

Rewriting follow-up questions

Search runs before generation and the index has no memory of the chat, so a follow-up such as "What about the annual plan?" retrieves the wrong chunks. Giving the model chat history (for example with previous_response_id) can't fix what the search already returned. The standard fix is query rewriting: a quick model call turns the history plus the latest message into a standalone query that carries every earlier constraint and the new one, with no pronouns — after "Which museums in Vienna are free on Sundays?", the follow-up "Do any of them have audio guides?" should be searched as "free Sunday museums in Vienna with audio guides". Pasting the whole history into the search text adds noise, and a semantic ranker only reorders a bad candidate set.

For comparison questions ("Is the onboarding process in Spain different from Japan?"), one query's top results tend to favour whichever item matches best. Decompose the question into one query per item, run each, and merge the results; raising k adds cost and noise without guaranteeing coverage. Query rewriting fixes retrieval only — keep the "say when the sources lack the answer" instruction too, or the model fills gaps from general knowledge.

In function calling, the model never runs your code: it returns a function_call item, your application executes the function, and the app sends the result back as a function_call_output item so the model can finish its answer. If the app skips the execution step, the backend API is simply never called.

1. Send the request with a tools array describing each function (name, description, JSON Schema parameters).
2. The output contains one or more function_call items, each with name, JSON arguments and a call_id.
3. The app runs each function with the parsed arguments.
4. The app sends a follow-up request (usually with previous_response_id) whose input contains one function_call_output per call, each with the matching call_id and the result as output.

for item in response.output:
    if item.type == "function_call":
        args = json.loads(item.arguments)
        result = TOOL_IMPLEMENTATIONS[item.name](**args)   # your code runs it
        outputs.append({
            "type": "function_call_output",
            "call_id": item.call_id,          # links this result to its call
            "output": json.dumps(result),
        })
followup = client.responses.create(
    model="logistics-assistant",
    previous_response_id=response.id,
    input=outputs,
    tools=tools,
)

When one response contains several function_call items, every one needs its own function_call_output with its own call_id, all in the request that continues the response; answering only some of them, merging results under one ID or splitting them across separate follow-ups breaks the match. In the Responses API the link between a call and its result is always the call's call_id; the other identifiers in play (the output item's own ID, the response ID used for chaining, and the field Chat Completions uses on its tool messages) identify different things.

Controlling which tools are called

Server-side execution of HTTP APIs is a different feature — the OpenAPI tool on Foundry agents — and agent tool design is covered in the AI-103 lesson on building agents.

A multistep pipeline splits a complex task into focused steps whose outputs feed the next step, instead of one long prompt that does everything. Prompt chaining — one focused call per step, run in order — makes each step testable and debuggable on its own, so when the final result is wrong you can see which step failed. Running the steps in parallel breaks the hand-off, because later steps need earlier results.

Structured outputs: the glue between steps

Each hand-off should be machine-readable with exactly the expected fields. JSON mode only guarantees syntactically valid JSON — fields can still be missing or renamed. Structured outputs with a JSON Schema and strict set to true constrain the output to the schema.

Strict schemas follow rules that reject otherwise valid JSON Schema: every property must be listed in required (make a field optional in practice with a null union such as "type": ["string", "null"]), every object must set additionalProperties: false, and some keywords (for example minLength and maxLength) aren't supported. output_text still holds the same data as a string.

Graph workflows with Microsoft Agent Framework

When a pipeline needs a fixed, auditable order, branching, parallel steps, human approval or survival across restarts, model it as a Microsoft Agent Framework workflow: a graph of executors (an AI agent or custom code that receives messages, processes them and emits results) connected by edges that route messages.

• Edge types: direct; conditional (route on a value in the message, such as an order total or a risk score); switch-case (pick one branch); fan-out (send to several executors so they run in parallel); fan-in (wait for several executors and pass all their outputs to one target, which runs once).
• Human-in-the-loop: a workflow can issue a request for external input, such as a reviewer's sign-off, and pause until it is answered.
• Checkpoints: workflow state is saved at superstep boundaries, so a long-paused run, or one whose process stops partway, resumes from the last checkpoint instead of starting over.

An agent left to decide the order itself, or a single prompt, gives up the deterministic step order auditors may require; a graph without checkpoints loses its place when the process restarts.

Foundry's built-in evaluators each measure one property, so you pick them by what you need to detect and by the data you have. AI-assisted quality evaluators (Groundedness, Relevance, Coherence, Fluency, Retrieval, Response Completeness) use a judge model deployment you supply; NLP metrics such as F1, BLEU and ROUGE, and Document Retrieval, are computed and need none; and risk and safety evaluators run on the Foundry evaluation service and need no judge deployment either.

The pairing to remember: Groundedness catches what was added beyond the context, while Response Completeness catches what was missed from a ground-truth answer. Use Document Retrieval to tune search settings (chunk size, top-k, vector versus hybrid) when you have labelled relevance data, and Retrieval when you don't.

Risk and safety evaluators

• Violence, Sexual, Self-harm, Hate and unfairness — harmful content in responses.
• Protected Material — copyrighted text, such as published book passages or news articles, reproduced verbatim.
• Code Vulnerability — security flaws in generated code, such as injection weaknesses, unsafe path handling or hard-coded credentials.
• Indirect Attack — responses manipulated by instructions planted in retrieved documents or other inputs.
• Ungrounded Attributes — inferred personal attributes the context doesn't support; Sensitive Data Leakage (preview) applies to agents.

An evaluator measures; a guardrail blocks. Prompt Shields addresses prompt injection as a runtime filter, not an evaluation metric, so measuring injection in test results needs the Indirect Attack evaluator. (Runtime guardrails are covered in the AI-103 responsible AI lesson; production monitoring in the lesson on managing and monitoring Foundry systems.)

A cloud evaluation is an eval definition plus one or more runs: evals.create declares the data shape and the testing criteria (the evaluators), and evals.runs.create runs it against a data source. How you configure the run depends on whether your data already contains responses.

To compare two model deployments on the same queries without writing generation code, use target completions with an azure_ai_model target and one run per deployment, then compare the runs. {{sample.*}} exists only when a target generates output during the run.

In the testing criteria, each built-in evaluator is referenced by name, such as builtin.coherence or builtin.violence, with a data_mapping from its inputs to {{item.*}} or {{sample.*}} fields. A quality evaluator such as coherence also carries initialization_parameters naming your judge deployment; a safety evaluator such as violence carries none.

The judge deployment is therefore only needed when the criteria include AI-assisted quality evaluators.

Where test data comes from

• Before launch, no test set: generate a synthetic set of test queries in the project.
• After launch: convert production traces into an evaluation dataset; continuous evaluation scores live traffic.
• Safety probing: AI red teaming sends adversarial prompts — it tests safety rather than producing ordinary quality queries.