AI-103: Build Foundry Agents — Tools, Memory, Multi-Agent Workflows and Evaluation

You define an agent's role, goals and boundaries in its instructions (the system message on the agent definition): what the agent is for, which topics it handles and declines, how it should use each of its tools, and what to do when a tool fails or returns nothing. In Foundry Agent Service a prompt agent is an agent definition — a model deployment, instructions and tools — and the instructions are the only place the model learns your business rules.

• Role and scope. State what the agent is for (say, a campus library assistant that answers catalogue and loan questions) and what it must not take on, with where to send people instead. Without an explicit boundary, an agent happily answers questions it has no data for.
• Tool-use guidance and decision rules. Describe each tool's purpose, and add an explicit precedence rule wherever two tools could answer the same kind of question — for a parts-ordering agent with a warehouse database and a manufacturer-website lookup, say which source to consult first for which question type. Removing one of the overlapping tools stops the confusion but also removes a capability you still need.
• Failure guidance. Spell out how to behave when a tool comes back empty or errors: admit the gap and gather more detail from the user instead of filling the hole with a plausible guess. Microsoft's example instructions include this kind of rule.
• What doesn't belong in instructions. Tool schemas live on the tool definitions (never paste an OpenAPI spec into the prompt), secrets live in project connections, and request settings such as tool_choice are set by the app on each request — the model can't set them.

Making tool use deterministic with tool_choice

Instructions steer the model; tool_choice on the request controls it. auto (the default) lets the model decide, required makes it call one or more tools, and none prevents tool calls. When a lookup must always happen before the agent answers — a seat-availability check before confirming a booking — set tool_choice to required on those requests. It guarantees a call, not which tool is chosen, so it doesn't fix selection between similar tools. temperature only reduces randomness, and strict mode only shapes arguments once a call is made; neither forces a call.

When one agent should become several

Tool-selection errors grow as one agent accumulates many tools from unrelated areas, and one long instruction block shared by several teams becomes hard to maintain. The fix is specialization: split the work into agents with narrow instructions and only their own tools (facilities, legal, procurement), and route between them with an orchestrator. Regrouping the same tools into toolboxes on the same agent, or moving to a model with a bigger context window, leaves the overloaded agent in place.

A function tool is defined by a name, a description and a parameters object written in JSON Schema; the model reads all three to decide when to call the function and how to fill its arguments. The descriptions do most of the work: two tools both described as "Looks something up" give the model nothing to tell find_shipment from find_return. Microsoft's troubleshooting guidance for wrong tool calls and malformed parameters is to state in each description when the tool applies, and to describe each parameter with its expected format and an example value (for instance, that a tracking number is twelve digits).

Strict mode

Setting strict=True on a function tool makes the model's arguments always conform to the schema, but the schema must then follow two rules: every property is listed in required, and additionalProperties is false. A field the model may leave empty (like access_notes above) stays optional by allowing null in its type — it is still listed in required.

FunctionTool(
    name="book_inspection",
    description="Book a site safety inspection. Use when "
                "the user asks to schedule an inspection.",
    strict=True,
    parameters={
        "type": "object",
        "properties": {
            "site_code": {"type": "string",
                          "description": "Site code, e.g. LDN-04"},
            "kind": {"type": "string",
                     "enum": ["fire", "electrical"]},
            "access_notes": {"type": ["string", "null"]},
        },
        "required": ["site_code", "kind", "access_notes"],
        "additionalProperties": False,
    },
)

Strict mode applies to function tools you define. It doesn't affect which tool is chosen, and it isn't a switch on MCP or search tools.

In Foundry Agent Service the service never runs your functions: when the model decides to call one, the response's output contains a function_call item (with name, JSON arguments and a call_id) and no final text. Your application runs the function and sends the result back as a function_call_output item carrying the same call_id, normally on the same conversation; the model then writes the answer.

HANDLERS = {"check_room": check_room, "hold_room": hold_room}

pending = [i for i in response.output if i.type == "function_call"]
outputs = [
    {
        "type": "function_call_output",
        "call_id": call.call_id,   # echo the call's call_id
        "output": json.dumps(
            HANDLERS[call.name](**json.loads(call.arguments))),
    }
    for call in pending
]
# every output goes back in ONE follow-up request
response = openai.responses.create(
    input=outputs,
    conversation=conversation.id,
    extra_body={"agent_reference": {
        "name": AGENT_NAME, "type": "agent_reference"}},
)

• call_id, not id. The item's own id, the response ID and the function name don't identify the call. (MCP approvals are different — they reference the approval request item's id; see the safeguards section.)
• Parallel calls. One response can hold several function_call items (say, exchange rates for three currencies requested in one question). Run all of them and return every output, each with its own call_id, in one responses.create call. Sending one output per request leaves the other call pending, and the request is rejected. Turning parallel_tool_calls off avoids the error but costs a model round trip per call.
• Polling or resending doesn't help. A response waiting on a function call won't complete by itself, and resending the question just produces another call request.
• The portal doesn't run functions. Function tools are added in code; the Foundry portal can run an agent that has them but doesn't execute the functions or let you edit their definitions, so a playground test stops at the call request. Test the whole loop from code. Code interpreter runs code in its own sandbox, not your function.
• Runs expire after 10 minutes of total elapsed time. If a function's backend can take longer (a payroll recalculation that sometimes runs for half an hour), return a job ID and status immediately and give the agent a second function that checks the job. Background mode and streaming change how the response is delivered; they don't extend the run.

An agent tracks a conversation in one of three ways: a server-side conversation object, previous_response_id chaining of stored responses, or store=false with the application keeping the history. The choice turns on who must hold the history and whether the service may persist it at all.

The agent definition itself is stateless across users — reusing the same agent name and version doesn't make it remember anyone. Conversations, chained responses and memory stores all keep data on the service, so none of them satisfies a "persist nothing" rule, even with short retention or deletion afterwards.

Long conversations are truncated, not trimmed

When a conversation grows past the model's context window, Foundry truncates the input automatically: the conversation still stores every item (listing its items shows them all), but only a subset is sent to the model. That is why something a user said early on — that they're vegetarian, say — can silently stop influencing answers with no error. Raising max_output_tokens or switching to response chaining doesn't change how much history fits. Facts that must last belong in memory, which retrieves them into context when relevant.

Agent memory in Foundry Agent Service is a memory store that extracts durable facts (user profile memory) and summaries (chat summary memory) from conversations and retrieves them into later sessions; an agent reads it through the memory search tool. Memory complements conversation history: it keeps what should survive across conversations and past context truncation, not the full transcript.

Scoping memories to a user

The memory search tool's scope decides whose memory partition is used. A fixed value (such as team-alpha) shares one partition by design. Setting scope to {{$userId}} resolves the user on each request:

1. If the request carries an x-memory-user-id header, its value is the scope.
2. Otherwise the scope falls back to the caller's Microsoft Entra tenant ID and object ID.

So the fallback works when each person's client calls the agent with that person's own Entra token. It fails when a backend calls on behalf of many users with one identity — a managed identity or service principal — because every user then resolves to the same scope and one user's preferences leak into another's chat. That shared-identity caller must send each end user's ID in x-memory-user-id; callers that already present per-user tokens are separated without it. Conversations don't partition memory, and RBAC roles don't change scope resolution.

Memory store settings

Three TTL rules matter in practice. TTL applies only to memory stores created after TTL support was introduced — an older store ignores the setting, so you create a new store with the TTL and attach it to the agent. Expiry counts from a memory's last update, which consolidation resets. And on a TTL store every memory expires, with no exception for details a user explicitly asked to be kept.

An agent retrieves from your content through a knowledge tool attached to the agent: the Azure AI Search tool queries an existing search index through a project connection, and the file search tool searches files you upload into a Foundry-managed vector store. (Deciding which knowledge tool fits a requirement is covered in the AI-103 lesson on choosing Foundry models, retrieval methods and agent tools; this section is about configuring them.)

Azure AI Search tool settings

• Connection and index. The tool's index resource names a project connection and an index_name. One Azure AI Search tool can target only one index; to search a second index, add another tool (or combine the content into one index).
• query_type sets the retrieval method (simple, semantic, vector, vector_simple_hybrid or vector_semantic_hybrid, the default). It changes how results are ranked, not which documents are eligible.
• top_k sets how many results come back — fewer results, same eligibility.
• filter is an OData expression applied to every query the agent makes, such as department eq 'finance'. It is the way to enforce a hard restriction on one shared index without splitting or rebuilding it; an instruction asking the agent to prefer certain documents still depends on the model.
• Citations. The tool returns documents the model can cite as url_citation annotations, but answers show sources only when the agent's instructions ask for citations (with retrievable title and URL fields in the index).

Keyless access. With Microsoft Entra ID authentication on the connection, the tool calls the search service as the system-assigned managed identity of the Foundry account that contains the project. That identity needs Search Index Data Contributor and Search Service Contributor on the search service; without them every call fails with 403, even though the developer's own account can query the index in the portal.

Private networking. A basic agent setup can't reach a search service whose public access is disabled behind a private endpoint — that needs the standard agent setup with virtual network injection. Key-based connections also aren't supported on private networking ("Unable to connect to Azure AI Search Resource" is the symptom), so the connection must use the project managed identity. Both changes are needed.

File search tool settings

store = openai.vector_stores.create(name="warranty-docs")
openai.vector_stores.files.upload_and_poll(
    vector_store_id=store.id, file=open("terms.md", "rb"))

tool = FileSearchTool(vector_store_ids=[store.id])

• Ingestion is asynchronous. Files are parsed, chunked and embedded after upload. Use the polling helpers (upload_and_poll, create_and_poll) or wait until the vector store's status is completed before testing. A run waits up to 60 seconds for files still processing only in the conversation's vector store, not the agent's.
• One store per agent, one per conversation. The agent's file search tool takes one vector store, so content every chat should search goes into that one store — combine sets rather than adding a second agent-level store. Per-chat uploads belong in a vector store attached to that conversation, which keeps them private to the chat.
• Conversation stores expire seven days after they were last used in a run, and later runs on that conversation fail; recreate a vector store with the same files and reattach it.
• Supported files include .pdf, .docx, .doc, .pptx, .md, .txt, .html, .json and common code files (text in UTF-8, UTF-16 or ASCII). Spreadsheet formats such as Excel workbooks aren't on the list.

Beyond retrieval, an agent reaches other systems through four kinds of tool: an OpenAPI tool for a REST API described by a specification, an MCP tool for a Model Context Protocol server, a function tool for code your app runs, and a Foundry toolbox that shares a curated set of tools across many agents.

OpenAPI tool requirements

• The specification must be OpenAPI 3.0 or 3.1; a Swagger 2.0 file must be converted first.
• Every operation needs an operationId made of letters, - and _ only (so a dotted name like orders.list fails).
• Request bodies must be application/json (or application/json-patch+json); form-encoded bodies aren't supported.

MCP tool configuration

An MCP tool on an agent takes a server_label, a server_url, an optional project_connection_id holding the server's credentials, and require_approval (covered in the safeguards section). An "Invalid tool schema" error usually means a tool definition on the server uses anyOf or allOf, or declares a parameter with more than one type — fix the server's definition, for example by splitting the parameter into single-typed ones. Non-streaming MCP calls time out after 100 seconds.

Content understanding and custom functions

Azure Content Understanding analyzers extract fields from documents, images, audio and video into a defined schema — one you write, or a prebuilt analyzer's — with confidence scores. An agent uses it like any other API — through a function tool whose code calls the analyzer, or an API-backed tool over its endpoint — and then reasons over the extracted fields. Custom business logic that must run in your environment (reading files on the user's device, calling an internal system with app credentials) is a function tool.

Foundry toolbox

A toolbox defines a set of tools once, injects credentials and refreshes tokens centrally, and exposes them through one MCP-compatible endpoint — so agents built with Foundry, Microsoft Agent Framework, LangGraph or custom code can share them. It addresses tool sprawl: many agents each repeating the same ticketing-system MCP or OpenAPI configuration with their own copy of the secrets.

• Supported tools include MCP, OpenAPI, Azure AI Search, file search, web search, code interpreter and A2A. Function calling can't go in a toolbox (the client runs the function), and some tools — SharePoint, Grounding with Bing, image generation, computer use — are supported only directly on the agent.
• Versions and endpoints. The developer endpoint /toolboxes/{name}/versions/{version}/mcp targets one version for validation. The consumer endpoint /toolboxes/{name}/mcp always serves default_version. Creating a version doesn't change the default: test the new version on its developer endpoint, then promote it, and agents on the consumer endpoint pick it up with no redeployment.
• Tool search. With hundreds of tools, sending every definition costs input tokens and lowers selection accuracy. Enabling tool search hides tools behind tool_search and call_tool so the model fetches what it needs; pinning tools in tool_configs keeps frequently used ones always visible.

Multi-agent solutions in Foundry are built in two ways: an agent delegates to another agent as a tool through the A2A (agent-to-agent) protocol, or a workflow orchestrates several agents in Microsoft Agent Framework. The difference is who owns the conversation.

Delegation with the A2A tool

The A2A tool connects a Foundry agent to any A2A-compatible endpoint — another Foundry agent or one built with LangGraph or another framework. The calling agent keeps the conversation: it sends the request, receives the result and combines it with its own answer, with no relay code in your app. Setup has a direction:

• Enable incoming A2A on the target agent so it exposes an endpoint.
• Add the A2A tool to the calling agent, through a project connection to that endpoint.
• For a Foundry target, use agentic identity auth with the audience https://ai.azure.com and grant the calling agent's identity the Foundry Agent Consumer role on the target's project — the least-privilege role that allows calling agent endpoints.

Connected Agents, the classic API's sub-agent feature, isn't available in the new Foundry Agent Service. A function tool could call another agent, but it puts the protocol, authentication and loop in your code.

Orchestration patterns in Microsoft Agent Framework

A handoff transfers the user to another agent; the A2A tool doesn't. Choose by whether the specialist must talk to the user itself.

Where orchestration runs

Foundry portal (visual) workflows retire on 1 December 2026. For new or existing workflows, Microsoft recommends Microsoft Agent Framework workflows — defined in code (or declarative YAML) and kept in source control, with built-in orchestrations and human-in-the-loop support — deployed to Foundry as a hosted agent. Agent Framework has no visual graph editor, so moving a portal workflow is a rebuild in code. Azure Logic Apps is the documented route for teams that want a low-code visual designer. A single prompt agent whose instructions describe several steps can't enforce their order or an approval gate.

A semiautonomous agent acts on its own for low-risk steps and pauses for a person before risky ones. The safeguard must be enforced by the platform or framework, not requested in the prompt: an instruction to "check with someone first" is a suggestion to the model, and a flag check inside the function only makes the call fail instead of pausing for a decision.

MCP tool approvals

The MCP tool's require_approval takes four forms:

allowed_tools is a different control: it removes tools from the agent entirely rather than gating them. When approval is needed, the response contains an mcp_approval_request item (server label, tool name, arguments). The app answers with an mcp_approval_response whose approval_request_id is that item's id, in a follow-up request that passes previous_response_id:

decisions = [
    {
        "type": "mcp_approval_response",
        "approval_request_id": req.id,   # the request item's id
        "approve": policy_allows(req.server_label, req.name),
    }
    for req in response.output
    if req.type == "mcp_approval_request"
]
response = openai.responses.create(
    input=decisions, previous_response_id=response.id, ...)

Human-in-the-loop in Microsoft Agent Framework

Approval mechanisms decide whether a step may run; checkpoints make the wait durable. A long wait across restarts usually needs both. An open streaming connection or a background response doesn't survive a process restart or gate anything.

Long-running agent work

• Background mode (background=True) returns immediately while the agent runs asynchronously; poll responses.retrieve while the status is queued or in_progress, then read the output.
• Slow MCP tools hit the 100-second synchronous timeout. Run the agent in background mode (a preview capability, available only on models that support it) and have the MCP server run the job as an MCP task that returns a related-task ID the runtime can poll.
• Slow function tools hit the 10-minute run expiry; return a job status at once and add a status-check function.

You monitor and evaluate an agent at two levels: agent evaluators score whether its behaviour was right (intent, task, each step of tool use), and traces show what actually happened in each run so you can find where it went wrong. (Setting up tracing, token analytics and latency breakdowns is covered in the AI-103 lesson on optimizing and operationalizing generative AI systems; this section uses them for error analysis.)

Choosing an agent evaluator

Read a failure step by step: calls that succeed but an answer quoting a total the tool never returned is an output-use problem; an amount written as "fifty euros" where the schema wants a number, or an undefined extra parameter, is an input problem; a complete answer that breaks a stated rule is an adherence problem. Task Navigation Efficiency's matching_mode sets strictness: exact_match (same steps, same order, nothing extra), in_order_match (all expected steps in order, extra steps allowed) and any_order_match (order ignored).

Error analysis with traces

• Tool spans without arguments. With AZURE_EXPERIMENTAL_ENABLE_GENAI_TRACING on, the SDK emits GenAI spans such as execute_tool, but message content and tool-call arguments are recorded only when OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT is true, set before instrumenting. Content recording can capture personal data, so keep it to development unless compliance allows it.
• Your own code between calls. The instrumentor traces SDK calls only. Decorate helpers with @trace_function (from azure.ai.projects.telemetry) to get a span per call, with parameters as code.function.parameter.<name> and the return value. These spans record parameters regardless of the content-recording setting, so keep sensitive values out of traced functions' arguments (pass an opaque record key rather than names or personal details).
• Linking traces to the agent. Client-side traces correlate with an agent in the Foundry portal when responses.create passes agent_reference with both the agent's name and id.