Generative AI Apps and Agents in Microsoft Foundry (AI-901)

The system message holds the standing rules that apply to every turn; the user prompt holds the specific request for this turn. A chat request to a generative AI model is a list of role-tagged messages, and each role has a different job:

Because the app sends the system message with every request and the model treats it as high-priority guidance, it is the place for anything that must hold whatever the user types: tone, staying on topic, refusing certain requests. Putting those rules in user prompts depends on each user repeating them, which never happens.

Safety screening is a separate layer. Content filters, which the current Foundry portal presents as guardrails (built on Azure AI Content Safety), detect and block harmful categories of input and output. They are a safety control, not a way to describe the assistant's role, tone or subject area; those behaviours are written as instructions.

A zero-shot prompt gives instructions only; a one-shot or few-shot prompt adds one or several worked examples, usually input and output pairs, so the model can copy the exact behaviour and format. Examples are the most reliable way to get a precise reply shape, such as a single classification label with no explanation, because they demonstrate the output instead of describing it. Describing a format in words leaves the model to interpret the description; an example removes the guesswork.

Few-shot examples condition the model for the current request only; nothing is learned permanently. That is the difference from fine-tuning, which trains a customized model on a dataset before it is deployed.

In a chat API, examples are supplied as example user and assistant turns placed after the system message and before the real user message. The model reads them as earlier conversation that shows the pattern, then answers the final user message in the same way:

messages = [
  {"role": "system", "content": "Rate the sentiment of each product review as Positive, Negative or Mixed. Reply with the rating only."},
  {"role": "user", "content": "Arrived quickly and works perfectly."},
  {"role": "assistant", "content": "Positive"},
  {"role": "user", "content": "Great screen, but the battery dies by lunchtime."},
  {"role": "assistant", "content": "Mixed"},
  {"role": "user", "content": "<the next review>"}
]

The examples work because they sit inside the same request as the real input: a model only sees what the current request contains, plus anything the request is explicitly chained to.

You ground a prompt by putting the facts the answer needs into the prompt and telling the model to answer only from them. A model knows nothing that happened after its training data was collected and nothing about your private documents, so a policy written last month or an internal manual must be supplied as grounding data. Telling the model "do not invent facts" on its own does not help much: it still lacks the facts.

Microsoft's prompt-engineering guidance adds two techniques that make invented content harder to slip through:

• Give the model an "out". Tell it what to reply when the supplied text does not contain the answer, for example "not found" or "I don't have that information". Without an allowed alternative, a model tends to fill the gap with a confident guess.
• Ask for citations. Require it to quote or reference the passage that supports each statement, which ties every claim to the source and makes unsupported claims visible.

The underlying principle: fabrication is a missing-facts problem. Generation parameters and style requests change how the model writes, not what it knows, so the fix is always to supply the facts and constrain the answer to them. For large or changing document sets, grounding is automated with retrieval, for example the file search or Azure AI Search tool on an agent (covered below).

To get parseable output, state the exact structure in the prompt and show an example of it. If an app reads each reply as JSON with the fields task and owner, the prompt should describe those fields and include a sample object; you can also "prime" the output by starting it, for example ending the prompt with the opening of the expected format. Generation parameters are not a substitute: a max output tokens limit truncates a reply and a stop sequence ends it as soon as a chosen string appears, so both only decide where the reply stops, not what it contains or how the model reads the prompt; a low temperature makes replies more repeatable in whatever format the model happened to choose. Only an explicit format, ideally demonstrated, defines the structure.

The "out" must fit the structure too. Giving the model an allowed answer for missing information (see grounding above) still applies when an app parses the reply, but that answer has to be expressed inside the format. If a value is absent from the source, the prompt should say how to represent it in the structure, for example null for that JSON field. A plain-text "not found" in place of the object, or any extra line written before or after the object (a note, a source line), breaks the parser just as a free-text reply does. Put supporting detail, such as a citation, in a field of the structure if the app needs it.

Clear prompts follow a few habits from the same guidance:

• Be specific about the task, audience and length instead of leaving the model to guess.
• Use clear syntax: delimiters such as ---, headings or quotation marks mark where pasted content starts and ends, and the prompt should also say what the delimited block is, for example "the text between the --- lines is a document to summarize". Without both, a sentence inside the pasted content (a request in an email, say) can be read and obeyed as if it were part of your instructions.
• Break the task down into smaller steps when one prompt asks for too much.
• Mind the order (recency bias). Information at the end of a prompt can influence the output more than information at the start. State the task clearly at the start; when a long document then sits between the instruction and the end of the prompt and replies drift, repeating the instruction after the content is worth testing, and evaluating the effect.

One caveat from the same Microsoft guidance: these techniques are written for standard chat models and are not recommended for reasoning models (such as the o-series and gpt-5 models), which plan their own steps and respond best to short, direct instructions.

You pick a model in the Microsoft Foundry model catalog and deploy it to your Foundry resource; the result is a named deployment, and both the playground and application code use that deployment. The standard order is: choose a model, check it is available in your region, deploy it, then test it. Fine-tuning, agents and hosted agents are later, optional steps that all build on a deployed model. (Foundry also previews "instant" models that can be used without a deployment in limited regions; deploying remains the standard workflow.)

The deployment name is what code passes as the model. When you deploy, you name the deployment, and that name, shown in the Name column of the project's deployed models, is the value of the model argument. Tutorials often name the deployment after the model (gpt-4o deployed as "gpt-4o"), which hides the difference: if gpt-4o is deployed as trip-helper, code passes model="trip-helper". The resource and project names go into the project endpoint instead:

https://<resource-name>.services.ai.azure.com/api/projects/<project-name>

The model playground is where you interact with a deployment without code:

• Write a system message and adjust parameters such as temperature and max output tokens, then chat to see the effect.
• Compare up to three models side by side: the input is synchronized, so each receives the same prompt, system message and parameters, and the responses stream next to each other.
• The Code tab gives sample code in several languages for the deployment you are testing, and Open in VS Code (for the Web) imports the sample and endpoint into an editor.
• Save as agent turns the current model, instructions and tools into an agent (see below).

A Foundry SDK chat client in Python creates an AIProjectClient from the azure-ai-projects package, gets an OpenAI client from it with get_openai_client(), and calls responses.create with the deployment name:

from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient

project = AIProjectClient(
    endpoint="https://fabrikam-ai.services.ai.azure.com/api/projects/travel-project",
    credential=DefaultAzureCredential(),
)
client = project.get_openai_client()

response = client.responses.create(
    model="trip-helper",                       # the deployment name
    instructions="You are a patient booking assistant.",
    input="Can I change my flight to Friday?",
)
print(response.output_text)

The pieces and their jobs:

Text generation always goes through the OpenAI client that the project returns; the project client's own operation groups manage project resources rather than answering prompts. Libraries for individual Azure AI services, and older standalone inference clients, are separate packages from the Foundry project SDK.

Authentication is Microsoft Entra ID only for the project client; you do not pass an API key. DefaultAzureCredential needs a signed-in identity, which on a developer laptop usually comes from az login, and that identity needs a role on the Foundry resource or project. Foundry User (formerly named Azure AI User) is the least-privilege role for developers. So a new machine needs both: a signed-in identity for the credential to use, and a role assignment that authorizes that identity.

Each responses.create call is independent: the model sees only what that call sends, so a follow-up such as "How long does it take?" makes no sense to it unless the call is linked to earlier turns. The Responses API offers two service-side ways to link them, so the client does not have to resend the history:

• previous_response_id: pass the id of the last response, and the service supplies that exchange (and the chain before it) as context.
• A conversation: create one with conversations.create() and pass its id on every call; the service stores the items of each turn in it.

first = client.responses.create(model="trip-helper", instructions=RULES, input="Suggest a scenic train route in Switzerland.")
follow = client.responses.create(
    model="trip-helper",
    instructions=RULES,                      # sent again on every call
    previous_response_id=first.id,
    input="How long does it take?",
)

Two details decide many design choices:

• instructions apply only to the response they are sent with. They are not carried over when you chain with previous_response_id, and a conversation stores message items, not instructions. To keep system rules on every turn, send them with every call.
• Chaining needs stored responses. The service can only supply an earlier exchange it has kept. Responses are stored by default; a response created with storage switched off cannot be chained to later.

The older pattern, a system message at the top of a messages list with the whole history resent each turn, also works, but it keeps the history on the client. Only these explicit links carry context; nothing else about two calls (the same deployment, the same client object) connects them.

A prompt agent in Foundry Agent Service is an agent defined entirely by configuration: a model deployment, instructions and optional tools. You create it in the Foundry portal (or with the SDK) without writing agent code, and Foundry runs it for you. The definition is stored in the project, and each saved change creates a new version of the agent. A hosted agent is the code-first alternative: you package your own agent code in a container that Foundry runs.

Moving behaviour from a raw model call into a prompt agent changes where it lives: the instructions sit in the agent definition, so the client sends only the user's input. Because every client calls the agent by name, editing its instructions or tools in the portal changes how it behaves for all of those clients at once, with no change to or redeploy of the client code. The agent still uses a model deployment, and it still needs a conversation or previous response id to remember earlier turns.

Tools extend what the agent can do. Choose them by what the task needs:

The near-twins to keep apart: code interpreter runs code in an isolated sandbox, so it cannot reach your internal network; file search retrieves text from uploaded files; a function tool's code runs in your app. Match the tool to where the data lives and how fresh it must be: public and current, uploaded documents, an existing index, or a system only your application can reach.

You test an agent in the agents playground of the Foundry portal before writing any client code. There you can set and edit the agent's instructions, add tools and knowledge, hold multi-turn conversations, and inspect tracing for each response (which tools were called, with what input and output) and evaluation data. Tuning the instructions there and re-testing is the quickest loop; each saved change becomes a new agent version.

A client app chats with an existing agent by getting an OpenAI client from the project that is bound to the agent, creating one conversation, and passing that conversation's id with every responses.create call, so every turn shares the history:

client = project.get_openai_client(agent_name="TripPlanner")
conversation = client.conversations.create()

for question in ["Plan three days in Lisbon.", "Make day two cheaper."]:
    response = client.responses.create(conversation=conversation.id, input=question)
    print(response.output_text)

Two principles sit behind this pattern. The conversation is created once and reused, because it is the conversation that holds the history. And it is the agent binding that applies the agent's stored instructions and tools; managing the agent's definition (creating versions) is a separate, administrative task from chatting with it.

Function tools need a round trip through your app. The service never runs a function tool's code. When the model decides to use one, the response contains a function_call item, with the function name, its arguments and a call_id, instead of answer text. The app then:

1. reads the function name and arguments from the function_call item;
2. runs its own function (for example, the flight-status lookup);
3. sends the result back in the same conversation (or chained to the response) as a function_call_output item carrying the same call_id;
4. reads the model's final answer from the next response.

for item in response.output:
    if item.type == "function_call" and item.name == "get_flight_status":
        result = get_flight_status(**json.loads(item.arguments))
        response = client.responses.create(
            conversation=conversation.id,
            input=[{"type": "function_call_output", "call_id": item.call_id, "output": json.dumps(result)}],
        )
print(response.output_text)

Until the app sends that output back, the model has nothing to answer with, which is why the function tool is the way to reach systems that only your application can see.