Computer Vision and Image Generation in Microsoft Foundry (AI-901)

Pick by the direction of the data: if an image goes in and words come out, you need a multimodal (vision-enabled) chat model; if words go in and an image or video comes out, you need a generative image or video model. Microsoft Foundry hosts both kinds in its model catalog, and they are deployed and called differently.

A text embedding model is neither: it turns input into vectors for search and similarity, and never describes or draws a picture. And not every chat model accepts images. If a request that includes an image part fails with an error saying image input is not supported, while the same request without the image works, the deployment is a text-only model. The fix is the model, not the message: deploy a vision-capable model and point the app at that deployment.

This lesson is about building with these models. What computer vision and image generation are as workloads is covered in the Task 1.3 lesson, and extracting fields or text from images into a fixed schema with Content Understanding is covered in the Task 2.4 lesson.

Use the playground that matches the model: the model (chat) playground for image understanding and the image playground for image generation. Both open from a deployment in the Foundry portal and let you experiment with no code.

• Model playground (chat). With a vision-capable chat deployment selected, attach an image to the chat and type a question about it, such as asking what is damaged in a photo. You see exactly how the model interprets your images and can refine the system instructions and prompt before coding.
• Image playground. When you deploy a text-to-image model, the portal offers an image playground where you type a prompt, choose options such as size and number of images, and review the generated pictures. Uploading a photo to a chat is not how you test image creation.
• Video playground. Sora deployments have a matching playground for trying text-to-video prompts.

The playgrounds are the fastest way to see how a deployment behaves; heavier tools for improving or measuring a model at scale come later. The playgrounds can also show sample code for the settings you tried, which is a convenient starting point for an app.

Send one user message whose content is a list of parts: an input_text part carrying the question and an input_image part carrying the picture. With the Responses API and the OpenAI Python SDK pointed at the Foundry resource's /openai/v1/ endpoint (client setup is covered in the Task 2.1 lesson), a lightweight vision app looks like this:

response = client.responses.create(
    model=deployment_name,          # a vision-capable chat deployment
    instructions="You inspect parcel photos for a shipping company.",
    input=[{
        "role": "user",
        "content": [
            {"type": "input_text", "text": "Which part of this parcel is damaged?"},
            {"type": "input_image", "image_url": photo_url}
        ]
    }]
)
print(response.output_text)

• The part types are input_text and input_image. Putting a URL inside an input_text part only gives the model the characters of the link, not the picture; input_file is for documents such as PDFs; output_* types describe what the model returns, not what you send.
• Don't mix up the two APIs' part names. The older Chat Completions API uses text and image_url as its content part types. In the Responses API those names are replaced by input_text and input_image, and image_url survives only as a field inside an input_image part (alongside the optional detail), not as a part type of its own.
• System-level guidance (role, tone, what to look for) goes in instructions or a system/developer message; the image itself belongs in the user message.
• The text answer is read from response.output_text, exactly as in a text-only chat app.

The model can receive an image in three ways, and the right one depends on whether the model's service can fetch the image itself.

The service fetches URLs itself, so a URL must be publicly reachable. Images behind a private endpoint, inside a virtual network, on localhost or addressed as file:/// paths cannot be fetched, and granting the Foundry resource's managed identity access to storage does not change how the image URL is fetched. When the image must stay private, have the app read the bytes and send them inline as a base64 data URL (or upload them as a file and pass the file_id).

A data URL has a fixed shape: data:, the image's MIME type (for example image/jpeg or image/png), ;base64,, then the base64 text. In Python that means base64.b64encode(data).decode("utf-8") and then f"data:image/jpeg;base64,{b64}". Base64 changes how the image travels, not how many tokens the model spends looking at it.

Use the detail setting on each input_image part, and choose the model size, to trade visual precision against tokens, latency and cost.

Two practical rules follow. If a high-volume job only needs coarse labels and input tokens are the problem, setting detail to low is the direct fix. If the model misreads small text after you lowered detail, set it back to high. The principle: only a higher detail level adds visual information. Settings that shape the reply (prompt wording, sampling, output length) cannot restore pixels the model never saw, and they do not change the tokens spent reading the image.

The second lever is the model itself. A large flagship model is wasted on simple sorting; a smaller, cheaper vision-capable model often classifies just as accurately at a fraction of the cost. As a rule, cost falls when the model or the image input gets lighter, not when the model is asked to do more work.

To have the model reason across several pictures, put every image and the instruction in the same request, each image as its own input_image part. A model only sees what is in the current request: sending images in separate calls and asking the second call to "remember" the first does not work unless the earlier context is passed along, and one input_image part holds exactly one image, so you cannot list two URLs in it.

"content": [
    {"type": "input_text", "text": "List every change between last week's building-site photo and today's."},
    {"type": "input_image", "image_url": last_week_url},
    {"type": "input_image", "image_url": today_url}
]

Supported image formats include PNG, JPEG, WEBP and non-animated GIF. An animated GIF is processed as a single frame (the first one), so a model only ever "sees" what that frame shows. Because the other frames never reach the model, no setting or prompt can recover them; when several frames matter, extract those frames in the app and send each as a separate input_image part. (Analyzing real video content is a different task: Content Understanding, in the Task 2.4 lesson, handles video.)

Structured output for app code

Give the model clear instructions and, when code consumes the answer, request structured output with a JSON schema. Without a schema the model may describe a photo in a different layout each time, which breaks a parser even when the content is right.

• Instructions set the task and constraints: what to look for, what to ignore, what to do if the image is unclear.
• A JSON schema (passed as the structured output format of the request) fixes the shape of the reply, for example an array of objects with defect and location fields for an inspection photo. The model's reply then conforms to the schema, so the app can load it as JSON.
• The principle: only a schema fixes the shape of the reply. Prompt wording and image settings change what the model notices and how it phrases things, not the structure your parser receives.

When the goal is to pull fixed fields out of many standard images or documents (invoices, IDs, receipts) with confidence scores, Content Understanding analyzers are the purpose-built option; see the Task 2.4 lesson.

Call client.images.generate() on an image generation deployment with a text prompt; GPT-image models return each picture as base64 in data[i].b64_json, which the app decodes and saves.

import base64

img = client.images.generate(
    model=image_deployment,
    prompt="A flat-style banner of a lighthouse at dawn",
    n=1,
    size="1024x1024",
    quality="medium",
    output_format="png"
)
with open("banner.png", "wb") as f:
    f.write(base64.b64decode(img.data[0].b64_json))

Two common mistakes: GPT-image models do not return download URLs and do not accept response_format, so code that reads data[0].url fails; and the base64 string must be decoded before writing it to disk, because writing the encoded text produces a file no image viewer can open.

Choosing an image model is a fidelity-versus-cost decision. A lighter model such as GPT-Image-1-Mini suits high-volume, low-stakes work like rough thumbnails and drafts; the flagship GPT-image models suit brand-critical visuals where detail and prompt accuracy matter most; FLUX models are a third-party alternative in the same catalog. A cheap chat model that accepts images is not an image generator at all.

Call client.images.edit() with one or more input images and a prompt; add an optional mask to restrict which pixels may change. Use editing whenever the result must contain the real content of an existing picture. images.generate only knows your words, so describing a mascot in text produces a new mascot, not yours.

A mask must follow two rules:

• It is a PNG with an alpha channel, and its fully transparent pixels mark the area the model may change; opaque pixels are kept. (It is the transparent region that gets edited, not the opaque one.)
• It has exactly the same dimensions as the input image. A smaller or JPEG mask is rejected, since JPEG has no transparency.

A vision chat model cannot do this job: it reads images and replies in text, it does not return a composed picture.

Give a Responses API request the image_generation tool: the chat model decides when to draw, a GPT-image deployment renders the picture, and the result comes back as an image_generation_call item in response.output whose result holds the base64 image.

response = client.responses.create(
    model=chat_deployment,
    input="Design a poster for a jazz night",
    tools=[{"type": "image_generation"}]
)
images = [o.result for o in response.output if o.type == "image_generation_call"]
with open("poster.png", "wb") as f:
    f.write(base64.b64decode(images[0]))

follow_up = client.responses.create(
    model=chat_deployment,
    previous_response_id=response.id,
    input="Make the background deep blue and add the title 'Jazz Night'",
    tools=[{"type": "image_generation"}]
)

The tool needs a GPT-image deployment as well as the chat deployment. The snippet above is simplified: in Foundry the image deployment is configured for the tool separately (the Responses how-to passes its name in the x-ms-oai-image-generation-deployment request header), so make sure both deployments exist.

This pattern fits chat-style design assistants where each turn refines the last picture. Passing previous_response_id links the new request to the earlier one, so the service carries the conversation, including the previous image, and the app does not rebuild and resend the whole history each turn. Calling images.generate afresh on every turn loses the earlier image, and a vision chat model on its own can describe the last poster but cannot redraw it.

Note where the output lives: each API has its own response shape. The Responses tool returns the image inside an output item; output_text carries only text, and data[i].b64_json belongs to the images API.

Deploy a Sora model and treat video generation as an asynchronous job: create it, poll its status by ID, and download the content only when it has completed. Rendering a clip takes far longer than an HTTP request should stay open, so the create call returns a job, not a video.

1. Create. POST /openai/v1/videos with the Sora deployment, the prompt and options such as size and duration. The response contains the video's id and an initial status (queued or in progress).
2. Poll. GET /openai/v1/videos/{id} every few seconds until the status is completed (or failed).
3. Download. GET /openai/v1/videos/{id}/content returns the MP4.

Image-to-video. Sora 2 can also start a clip from a still image: pass the picture as input_reference alongside the prompt and size, and it becomes the clip's starting frame. The rule to remember is that the reference image's resolution must match the requested video size exactly, and the size must be one of the landscape and portrait sizes the model supports, so a photo with different dimensions must be cropped or resized to the target size first. In the Python SDK the same create step is client.videos.create() (with input_reference as an argument), followed by the same poll and download steps. Remixing is a different feature: it modifies a clip you already generated and takes that completed video's ID, not an image.

The principle: the create call only ever returns a job. Every create starts a new job, and the MP4 is fetched separately once that job reports completed.

Responsible AI safeguards apply to every generated image and video. Foundry runs prompts (and outputs) through its content filtering system. When a prompt is judged harmful, for example a request for hateful or sexual imagery, the call fails with a content-filter error and no image is returned, while ordinary prompts keep working. That pattern, failing only for certain prompts, points to the safety system rather than to quota, an unsupported size or a wrong deployment, which would fail regardless of the prompt's content. The principles behind these safeguards are covered in the Task 1.1 lesson.

Tightening the filter is a per-deployment setting. The content filter scores categories such as violence, hate, sexual and self-harm content by severity, and blocks anything at or above each category's threshold. If the default lets milder content through that your audience should not see, create a custom content filter with a stricter (lower) threshold for that category and assign it to the image or video deployment. Don't confuse this with Prompt Shields: they detect jailbreak and prompt-injection attempts, meaning users trying to override the model's instructions, and they do not change the severity level at which a category is blocked.