AI-103: Multimodal Understanding with Vision Models and Azure Content Understanding

You analyze an image with a multimodal model by sending it to a vision-capable model deployment in Microsoft Foundry as an image content part alongside your text instructions, then reading the model's text answer. With the OpenAI-compatible Responses API, a user message's content array mixes input_text parts and input_image parts, and the model reasons over all of them together.

An input_image part carries the image in its image_url in one of two forms:

• A URL the service can fetch — for example, an image in Azure Blob Storage that the service can reach.
• A base64 data URL such as data:image/jpeg;base64,<data> — the pixels travel inside the request, so nothing has to be stored anywhere first. Use it when an image can't be staged anywhere the service can reach, or exists only on the caller's device.

A path on the caller's disk is neither: the service can't open C:\photos\x.jpg on your laptop. Likewise, putting a file name into an input_text part gives the model no pixels, and the file_search tool retrieves text chunks from documents in a vector store — it doesn't show the model an image.

import base64

with open("sticker.jpg", "rb") as f:
    data_url = "data:image/jpeg;base64," + base64.b64encode(f.read()).decode()

response = client.responses.create(
    model="gpt-4.1-vision",          # your deployment name
    input=[{
        "role": "user",
        "content": [
            {"type": "input_text", "text": "Read the small print on this warning sticker."},
            {"type": "input_image", "image_url": data_url, "detail": "high"},
        ],
    }],
)
print(response.output_text)

Image analysis with a chat model is the most flexible option: one deployment can caption, answer questions, compare images and write alt text, all controlled by instructions. When you need the same structured fields from many images or videos, repeatably, Azure Content Understanding (covered below) is usually the better fit.

What the detail setting controls

The detail setting controls the resolution at which the model looks at the image, and therefore the trade-off between cost and fine detail. It doesn't control how long or how detailed the answer is.

The classic symptom of detail set too low is a model that confidently misreads fine print — swapping look-alike characters — in an image a person can read easily by zooming in. The fix is to raise detail, not to tune other parameters:

• Temperature 0 makes output more repeatable, but a model reading an illegible image then repeats the same misread.
• More output tokens changes the length of the answer, not the input resolution.
• Pre-shrinking images to 512 x 512 throws away exactly the detail you need.

You control caption style with instructions: state the length (one short sentence, or a paragraph), the audience and what to cover (materials, colour, condition, standout features). The same vision-capable deployment produces both styles — one prompt per style, chosen per request — so there's no need for separate deployments.

• Fine-tuning two deployments for two caption lengths would work, but it costs training data, training runs and extra deployments to get what an instruction already gives you.
• detail changes the input resolution, not the output length.
• A low max_output_tokens truncates a long caption mid-sentence; it doesn't summarise. Ask for brevity in the instructions instead.

Several images in one request

A single chat request can carry several image parts — Microsoft documents a limit of 10 images per chat request for vision-enabled chat models. Sending a set of related images together lets the model reason across them and write one caption about the whole set (a vehicle photographed from four sides, a product from several angles).

When you need a caption per image, use one request per image (or ask for a numbered caption per image in one request). When you have more images than one request accepts — frames from a long video, for instance — your code has to sample and batch them.

Question answering grounded in visual evidence means the model answers only from what the image or video actually shows, says so when the evidence doesn't contain the answer, and can point to where the evidence is. Hallucinated answers about images usually have two causes, and each needs its own fix:

1. Nothing restricts the model to the image. Instruct it to answer only from visible content and to reply that it can't tell when the image doesn't show the answer. An abstain path removes the pressure to invent a plausible answer.
2. The evidence is hard to read. Small labels, handwriting or fine detail need detail set to high so the model can read what it must cite.

Things that look helpful but don't ground answers: pasting long reference lists of every possible item name into the system message gives the model more names to borrow; temperature 0 only makes a wrong guess consistent; fine-tuning on past answers teaches typical answers, not the contents of a new photo.

Grounding answers in video

For a library of recorded videos, you don't send video to a chat model on each question — a chat request takes a handful of images, not hours of footage. Instead, build a retrieval pipeline (RAG over video):

1. Analyze each video once with Content Understanding's prebuilt-videoSearch analyzer. It returns RAG-ready output per segment: a natural-language description of what is visible, the transcript and the segment's start and end times.
2. Index those timed segments in Azure AI Search.
3. At question time, retrieve the relevant segments and instruct the model to answer from them and cite their start times.

Because the segment descriptions come from the frames as well as the audio, actions that appear on screen without being described aloud are findable. A transcript-only index (for example from prebuilt-audioSearch) misses them, and fine-tuning on transcripts gives the model neither the visual steps nor timestamps to cite.

Accessible alt text conveys the purpose and information of an image in its context, concisely, so screen reader users get what sighted readers get. W3C's WAI images tutorial, which explains how to meet WCAG success criterion 1.1.1 (Non-text Content), is the reference; a generation pipeline encodes it in the model's instructions and first decides what kind of image it's looking at.

Instructions that make good alt text

• Describe purpose in context. Pass the surrounding article text with the image; the same screenshot needs different alt text in different articles.
• Transcribe informative text that appears in the image.
• Don't start with "Image of" or "Picture of" — screen readers already announce that the element is an image.
• Don't inventory every colour, size and position. Styling isn't the information; exhaustive detail hides the point.
• Leave out file metadata (file name, format, pixel size).

Two traps for charts: packing every value into the alt attribute creates a long, unstructured announcement; leaving the alt empty and putting the data in a title attribute hides the chart, because an empty alt marks it as decorative and title text isn't reliably announced.

Azure Content Understanding in Foundry Tools is a service that turns documents, images, audio and video into structured, schema-defined output by using generative models. You call an analyzer — either a prebuilt one or a custom one you define with a fieldSchema — and get back Markdown content plus the field values you asked for.

Base analyzers and prebuilt analyzers

A custom analyzer must derive from one of four base analyzers via baseAnalyzerId: prebuilt-document, prebuilt-image, prebuilt-audio or prebuilt-video. For photos whose value is in what they show (parking-space occupancy, crop health, scene content) the base is prebuilt-image.

Alongside the bases, Content Understanding ships ready-made analyzers for common jobs — for example prebuilt-imageSearch, prebuilt-videoSearch and prebuilt-audioSearch, which produce descriptions and transcripts designed for retrieval (RAG) indexes, and document analyzers such as prebuilt-layout. These are finished analyzers, not bases for your own fields, and prebuilt analyzers can't be edited.

Field methods

When a downstream system groups results by a fixed set of categories, use classify with an enum: generate can return wording outside the list, and omitting method leaves the choice to the service. Image analyzers support generate and classify, not extract.

Text-heavy images belong in document analyzers

Microsoft notes that image analyzers aren't optimized for analysis based mainly on text. A photo of a receipt, a form or an ID card is really a document: analyze it with a document analyzer (based on prebuilt-document), which runs OCR, supports extract fields and can return each field's source location and confidence when estimateFieldSourceAndConfidence is enabled (or estimateSourceAndConfidence on individual fields). Confidence and grounding locations are what let you send low-confidence values to human review — and per-field source and confidence are a document analyzer feature, so they aren't available from an image analyzer. Setting returnDetails on an image analyzer adds detail to the response, but it isn't the switch for per-field confidence. Parsing values out of a prebuilt-imageSearch description is no substitute.

Content Understanding uses your own Foundry model deployments for every operation that needs a generative model, so an analyzer only works once suitable models are deployed in your Foundry resource and mapped. If analysis fails because no deployment is mapped, the fix is to deploy models and map them; credentials and role assignments don't create model deployments.

There are two places to map model names to deployments:

So: to make every analyzer on a new resource work without touching each request, deploy supported models and set the defaults. To route one application's requests to a different deployment (a test deployment of a newer model version, say) while all other callers stay on the defaults, pass modelDeployments per request — the override applies to that request only.

A custom analyzer's models property names models from the Foundry catalog (or aliases), never deployment names; the service resolves those to deployments through the defaults or the per-request mapping.

Content Understanding analyzes video either as one whole unit or as segments that you describe, and fills your fields once per unit. The switch is enableSegment in the analyzer's config.

The contentCategories description does the real work: "create a segment for each product the host demonstrates; skip the introduction and closing credits" or "create a segment for each period in which a forklift is in view" defines both where segments start and what to leave out. Set the category's analyzerId to route each segment to a sub-analyzer (for example, a SubAnalyzer that fills product fields for each demonstration segment); a category without an analyzerId creates segments but routes nothing.

Video analyzers support only one contentCategories entry. Document analyzers can define several categories (one per kind of content, each routed differently), but that pattern is rejected for video — use one category whose description says what to skip.

Custom segments vs prebuilt-videoSearch

prebuilt-videoSearch also returns timed segments, but its segments are automatic scene chunks designed for search. When you need time ranges that follow something specific — a presenter on camera, a single demonstration — the boundaries must come from your own category description, not from scene detection.

A Content Understanding audiovisual response always includes the Markdown content, timing information (startTimeMs, endTimeMs), key frame timestamps (keyFrameTimesMs) and your fields. Setting returnDetails to true adds the heavier elements:

• transcriptPhrases — each spoken phrase with its speaker and timing.
• cameraShotTimesMs — the times at which the video cuts to a different shot.

returnDetails only changes what's returned; it doesn't split the video, change the frame rate or alter input limits. Related config options do other jobs: locales helps transcription, omitContent drops the original content object from the response, and segmentPerPage is a document option.

Sampling: what the analyzer can miss

The video analyzer samples roughly one frame per second and scales frames to 512 x 512. Events that last a fraction of a second, or small objects, can fall between samples or be lost in downscaling — and no field description, segmentation or returnDetails setting changes that. For fast, small visual events (a brief flicker on an instrument gauge, an animal darting across a trail-camera frame), extract frames yourself at a higher rate and full resolution and send them in batches to a vision-capable model at high detail (or to a purpose-built detection model, for example one trained with Azure Machine Learning AutoML).

Input size and duration

A long or large recording that fails on direct upload is fixed by storing it and analyzing it by URL. Compressing the file only helps if it also fits the duration limit.

A standard (single-task) Content Understanding analyzer runs one analysis pass over one input to extract, generate or classify fields — it supports every modality and suits direct, field-by-field output such as a vehicle's body type and visible damage from a photo. Pro mode added multistep reasoning, multiple inputs and reference data, but it existed only in the 2025-05-01-preview API, which is retired. Microsoft's documented successor is agentic mode.

The workflow selector accepts "default" or "agentic" — there is no "pro" value, and the GA 2025-11-01 API has neither pro mode nor the agentic workflow. Migrating isn't drop-in: review the schema, because agentic mode isn't a copy of pro mode.

Choosing between them: if the answer is visible or printed (what a photo shows, a printed total), use a standard analyzer. If the answer has to be derived — a net figure calculated from several tables in a financial statement, or totals checked against line items — use an agentic document analyzer. A photo workload always stays standard, because agentic mode doesn't support image analyzers.

You identify objects and components with Content Understanding by shaping the field schema around them: an array of objects returns one entry per item found, and for video, segments give each appearance a time range.

• Images — one record per object. For "every piece of furniture in a hotel-room photo with its name and state", define an array field whose items are objects with a name property and a state property (the state constrained with an enum such as Intact, Scratched, Broken). Each item comes back as its own record, ready for one housekeeping ticket per item. A free-text sentence forces you to parse prose; one photo-level classification can't say which item is broken; an object holding comma-separated lists loses the pairing between each name and its condition.
• Video — when and where something appears. Enable segmentation with one category that creates a segment wherever the object is visible; each segment carries start and end times, and the per-segment fields describe what's in it. Transcript search misses appearances nobody mentions, camera shot times mark cuts rather than appearances, and a whole-video field holds a single value.

What not to build on

Azure AI Vision Image Analysis (v3.2 and v4.0) and Custom Vision are deprecated, with retirement scheduled for 25 September 2028. Microsoft points new builds to Content Understanding and Foundry's multimodal models instead, so treat them as legacy rather than the choice for a new solution. When you need pixel-precise bounding boxes for a custom object class at production-line speed, a purpose-built detection model (Microsoft's Custom Vision migration guidance points to Azure Machine Learning AutoML) is the right tool — that's outside what this exam task tests.

Image generation and editing belong to the sibling task on image- and video-generation solutions, and content filtering, prompt injection through text in images and visual policy rules belong to the responsible-AI task for multimodal content.