Information Extraction with Azure Content Understanding in Foundry (AI-901)

Azure Content Understanding in Foundry Tools is the Microsoft Foundry service that turns documents, images, audio and video into structured output: markdown text plus named fields returned as predictable JSON. Every request names an analyzer, which is a reusable unit of configuration: what kind of content it expects, which processing options are switched on, which models it uses, and (optionally) a field schema listing the values to return.

Because the analyzer is fixed, every file sent to it is processed with the same settings and comes back in the same JSON shape. That is the key difference from simply prompting a chat model with a file: a prompt can return a different structure each time and is not a reusable extraction configuration.

A schema is applied semantically. Each field has a name and a description, and the service finds the value that matches the meaning of the field rather than a fixed label or position. One PolicyNumber field described as "the insurance policy reference" therefore finds the value whether a form labels it "Policy ref", "Pol. no." or prints it unlabelled beside the insurer's logo: a single well-described field covers every layout and label variant.

Analyzers come in two kinds:

• Prebuilt analyzers: ready to call by ID, for common content (invoices, receipts, call recordings, search-ready markdown).
• Custom analyzers: you define them, derived from a base analyzer, when you need your own list of fields (for example the warranty codes and technician IDs on your repair tickets).

Pick a prebuilt analyzer by what you need back: plain content, search-ready content with summaries, or a ready-made set of fields for a known document type. Prebuilt analyzers fall into five groups.

• Known document type, no schema design wanted: use the domain analyzer. Supplier invoices go to prebuilt-invoice (vendor, invoice number, line items, amount due); sales receipts go to prebuilt-receipt, its near-twin. prebuilt-invoice covers more than invoices: utility bills, sales orders and purchase orders use the same schema (dedicated analyzers such as prebuilt-utilityBill and prebuilt-purchaseOrder also exist when you want one tuned to that type), while receipts from shops and restaurants belong to prebuilt-receipt.
• An identity document: use prebuilt-idDocument. It covers passports, driver licences, national ID cards and residency permits from many countries and returns identity fields such as first and last name, date of birth, document number and expiry date, so a check-in or onboarding app can read an ID without a custom schema. Legal agreements go to prebuilt-contract (parties, dates, terms).
• A procurement document of unknown type: use prebuilt-procurement, a composed prebuilt analyzer. It classifies each document (invoice, receipt, purchase order and so on) and routes it to the matching procurement analyzer for extraction, so you get tuned named fields without building or maintaining a classifier of your own. Other composed prebuilts work the same way for their domain, such as US tax forms and US mortgage documents.
• Utility analyzers are not domain schemas: prebuilt-documentFields returns whatever key-value pairs it finds, with generic keys rather than a tuned, named schema (domain analyzers fall back to it when a document matches none of their schemas), and prebuilt-documentFieldSchema analyses sample documents and proposes a field schema, which helps you start designing a custom analyzer. When a domain analyzer exists for the document type, it returns more reliable, consistently named fields than either utility.
• Content for a search index or chat assistant: use the matching …Search analyzer. prebuilt-documentSearch keeps headings and tables in markdown, describes figures such as diagrams and adds a summary of the whole document.
• Just the text, or text plus table structure, at the lowest cost: use prebuilt-read or prebuilt-layout.

The content extraction analyzers, prebuilt-read, prebuilt-layout and prebuilt-digitalParse, run without a generative (chat completion) or embedding model deployment. They perform OCR, layout analysis or digital-file parsing only, so they consume no generative tokens and work on a brand-new Foundry resource before you have deployed anything.

• prebuilt-read returns the text (printed and handwritten) with no structure beyond lines and words.
• prebuilt-layout adds structure: tables with their rows and columns, sections, headings and figures. Choose it when a requirement mentions tables but forbids generative cost.
• prebuilt-digitalParse parses born-digital files (for example a PDF exported from a word processor) directly, without running OCR.

Every analyzer that produces something the file does not literally contain, such as named fields, a summary, a figure description, sentiment or topics, relies on a connected chat completion deployment (and usually an embedding deployment). That covers the RAG analyzers, the domain analyzers such as prebuilt-invoice and prebuilt-callCenter, and custom analyzers with fields, segmentation or classification.

Content Understanding runs its generative steps on your own chat completion and embedding deployments in the Foundry resource, so you deploy those models first and then tell the service which deployment serves each model. There are two places to do that:

• Resource-level defaults: map each model name to a deployment once, in the Foundry portal or Content Understanding Studio (its resource settings can also deploy the models for you), or with a PATCH to /contentunderstanding/defaults. Every analyzer on the resource then resolves its models through that mapping. Prebuilt analyzers refer to their models through aliases such as prebuilt-analyzer-completion and prebuilt-analyzer-embedding, which are resolved through the same modelDeployments mapping. So if the deployments exist but no mapping does, an analyzer such as prebuilt-invoice fails because its required models cannot be resolved to a deployment.
• Per request: pass a modelDeployments object in an analyze request to override the defaults for that call only, for example a one-off evaluation that tries a newer chat deployment while production traffic keeps the default.

An analyzer definition also has a models property (for example models.completion), but it holds catalog model names, not deployment names. The service maps those names to deployments at run time using the defaults or the request's modelDeployments.

You build a custom analyzer by deriving it from one of exactly four base analyzers, named in its baseAnalyzerId property, and adding a fieldSchema with your fields. Only these four can be parents; RAG, layout and domain analyzers cannot. That is different from copying: you can fetch any prebuilt analyzer's definition (GET /analyzers/{id}) and use it as a template for a new custom analyzer, but the copy still declares one of the four as its baseAnalyzerId.

Choose the base by the content, not the file extension. A phone photo of a handwritten inspection checklist is a document that happens to be stored as an image, so it derives from prebuilt-document, which reads printed and handwritten text. A field that depends on what is both shown and said, such as "the dishes cooked and named in a recipe video", derives from prebuilt-video, which combines frames with the transcript.

A minimal definition looks like this:

{
  "baseAnalyzerId": "prebuilt-document",
  "config": { "estimateFieldSourceAndConfidence": true },
  "fieldSchema": {
    "fields": {
      "InvoiceNumber": { "type": "string", "method": "extract",
                         "description": "The supplier's invoice number" },
      "DocumentType":  { "type": "string", "method": "classify",
                         "enum": ["Invoice", "Receipt", "Contract"] }
    }
  }
}

extract returns a value verbatim (documents only), classify picks one value from a fixed list in enum, and generate writes free-form text from the content (any content type). The method is one of three properties on every field in a fieldSchema, alongside its type and description.

• A value restricted to a fixed list is a classification: use classify with the categories in enum. The enum list is what constrains the output.
• extract works only for document content. In audio, image or video analyzers, even a literal value such as a booking reference spoken in a recording is produced with generate.
• extract also needs source and confidence estimation for that field: either set estimateFieldSourceAndConfidence: true in the analyzer config, as in the example above, or set estimateSourceAndConfidence: true on the field itself (the field-level setting overrides the analyzer-level one).

Confidence scores and grounding tell an app how far to trust each extracted value. When estimateFieldSourceAndConfidence is set to true in a document analyzer's config, each extracted field value carries a confidence score between 0 and 1 and its source: the page number and bounding region on that page where the value was found, which a review tool can use to highlight it. An app checks each field's confidence: high values flow straight into the business system, low values go to a human reviewer, who uses the source location to verify them.

Three document settings are easy to confuse because all of them change what comes back:

A single file often contains several documents, such as a mortgage pack that bundles a contract, several bank statements and an ID scan in one PDF. Two settings handle this:

• enableSegment: true splits the content into segments, one per logical document.
• contentCategories describes each category (for example Contract and BankStatement), so every segment is classified, and each category can name an analyzerId (such as prebuilt-contract or a custom statement analyzer) to route its segments to.

For procurement documents the routing step already exists as a prebuilt: prebuilt-procurement classifies each document and hands it to the right procurement analyzer. Build your own contentCategories analyzer when your categories or target analyzers are not covered by a composed prebuilt, or when one file holds several documents that must be split first.

By contrast, segmentPerPage splits strictly by page, which suits files where every page is its own document but cuts apart any document that runs over several pages.

For images without text, Content Understanding describes or analyses what is shown; for images that contain text, it treats them as documents. The choice is:

• A ready-made description of each photo, for example to put in a search index: prebuilt-imageSearch, which returns a one-paragraph description with no field design.
• Your own fields from photos (for example the vehicle type or whether a dent is visible in an insurance photo): a custom analyzer derived from prebuilt-image, using classify or generate fields. prebuilt-image is a base, not a finished analyzer.
• Photos or scans of documents, including handwriting: prebuilt-documentSearch, a domain analyzer, or a custom analyzer derived from prebuilt-document. These read the printed and handwritten text and can extract fields from it.

Analysing images for captions, tags or objects with vision models, and generating images, are covered in the computer vision lesson for this exam.

Audio analyzers transcribe speech and add generated fields; video analyzers combine sampled frames with the transcript and return results per segment.

• Timing: audio content includes transcriptPhrases; each phrase has its own startTimeMs and endTimeMs, which lets an app line text up with the audio, for example to highlight each line of a subtitle track as it is spoken. The prebuilt audio analyzers turn on returnDetails, which is why they return these phrases; a custom analyzer based on prebuilt-audio needs returnDetails: true in its config to get them. The startTimeMs and endTimeMs on the audio content itself only bound the whole recording (or segment), so they cannot locate an individual sentence. cameraShotTimesMs, the list of shot boundaries, is a video-only detail and never appears in audio results.
• Video segmentation: by default prebuilt-videoSearch cuts video into scene-based segments. In a custom video analyzer, enableSegment: false treats the whole video as one segment; enableSegment: true with a contentCategories description (for example "one product reviewed in a vlog, excluding the sponsor segments") creates custom segments, and fields are then generated for each one. A video analyzer supports one content category.
• Frame sampling: video is sampled at roughly one frame per second and frames are resized to about 512 x 512 pixels, so fine detail and anything on screen for less than a second can be lost from the visual input, while the audio track is transcribed separately.

Try the analyzer in the Foundry portal first, then call it from code with the azure-ai-contentunderstanding package. In the new Foundry portal, Content Understanding lets you choose a prebuilt or custom analyzer, run it on a sample or uploaded file and inspect the fields and the JSON result, with no code, which is the quickest way to check that an analyzer returns what you expect.

from azure.ai.contentunderstanding import ContentUnderstandingClient
from azure.ai.contentunderstanding.models import AnalysisInput
from azure.core.credentials import AzureKeyCredential

client = ContentUnderstandingClient(
    endpoint="https://<resource>.services.ai.azure.com/",
    credential=AzureKeyCredential(key))

poller = client.begin_analyze(
    analyzer_id="prebuilt-receipt",
    inputs=[AnalysisInput(url=receipt_url)])
result = poller.result()

for content in result.contents:
    print(content.markdown)   # structured text: headings, tables
    print(content.fields)     # named values (with confidence when estimation is on)

• The client takes only the Foundry resource endpoint and a credential: an AzureKeyCredential for key authentication or a Microsoft Entra ID token credential such as DefaultAzureCredential() (the identity then needs the Cognitive Services User role on the resource). The analyzer ID and input files are per-request arguments of begin_analyze, and model deployments come from the resource defaults or a request's modelDeployments.
• The result: result.contents holds one item per content (or segment). Read markdown to chunk text for a vector index while keeping headings and tables; read fields for schema values such as a generated summary or an invoice total.

Analysis is a long-running operation. In the SDK, begin_analyze returns a poller and result() waits for the finished analysis. Over REST, a POST to /contentunderstanding/analyzers/{analyzerId}:analyze returns at once with no fields and an Operation-Location header; the app sends GET requests to that URL until the status is Succeeded, then reads the result.