AI-103: Extract Content from Documents with Azure Content Understanding

For most AI-103 document scenarios the answer is Azure Content Understanding in Foundry Tools, with Azure Document Intelligence kept for highly structured forms you can label and for anything that must run on premises. Both are managed services that run OCR, layout analysis and field extraction for you and return per-field confidence and grounding; building your own extraction by prompting a model deployment in Microsoft Foundry gives you neither of those without custom work.

A model that writes its own "confidence: 0.9" or page number into JSON is not a substitute: a self-reported number isn't calibrated, and a page number the model types isn't verified grounding. When a scenario mixes requirements, split it: each workload goes to the tool its deciding constraint points at.

This lesson covers document extraction (AI-103 task 5.2). Indexing that output into Azure AI Search, enrichment skills and hybrid or vector retrieval belong to the sibling task Build retrieval and grounding pipelines; image, audio and video analyzers and how analyzers get their model deployments are covered in the multimodal understanding lesson.

Content Understanding has three content extraction analyzers that turn a file into text without any field schema, and you pick by what the file is and how much structure you need: prebuilt-read for text, prebuilt-layout for text plus structure, and prebuilt-digitalParse for machine-readable files. None of them needs a generative or embedding model, so they work on a Foundry resource with no model deployed.

A scanned page is an image, so prebuilt-digitalParse can't read it; Microsoft points scans to read or layout. The preview API extends metadata output to read and layout, but on the GA version digitalParse is the one that returns it.

Cost follows the work done

Content extraction is billed on meters that reflect the processing actually performed: born-digital files such as DOCX are charged at the minimal rate whichever analyzer processes them, OCR-only reading of scanned pages uses the basic meter, and layout analysis uses the more expensive standard meter. RAG analyzers and custom fields add contextualization and generative-model token charges on top. So the cheapest analyzer that meets the requirement wins: read when structure is never used, layout when it is, and a RAG or custom analyzer only when you need what its generative features add.

A document analyzer's config switches individual extraction steps on and off, and the defaults split cleanly: the extraction options are on by default and the generative figure options are off. A custom analyzer based on prebuilt-document inherits these defaults and overrides only what its config sets, so an empty config still runs OCR, layout, formula and barcode detection.

Two habits follow. First, switch off what the content doesn't need: OCR on born-digital files and formula detection on documents without equations add work without adding output. Second, keep on what the output depends on: layout for tables and headings, barcode detection for an ID printed as a barcode, formulas for papers whose equations must come through as LaTeX. Each kind of output traces back to its own switch, so when one kind goes missing, check the option that produces it.

Microsoft documents figure description and chart and diagram analysis as supported only for PDF and image files, so don't expect figure output from other formats.

returnDetails is a different kind of switch: it controls whether the response includes detailed information such as confidence scores, bounding boxes, text spans and metadata, rather than changing what is extracted or how the markdown is written. Per-field confidence and grounding are turned on by their own settings, estimateFieldSourceAndConfidence and estimateSourceAndConfidence (see below).

Every document analysis returns the content as GitHub Flavored Markdown that keeps the document's structure in forms a language model reads well, and anything markdown can't express natively is written as HTML. Knowing the encoding lets you write agent instructions and chunkers that rely on it.

Page metadata lives in comments

Markdown has no concept of a page, so page metadata is carried in HTML comments. A chunker that must cut at page boundaries splits on the page break comment, and one that must strip repeating running heads drops the page header and footer comments. Sections are headings that follow the document's logical structure, not its pages.

HTML or markdown tables

Setting tableFormat to markdown produces pipe tables, which look simpler to chunk but can't express rowspan or colspan. Financial statements with two-level column headers or merged cells then flatten, and values end up under the wrong column. The default html format keeps that structure inside the markdown.

Figure description versus figure analysis

Without figure options, a drawing appears only as an image reference with its labels. enableFigureDescription adds a short natural-language description, which suits search and short summaries. enableFigureAnalysis goes further: it extracts the data behind charts in Chart.js form (or a table) and writes diagrams such as flowcharts in Mermaid syntax, so an agent can read the plotted values or follow a diagram's branches. Descriptions summarise and lose detail; analysis preserves values and branches. Use both when you need both.

prebuilt-documentSearch is the Content Understanding RAG analyzer Microsoft recommends for ingesting documents into retrieval workflows: in one call it returns layout-preserving markdown, generated figure descriptions, chart and diagram analysis (Chart.js and Mermaid), handwritten annotations, a one-paragraph summary of the whole document and chunk-ready output, with no schema to write. It handles PDFs, Office files and scanned images, although figure descriptions and chart and diagram analysis are supported only for PDF and image files.

Because it uses generative models, it needs model deployments on the resource and costs more than plain layout extraction. Choose it when the knowledge base needs figures and annotations as text and a document summary; choose prebuilt-layout when structure alone is enough and no model is deployed. As a prebuilt, it needs no analyzer definition of your own.

Adding your own fields to RAG-style output

A custom analyzer can derive only from the four base analyzers: prebuilt-document, prebuilt-image, prebuilt-audio and prebuilt-video. A definition whose baseAnalyzerId is prebuilt-documentSearch or prebuilt-layout fails to create. To get documentSearch-style output plus your own fields, base the analyzer on prebuilt-document, add the fieldSchema, and turn on enableFigureDescription and enableFigureAnalysis, which that base leaves off by default.

Structured output comes from a custom analyzer's fieldSchema: each field has a name, a type, a description that acts as the prompt, and a method that says how the value is produced. You create the analyzer once (a PUT of its definition) and every analysis returns those fields beside the markdown.

Decide field by field. A policy number printed on a claim form is extract; a document type that must be one of a fixed list (claim, invoice, correspondence) is classify (generate could word it any way); an end date that a contract never states but that follows from the start date and term is generate (extract has nothing to copy). Set the method explicitly when it matters: leaving it unset lets the service choose from the type and description. An extract field also requires source and confidence estimation for that field (its estimateSourceAndConfidence must be true); the sample below satisfies this with the analyzer-level estimateFieldSourceAndConfidence.

{
  "description": "Supplier invoices",
  "baseAnalyzerId": "prebuilt-document",
  "config": {
    "enableFormula": false,
    "estimateFieldSourceAndConfidence": true
  },
  "fieldSchema": {
    "fields": {
      "InvoiceNumber": {
        "type": "string",
        "method": "extract",
        "description": "Invoice identifier, labelled Invoice No or Ref, top right of the header"
      },
      "ExpenseCategory": {
        "type": "string",
        "method": "classify",
        "enum": ["Goods", "Services", "Travel"],
        "description": "Category of spend the invoice covers"
      },
      "DueDate": {
        "type": "date",
        "method": "generate",
        "description": "Payment due date; when not printed, invoice date plus the payment term"
      },
      "LineItems": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "Description": { "type": "string", "method": "extract" },
            "Quantity": { "type": "number", "method": "extract" },
            "UnitPrice": { "type": "number", "method": "extract" }
          }
        }
      }
    }
  }
}

Writing descriptions that extract well

Microsoft's best practices for field schemas are mostly about the description:

• Be detailed. Give the format and a location hint, as in Microsoft's invoice-date example: "typically found at the top right corner"; a one-word description gives the model too little to go on.
• List every alias a template might print; the same example adds "May be labeled as Invoice Date, Billing Date, or Issue Date".
• Use affirmative wording that says what the field is; "not the invoice number" style descriptions are less effective.
• Match the document's language. For Italian invoices, name and describe fields in Italian (Fornitore, not Vendor); a schema in another language can significantly reduce accuracy. (locales is an audio and video transcription setting, not a document one.)
• Use structured types for repeated data. Line items belong in an array whose items are objects with typed properties, never in a string field asked to hold "a JSON list"; the service then returns well-formed per-row values and normalizes typed values such as numbers.
• Keep nesting shallow, and don't ask the model to embed confidence in a value; confidence has its own setting.

Refining descriptions is the first fix for poor extraction, before labeled samples.

Content Understanding can return, for each field, a confidence score from 0 to 1 and its grounding (where in the document the value came from), so you can post confident values automatically and route doubtful ones to a person who can jump to the exact spot. It works for extract, classify and generate document fields alike, so an inferred value can still drive threshold-based routing.

Confidence and grounding add tokens, and Microsoft's cost guidance is to enable them selectively. When only a few fields drive review routing, set the field-level switch on those fields rather than the analyzer-level one (fields that use extract need it on regardless).

source versus spans

The two are the visual and logical locations of the same value. They answer different questions, so each feature reads the property that matches what it displays.

Patterns that look similar but don't work

Any certainty the model is asked to express through a field's value is a model opinion, not a calibrated per-field score, and it gives no location. Only the confidence and grounding settings produce both.

Labeled samples help when a zero-shot custom document analyzer has low accuracy or low confidence on a layout it handles poorly, after the field descriptions have already been refined. In Content Understanding Studio you add a representative document, correct or label its field values, and rebuild the analyzer so it uses the sample and generalizes the pattern to similar documents.

1. Refine descriptions first: aliases, location hints, language, types.
2. Add one labeled sample per distinct template or layout that underperforms. Several samples of one layout don't teach the others.
3. Rebuild the analyzer.
4. Measure on a fixed, representative test set before and after, so you can show results improved rather than merely changed. Keep the labeled samples out of that test set.

What labeling can't fix

• Generate-method fields. Training doesn't currently support fields that use generate, so a summary that leaves things out won't improve from samples; improve its description instead.
• OCR misreads. If OCR reads the letter l as the digit 1, labeling the right character doesn't change what the text recognizer reads.
• Non-document analyzers. Training applies to document analyzers only.

Workarounds elsewhere in the pipeline hide a weak layout rather than fix it; the fix belongs in the analyzer.

An analyzer's contentCategories classifies content into named categories, each with a description and, optionally, an analyzerId that processes content of that category; enableSegment decides whether the file is first split into segments. This is how one call handles a PDF that holds an invoice, a receipt and a delivery note one after another.

Classification always picks one of the defined categories, so content that fits none of them (a cover letter, a blank separator sheet) is forced into the closest match unless you add an "other" category with a description and no analyzerId. Category descriptions stay affirmative too: the classifier needs somewhere valid to put the page, not a list of exclusions.

Define categories by semantic document type (invoice, credit memo, bank statement), not one per supplier's visual template; layout differences are handled within an analyzer by its descriptions and labeled samples. Running every analyzer over the whole file instead of splitting mixes one document's values into another's fields.

On the GA API (2025-11-01), analysis is asynchronous: you start it against an analyzer, poll an operation until it succeeds, then read the result's contents array, where each content object carries markdown (the layout-preserving text) and fields (the schema values).

REST

1. POST {endpoint}/contentunderstanding/analyzers/{analyzerId}:analyze?api-version=2025-11-01 with the document URL in inputs in the JSON body.
2. The service answers 202 Accepted with no analysis result in the body and an Operation-Location header pointing at contentunderstanding/analyzerResults/{id}.
3. GET that URL until status moves from NotStarted or Running to Succeeded.
4. Read result.contents[0].markdown to chunk and embed, and result.contents[0].fields for structured values.

GET /analyzers/{analyzerId} is a different resource: it returns the analyzer's definition, not an analysis result.

Python SDK

The ContentUnderstandingClient wraps the same flow: begin_analyze returns a poller and poller.result() waits for the finished analysis.

from azure.ai.contentunderstanding import ContentUnderstandingClient
from azure.ai.contentunderstanding.models import AnalysisInput
from azure.identity import DefaultAzureCredential

client = ContentUnderstandingClient(
    endpoint=endpoint, credential=DefaultAzureCredential()
)
poller = client.begin_analyze(
    analyzer_id="invoice-analyzer",
    inputs=[AnalysisInput(url=doc_url)],
)
result = poller.result()

for content in result.contents:
    text_for_rag = content.markdown   # layout-preserving text
    values = content.fields           # schema values (custom analyzers)

Use markdown as the source for chunking and embedding; fields holds only what the analyzer's schema defines, pages holds per-page layout detail such as words and lines, and warnings holds diagnostics. Prefer keyless authentication (Microsoft Entra ID through DefaultAzureCredential) over resource keys.

Copy prebuilt analyzers for production

Prebuilt analyzer definitions can change between API versions. When output feeds a fixed downstream mapping, such as invoice fields into an ERP system, retrieve the prebuilt definition (GET the analyzer) and create your own analyzer from it (PUT under your own ID). Your copy keeps its schema and behaviour while you move to a newer API version for other features.