AI-103: Build Retrieval and Grounding Pipelines with Azure AI Search

An indexer is the pull-based way to ingest content: it connects to a data source (Blob Storage, ADLS Gen2, Azure SQL, Cosmos DB and others), cracks each file, optionally runs a skillset of enrichment skills, and writes the results to an index. The alternative is the push model — your own code (a Function, a Logic App, an app) calls the documents API — which works with any source but means writing and hosting that upload code. When the source is supported and the requirement is "no upload code", use an indexer.

Schedules and change detection

Give the indexer a schedule (for example hourly) and it re-runs on its own. For Azure Storage, change detection is built in: the indexer compares each blob's LastModified timestamp and processes only new or edited files. Database sources need a change detection policy, such as a high-water-mark column.

Deletion detection

A deleted file does not leave the index by itself. A deletion detection policy on the data source tells the indexer how to recognise deletions:

• Native blob soft delete — enable blob soft delete on the storage account; when the indexer next processes a blob that is in the soft-deleted state, it removes the matching search document. Nothing happens between runs, so set the soft delete retention period well above the indexer interval, or the blob may be purged before the indexer sees it. This policy isn't supported when blob versioning is enabled.
• Soft delete column — a metadata property (such as IsDeleted) that you set to a marker value before deleting the file. It is the alternative for sources that can't use native soft delete, and it relies on everyone flagging files first.

The policy must exist from the first indexer run. Files deleted before a policy existed leave orphaned documents that resetting the indexer does not clean up — a reset only reprocesses files that still exist. Microsoft's guidance in that case is a new index and indexer with the policy configured from the start.

Field mappings and output field mappings

An indexer has two kinds of mappings, and they run at different times:

A skill output placed in fieldMappings is never found, so the field silently stays empty. Field mappings can also apply mapping functions. The classic case is the document key: keys may contain only letters, digits, underscores, dashes and equal signs, so a raw blob URL is rejected. Mapping metadata_storage_path to the key with base64Encode gives a stable, unique, valid key per file. Pick the function by its output: a key needs one that emits only allowed characters, and an analyzer only affects how a searchable field is tokenized, never which characters a key may hold.

Text that exists only as pixels — a scanned TIFF, a photographed form, a picture inside a PDF — needs image extraction plus the OCR skill. Document cracking only pulls out text that is stored as text, so a scan produces an empty content field.

1. Extract the images. Set the indexer's imageAction configuration: none (the default) extracts no images; generateNormalizedImages extracts embedded images and image files; generateNormalizedImagePerPage renders each PDF page as an image (other file types behave like generateNormalizedImages). The results land in /document/normalized_images/*, each with metadata such as its contentOffset in the text.
2. Read the text. The OCR skill runs over /document/normalized_images/* and outputs text (printed and handwritten) and layoutText (the same text with bounding-box positions).
3. Merge it back. OCR text is stored per image. You can map it to its own field (keyword search then finds it), but chunking and embedding skills read /document/content. The Text Merge skill takes the content, the OCR text and each image's contentOffset and produces one merged string with the image text inserted where the image was. Chunk and embed that output.

Two near-twins to keep apart: the Image Analysis skill describes what a picture shows (tags, captions, objects) rather than reading its text, and parsingMode: text reads a blob as plain text — neither helps with a scan. dataToExtract controls whether cracking extracts content and metadata or metadata only. Output field mappings can't fix a merge problem either, because they run after every skill has finished.

For a new skillset that needs structure — headings, tables, figures, page numbers — use the Azure Content Understanding skill. It extracts text and images, outputs tables and figures as Markdown, treats a table that continues across pages as one unit, and chunks the content itself (chunks may span pages), so no separate Text Split skill is needed. Microsoft keeps the Document Layout skill for existing pipelines but recommends Content Understanding for new ones.

The skill's extractionOptions decide what comes back: images extracts figures (to display beside answers or to verbalize), and locationMetadata adds page information to each chunk, which is what lets an agent cite a page range. Leave out locationMetadata and there is nothing to cite.

Both layout skills analyse the original file, not cracked text. For blob sources the file reaches them through the /document/file_data node, which exists only when the indexer's configuration sets allowSkillsetToReadFileData to true — otherwise the skill reports its file_data input as missing. Content Understanding handles documents and images; it does not accept audio or video files (see the audio and video section below).

Index projections turn one source document into many search documents — one per chunk — which is what a RAG index normally needs. They are defined in the skillset: each projection has a selector naming the target index, the sourceContext to iterate (for example /document/pages/*), the parentKeyFieldName, and mappings from enriched nodes to chunk fields.

Rules that break pipelines when ignored

• Don't map the parent key. The field named by parentKeyFieldName is filled by the projection with a generated key the indexer uses to find, replace and delete a document's chunks when the source changes. Map every child field except the document key and the parent key; mapping the parent key yourself disrupts change tracking, so edited files leave their old chunks behind.
• Index schema. The document key must be an Edm.String field using the keyword analyzer; the parent field must be an Edm.String that is filterable and is not the key (many chunks share one parent, so it isn't unique). Nothing else is required — no sortable chunk field, no retrievable vector.
• projectionMode. The default, includeIndexingParentDocuments, also writes one document per source file whose chunk and vector fields are empty — a batch of 12 files split into 300 chunks becomes 312 documents, and the empty ones can reach the model as useless context. skipIndexingParentDocuments writes only the chunks, which Microsoft recommends when chunks are what you search.

parsingMode: jsonArray is unrelated: it splits JSON arrays that already exist in the source files, not chunks created by a skill.

The Azure OpenAI Embedding skill calls an embedding deployment during indexing and writes a vector for each input. Three settings decide whether it works:

• Dimensions must match the field. text-embedding-3-large supports 1–3,072 dimensions and text-embedding-3-small 1–1,536, chosen with the skill's dimensions property; omit it and you get the model's maximum. text-embedding-ada-002 is fixed at 1,536. The vector field must declare the same number, and an existing field's dimensions can't be changed — shrinking vectors means a new field (and matching mapping or projection).
• Input size. The skill accepts roughly 8,000 tokens of input whatever the model. Long documents must be chunked first (Text Split or a layout skill) and embedded in the chunk context, such as /document/pages/*, which also gives the one-vector-per-passage shape retrieval needs. The limit applies per input, so the fix is always smaller inputs, not a different setting on the skill or the indexer.
• Authentication. With an apiKey the skill uses the key. With both apiKey and authIdentity empty, it uses the search service's system-assigned managed identity. Set authIdentity to a user-assigned identity's resource ID to use that identity instead. Setting both a key and an identity is rejected. The identity needs Cognitive Services OpenAI User on the Azure OpenAI resource — the least-privilege role; a 403 usually means the identity actually in use doesn't hold it.

To embed queries the same way at search time, add a matching vectorizer to the vector profile (integrated vectorization). Choosing integrated vectorization and a chunking strategy is covered in the lesson on choosing Foundry services (task 1.1).

Built-in text skills add or transform text during indexing. The ones RAG pipelines use most:

Text Translation settings: defaultToLanguageCode (the target, such as en) is required. Leave fromLanguageCode and defaultFromLanguageCode unset and the skill uses the Translator's automatic language detection — the right setting when nobody knows each file's language. suggestedFrom is only a fallback when detection fails. The source setting describes the input and the target describes the output you want, so fix only the target and let detection handle the source.

Attaching a billable Foundry resource

Skills that call Foundry Tools (OCR, Entity Recognition, PII Detection, translation and the like) run free for only 20 documents per indexer per day. After that, enrichment stops partway through the day and the indexer logs errors; tolerating failures (maxFailedItems) only leaves documents unenriched. The fix is to attach a Foundry resource to the skillset:

• Keyless (AIServicesByIdentity) — the search service's managed identity calls the resource, which needs Cognitive Services User for that identity. No keys, and no same-region requirement.
• Key-based (AIServicesByKey, or the classic CognitiveServicesByKey, which calls a regional endpoint and expects the resource in the search service's region).

Use the Custom Web API skill when enrichment needs logic no built-in skill provides — your own normalization rules, a partner API, a proprietary model. It sends batches of enrichment-tree values to an HTTPS endpoint you host (often Azure Functions) and adds the returned outputs back to the tree, where later skills such as the embedding skill can read them. Other skills only look similar: the GenAI Prompt skill calls a deployed chat model, the Shaper skill restructures existing values, and the Conditional skill chooses between existing values.

The contract

The request body is {"values": [...]}; each record has a unique recordId and a data object. The response must be a values array with one entry per record, each carrying its recordId, a data object with the outputs, and errors and warnings (which may be null). Records are matched by recordId, not order — a record returned without its ID is invalid and is not enriched, even though the call returned HTTP 200. The skill expects a synchronous response; it doesn't poll.

Throughput settings

So an endpoint that needs a few seconds per record (say 2.5 seconds) times out at the defaults — a full batch would take far longer than 30 seconds. Size batchSize so that per-record time × records per batch fits under a raised timeout. httpMethod (POST or PUT) only changes the verb. For keyless calls, the skill can authenticate to a Microsoft Entra–protected endpoint with the search service's managed identity by setting authResourceId.

Images reach a RAG index by one of two paths, and the requirement decides which:

OCR or Image Analysis on a diagram keeps labels or tags but loses arrows and sequence. The general rule: a query image can only be compared with indexed content if a model that embeds images produced both vectors — any text-only stage in the path rules out searching by photo.

GenAI Prompt skill limits: it calls chat completions endpoints only (Responses API endpoints aren't supported), and its request timeout is fixed at 30 seconds with no property to raise. Reasoning models such as o3 can exceed that during bulk indexing, so use a fast multimodal non-reasoning model; moving the call into a Custom Web API skill would allow up to 230 seconds, but means hosting code.

Audio and video

Indexers and skills don't transcribe or segment media. Use Azure Content Understanding's RAG analyzers first, then chunk, embed and push the output into the index: prebuilt-videoSearch transcribes speech and splits a video into described segments with start and end times (keep them so answers can point to the segment), and prebuilt-audioSearch transcribes recordings and adds a short summary. prebuilt-documentSearch is the document equivalent, and prebuilt-callCenter is call analytics, not RAG content. Building custom analyzers and field extraction is covered in the Content Understanding lesson (task 5.2).

Configure grounding search with a searchable Collection(Edm.Single) vector field whose dimensions match the embeddings and whose vector profile names an algorithm, add filterable nonvector fields for any filter, and then layer hybrid (text plus vector) or semantic ranking on top at query time.

Vector fields and algorithms

• A vector field is Collection(Edm.Single) (or a narrower type), searchable, with dimensions and a vector profile naming an algorithm and, optionally, a vectorizer. HNSW is approximate and fast at scale; exhaustive KNN compares every vector — exact but costly, for small indexes or ground-truth checks. A query can also set exhaustive: true to brute-force an HNSW field. The algorithm never changes how many dimensions a field accepts.
• Vector fields can't be filterable. To restrict a vector query (one department, one tenant), add a filterable nonvector field and pass a filter. A scoring profile only reorders results; it doesn't exclude anything.
• stored and retrievable. stored: false removes the source copy of each vector to save storage. It can only be set when the index is created — a vector field added to an existing index can't set it to false — and it can't be reversed, and retrievable must stay false while it is. Without the stored copy, a merge or mergeOrUpload that omits the vector silently drops it, and the document stops matching vector queries. Either send the full vector with every partial update, or create and load a new vector field that keeps stored: true.

Filter modes

Hybrid and semantic

A hybrid query sends search text and a vectorQueries entry in one request; the service runs both and merges them with Reciprocal Rank Fusion, so exact terms (part numbers, codes) and meaning both count. Semantic ranking needs a semantic configuration on the index (a title field, content fields and keyword fields in priority order) and reranks the top results with queryType: semantic; it works with or without vectors. When to choose each method is taught in task 1.1, and writing the query in code in task 2.1.

An index or knowledge base becomes a grounding source for agents through a tool, and agents in turn are the steps a Foundry workflow orchestrates. A workflow therefore reaches the retrieval pipeline through the agents it calls, so configure grounding once on the agent's tool and every workflow that uses that agent inherits it. There are two connection patterns.

The Azure AI Search tool (one index)

• A project connection to the search service plus the tool definition with an index_name. Each tool targets exactly one index; a comma-separated list isn't supported.
• query_type is one of simple, semantic, vector, vector_simple_hybrid or vector_semantic_hybrid (the default). Vector and hybrid types need vector fields; for an index with only text fields and a semantic configuration, semantic is the best available ranking.
• Private search services: only a standard agent setup with virtual network injection reaches a search service with public access disabled (a basic setup can't). In that configuration the connection must use Microsoft Entra ID with the project managed identity — API keys aren't supported — and that identity needs Search Index Data Contributor and Search Service Contributor on the search service.

A Foundry IQ knowledge base over MCP (several sources)

• A knowledge base references one or more knowledge sources, including existing indexes, without copying content. It exposes an MCP endpoint (…/knowledgebases/<name>/mcp) whose retrieval tool is knowledge_base_retrieve, so one agent tool can ground answers in several indexes — and any MCP-capable client or workflow step can call it.
• The agent's MCP tool references a RemoteTool project connection with ProjectManagedIdentity authentication (no stored keys). CognitiveSearch connections are for the Azure AI Search tool, CustomKeys store secrets, and AzureOpenAI is for models.
• Roles: the project's managed identity needs Search Index Data Reader on the search service (Contributor only if it writes) — without it every call returns 403. Separately, if the knowledge base uses an LLM for query planning or answer synthesis, the search service's identity needs Cognitive Services User on the Foundry resource.
• Per-user results: the managed identity authorizes the call but says nothing about which user is asking, so permission-aware sources return unfiltered results. Add an x-ms-query-source-authorization header to the MCP tool whose value is a {{placeholder}} bound to a structured input, and have the app pass the signed-in user's Azure AI Search token in structured_inputs on every request. Permission trimming needs the user's own token at query time; anything fixed in the agent's configuration identifies the agent, not the person asking.

Filters, citations and file search inside agents are covered in the agents lesson (task 2.2).