Databricks GenAI Engineer: Document Extraction, Filtering and Chunking into Delta Tables

The right source documents are the authoritative, current documents written for the audience the application serves, and nothing that audience must never see. A RAG application can only be as accurate as its knowledge base: the generator answers from what retrieval returns, and retrieval can only return what was ingested.

Judge every candidate source on four questions:

• Is it authoritative? Official manuals, published policies and reference documentation beat forum threads, chat logs and support tickets, which are useful evidence of what people ask but are unofficial, noisy and often wrong.
• Is it current? Documentation for the release or policy year the assistant supports. Archived editions describe behaviour that may no longer apply and compete with the current text at retrieval time.
• Does it hold the knowledge the use case needs? An assistant about an organisation's own rules needs the organisation's own documents. General public material only repeats what a base model already knows, and product documentation for a tool explains the tool, not how your organisation has decided to use it.
• Is it appropriate for everyone who can query the index? Personal records (salary spreadsheets, case files, submitted claims, tickets containing personal data) answer no policy question and become retrievable by anyone once indexed.

Sensitive content: leave it out at ingestion

If an audience must never see a document, the reliable control is to keep it out of the index that audience queries. Controls applied later in the chain (instructions to the model, filters on queries, partial masking of identifiers) all leave the record retrievable and depend on every later step behaving; content that was never ingested cannot leak. A record that answers none of the application's questions also adds nothing to its accuracy, so excluding it costs nothing. Where different users genuinely need different documents, that is a design for per-user filtering or separate indexes, not a reason to put everything in one shared index.

Missing knowledge needs a new source, not retrieval tuning

When the answer to a question is not in any indexed chunk, the fix is to ingest the source that contains it. No retrieval setting can return text that was never ingested: smaller chunks, more results, hybrid search, a re-ranker or a larger embedding model only change how existing content is matched and ordered.

Before spending effort on retrieval tuning, find out which problem you have. Take the failing evaluation questions and check whether their expected facts exist anywhere in the chunks table (a simple text search over the chunk column is enough):

• The facts are absent → a coverage gap. Find the team or system that owns that knowledge (for example, warranty terms that live only in a partner portal the crawler never visits) and add it to the pipeline.
• The facts are present but not retrieved → a retrieval problem: chunking, embedding model, query formulation or ranking. Those are tuned and measured in the sibling lesson on retrieval evaluation, advanced chunking and re-ranking.

Re-running the set with more results or a different search mode does not answer the question cleanly: a poor result still leaves you unsure whether the knowledge exists. And answering from the base model's own knowledge instead is not a fix: it does not know your organisation's rules, so it invents plausible ones.

Choose the extraction package by the source format and what must survive extraction: plain text, tables, layout, or text that exists only as an image. The first question for any PDF is whether it has a text layer. Digitally generated PDFs do; scanned PDFs usually contain only page images, and every text-layer parser returns empty strings for them.

Third-party packages have to be installed on the cluster (or in the job environment). When policy forbids installing libraries, or the pipeline must be SQL-only, use the built-in ai_parse_document function described next.

ai_parse_document is a built-in Databricks AI function that turns the binary content of a document into structured layout elements, with nothing to install. It reads PDF, image (JPG/JPEG, PNG, TIFF), Word (DOC/DOCX) and PowerPoint (PPT/PPTX) files, including scanned pages, and you call it from SQL over files read from a Unity Catalog volume in binaryFile format:

SELECT path, ai_parse_document(content) AS parsed
FROM read_files('/Volumes/main/raw/docs', format => 'binaryFile');

The result is a VARIANT whose document.elements array lists each element with its type, content and a confidence value. Element types include text, title, section_header, table, figure, caption, footnote, page_header, page_footer, page_number and signature. Those types are what make it useful for chunking: headers give you section boundaries, tables arrive as tables, and running headers, footers and page numbers can be filtered out by type before you concatenate anything.

Know what it is not: ai_extract pulls named entities out of text you give it, ai_classify assigns a label to text and ai_summarize condenses text. None of them reads a document file or returns its layout. Reading binary files with format => 'text' just decodes bytes as lines.

Limits and options

The function runs on recent Databricks Runtime versions and serverless compute, and it has per-document limits: a maximum file size and a maximum page count. A document longer than the page limit (500 pages) is parsed by restricting each call to a range of pages with the pageRange option and processing the document in slices; limits are properties of the function, not of the compute you run it on. Its other options (for example, where to save rendered page images and which element types receive AI-generated descriptions) shape the output, not the limits. Check the function reference for the current values before relying on them.

Filtering means removing text that repeats, answers nothing or is out of date before you chunk, because it otherwise dominates chunk embeddings, matches queries that share its words and crowds genuine answers out of the top results. Clean before you chunk: once boilerplate is inside every chunk, no query-time setting removes it.

Workarounds that keep the noise do not fix it. A metadata filter narrows which documents are searched, but if every document carries the same footer the noise is still inside the filtered set. Larger chunks dilute boilerplate but keep it in every chunk. A re-ranker reorders the candidates retrieval found; it does not stop boilerplate pulling the wrong candidates in.

Do not strip what embeddings need

Dense embedding models are trained on natural sentences. Lowercasing, stripping punctuation or removing stop words (habits from old keyword search) removes meaning rather than noise and usually degrades retrieval. Do not drop tables either: they often hold exactly the figures users ask about.

Handling document versions

Superseded versions in the index produce stale answers, because the retriever returns whichever version scores highest and old and new wording are equally relevant to the question. How you fix it depends on whether anyone needs the history:

Things that do not solve it: merging versions into one file (position in a file does not decide which chunk scores highest), adding a version column without filtering on it, or adding a re-ranker (it scores relevance, not recency). A separate index per year can keep years apart, but every reissue then adds an index to create, sync and monitor plus routing logic to update, which is far more maintenance than one index with date columns.

A chunking strategy should follow the document's own structure where it has one, so each chunk covers one subject and can carry its section titles. Fixed character or token counts cut wherever the count runs out, which is how a chunk ends up holding the end of one feature's section and the start of the next.

Two near-misses: flattening HTML with BeautifulSoup get_text() before splitting throws the header structure away, and RecursiveCharacterTextSplitter.from_language(Language.HTML) prefers to break at tags but does not record which section a chunk belongs to. Header splitters are often followed by a size-based splitter so that a very long section still fits the embedding limit. Splitting where the meaning shifts (semantic chunking) and parent/child retrieval are covered with retrieval evaluation in the sibling lesson.

Tables

Fixed-size chunks cut tables in the middle, so rows arrive without their column headers and are misread. Keep each table usable:

• Large tables: split into row groups that fit the embedding limit and repeat the header row in every group. Keeping a large table whole truncates its later rows; more overlap repeats neighbouring rows, not the header at the top.
• Single-value lookups: write each row as self-describing text that names every column next to its value (for example Model: X200; Weight: 4 kg), so a question that names the attribute matches the row. Bare values in column order, or pipe-delimited rows, lack the attribute names.
• Never flatten a table into prose that loses the link between each value and its column, and never leave it out of the index.

Chunk size is bounded by two different model constraints: each chunk must fit the embedding model's input limit, and the retrieved chunks together must fit the generator's context window alongside the prompt and the answer.

The embedding limit

An embedding model reads at most a fixed number of tokens per input, and anything beyond that is silently truncated, not split into windows and combined. A 1,500-token chunk sent to a model with a 512-token limit is represented only by its first 512 tokens, so questions about the end of a section never retrieve it. More results, more overlap or a bigger generator cannot fix text that was never embedded; re-chunking under the limit can. For example, the Databricks-hosted databricks-bge-large-en has a 512-token window (and returns 1,024-dimensional vectors, which is the output size, not an input limit), while databricks-gte-large-en accepts 8,192 tokens.

Measure length in the embedding model's own tokens

LangChain splitters measure chunk_size with a length function, which by default is Python len: characters. Four characters per token is only an average, so character-sized chunks either fall far short of the limit or sometimes exceed it. A tokenizer-backed splitter is better, but only if it is the embedding model's tokenizer:

from transformers import AutoTokenizer
from langchain_text_splitters import RecursiveCharacterTextSplitter

tok = AutoTokenizer.from_pretrained("BAAI/bge-large-en-v1.5")
splitter = RecursiveCharacterTextSplitter.from_huggingface_tokenizer(
    tok, chunk_size=500, chunk_overlap=50
)

from_tiktoken_encoder(encoding_name="cl100k_base") counts OpenAI tokens; a different vocabulary gives different counts for the same text, so some chunks that are under the limit in tiktoken tokens exceed it in the model's own tokens. Overlap is taken from within chunk_size, not added on top.

The generator's context budget

Everything sent to the generator must fit its context window:

prompt template and history + (chunks retrieved × chunk size) + tokens reserved for the answer ≤ context window

Rearranged, the largest chunk size is (context window − prompt and history − answer reserve) ÷ chunks retrieved. For example, with a 16,384-token window, a 2,384-token prompt template and history, and 2,000 tokens reserved for the answer, 12,000 tokens remain for retrieved text: eight chunks can be at most 1,500 tokens each, while ten chunks must shrink to 1,200. If you must pass more chunks, they have to be smaller. A larger embedding window does not help (the error is the generator's), and raising max_tokens only reserves more of the window for the answer.

Small versus large chunks

A paragraph- or section-sized chunk of a few hundred tokens is the usual middle ground: one subject per chunk, with its qualifications kept together. Replacing chapters with summaries loses the step-level detail specific questions ask about.

Chunk overlap copies the end of each chunk into the start of the next, so a sentence or step that crosses a chunk boundary appears whole in at least one chunk. It is the direct fix when a retrieved chunk ends mid-sentence, for example with a list introduced in one chunk whose items fall in the next.

A modest overlap of roughly 10-20 percent of the chunk size is the usual setting. Too little (zero) splits boundary facts again; too much (half the chunk) makes neighbouring chunks mostly identical, so near-duplicates crowd the top results and the number of chunks to embed and store roughly doubles. Deduplicating results at query time hides the crowding but not the cost, and halving the chunk size multiplies the chunk count again. Overlap cannot fix truncation either: an over-long chunk is still cut at the embedding limit.

Chunks are written to a Unity Catalog Delta table with a three-level name (catalog.schema.table), one row per chunk with a unique id, because that table is what you can grant to a team and what a Delta Sync vector index uses as its source. (Databricks now calls the product Databricks AI Search, formerly Databricks Vector Search and earlier Mosaic AI Vector Search; the exam guide and SDK still say vector search.)

The sequence

1. Read the raw files from a Unity Catalog volume as binaryFile.
2. Parse them (a Python package or ai_parse_document) and filter extraneous content.
3. Chunk, explode to one row per chunk, and assign a unique chunk id; keep metadata columns beside the text.
4. Write to the Unity Catalog table, for example chunks_df.write.saveAsTable("main.rag.doc_chunks").
5. Enable change data feed: ALTER TABLE main.rag.doc_chunks SET TBLPROPERTIES (delta.enableChangeDataFeed = true) (or set it when creating the table).
6. Create the Delta Sync index on the table, naming the id as primary key and the text column as the embedding source.

A Delta Sync index on a standard endpoint needs both a unique primary key column and the source table's change data feed, which it reads to sync inserts, updates and deletes, so the index requires the feed before it can be created. (Tables with row tracking enabled supply a change feed automatically.) Precomputed embedding columns are needed only for self-managed embeddings, partitioning is not required, and _id is a reserved column name, so do not use it for your key. Change data feed is the table feature that publishes row-level changes; other table properties tune storage or writes and do not make an index possible.

What does not work as an index source

• A temporary or global temporary view: session- or application-scoped, not a governed table.
• Delta, Parquet or CSV files saved to a path (a volume or DBFS): files, not a registered table, so they cannot be granted or indexed. Register them with saveAsTable instead.

Metadata columns

Store the source path or URL, page number and business attributes (product line, document type, effective dates) as columns on each chunk row. Columns come back with search results, so the application can show a reliable citation link instead of asking the model to produce one, and they can be used as filters, letting one index serve several products. Prepending the source to the chunk text adds noise to the embedding and gives nothing to filter on; a lookup table joined after generation is too late to filter retrieval.

Process only new or changed files and append their chunks, so the Delta Sync index re-embeds only what changed. A nightly job that re-parses everything and writes with mode("overwrite") rewrites the whole table and forces the index to re-embed every chunk.

Auto Loader is the least-code way to pick up only new files: it records which files it has processed in its checkpoint.

(spark.readStream.format("cloudFiles")
    .option("cloudFiles.format", "binaryFile")
    .load("/Volumes/main/raw/docs")
    # parse, filter, chunk, add ids and metadata
    .writeStream
    .option("checkpointLocation", "/Volumes/main/rag/chk/doc_chunks")
    .trigger(availableNow=True)
    .toTable("main.rag.doc_chunks"))

• The checkpoint is the memory of what was processed, so its checkpointLocation must stay the same from run to run; a new or empty checkpoint means every file looks new again.
• trigger(availableNow=True) processes what is available and stops, which suits a scheduled job.
• Auto Loader's file tracking replaces custom bookkeeping you would otherwise write and maintain yourself.

Replacing a document that changed

When an edited version of an existing document arrives, appending its new chunks leaves the old ones in the table and the index, so search returns both. Delete the rows for that document (by its path or id), then append its new chunks (a replaceWhere overwrite scoped to that document achieves the same). The deletes flow through the change data feed, so the index drops the old vectors and embeds only the new rows. Table maintenance commands work on files, not on current rows, so they never remove an outdated document's chunks; only a delete (or a scoped overwrite) does that.

A chunking job needs read access to the source volume and write access to the target schema or table, on top of USE CATALOG and USE SCHEMA on the parents. Grant the least that does the job:

Run scheduled pipelines as a service principal holding exactly these grants, so access does not depend on any one person's account.