A company wants to make a 200-page product manual searchable by a generative AI application. It first splits the manual into smaller sections of a few paragraphs each. What is this practice called?
Choose one.
Chunking means dividing a large document into smaller pieces so an application can store, search and retrieve the relevant parts.
A 200-page manual does not fit in one context window, so the application splits it into paragraph-sized sections and later pulls only the relevant ones into a prompt. That is chunking. Tokenizing also splits text, but into tiny word-level units that the model reads; fine-tuning further trains a model; and embedding converts each piece into numbers that capture its meaning, which usually happens after chunking.
- Notice what is being split: a whole document into sections of a few paragraphs.
- Match document-level splitting for retrieval to chunking.
- Separate it from tokenizing (word-level units), embedding (numbers for meaning) and fine-tuning (training).
Exam tip: Chunking splits a large document into sections an application can handle; tokenizing splits text into the units a model reads.
Generative AI Concepts: Tokens, Embeddings, and Foundation Models (AIF-C01) — the lesson that teaches this.