Generative AI Concepts: Tokens, Embeddings, and Foundation Models (AIF-C01)

Generative AI is artificial intelligence that generates new content instead of only sorting or scoring existing data. Give it a prompt and it produces an original response: a paragraph, a summary, an image, a snippet of code, or a spoken reply. This ability to create is what separates it from traditional machine learning, which typically predicts a label or a number.

The engine behind modern GenAI is the foundation model (FM). A foundation model is a large, general-purpose model pre-trained on a very broad dataset so that a single model can be adapted to many different tasks, such as answering questions, translating, and drafting. Because the FM already knows a great deal about language or images from pre-training, you can often use it directly or adapt it lightly, rather than training a new model from scratch for every job.

A large language model (LLM) is a foundation model specialized for text. It has been trained on enormous amounts of written material so it can predict and generate language fluently. For Task 2.1 you only need recognition-level understanding: know that GenAI creates content, that a foundation model is the reusable, broadly trained base, and that an LLM is the text-focused kind of foundation model. The deeper mechanics of customizing and prompting these models belong to Domain 3, not here.

Before a language model can work with your text, it breaks that text into tokens. A token is a small piece of text, often a word, part of a word, or a punctuation mark. As a rough guide, one token is about four characters of English, so a short sentence might be a dozen tokens. Models do not read letters or whole documents directly; they read sequences of tokens, and everything the model does, from reading your prompt to writing its answer, is measured in tokens.

Two facts matter for the exam. First, every model has a maximum number of tokens it can consider at once, called its context window, which limits how much text you can send plus how much it can generate in one call. Second, because tokens are the unit of work, they are also the unit of billing, which this lesson covers later.

Chunking is a related but separate idea. It means splitting a large document into smaller, manageable pieces before feeding it to a model or storing it for later retrieval. If you have a 200-page manual, you cannot fit it all into one context window, so you divide it into chunks, for example a few paragraphs each. Chunking makes long content searchable and lets an application pull only the most relevant pieces into a prompt. Keep the pair straight: tokenizing splits text into the tiny units a model reads, while chunking splits a large document into sections an application can manage.

Computers cannot compare meaning directly, so GenAI applications convert text or images into numbers that capture meaning. An embedding is a numerical representation of a piece of content, such as a word, sentence, or document, produced by a model so that items with similar meaning end up with similar numbers. An embedding is expressed as a vector: an ordered list of numbers, such as [0.12, -0.44, 0.98, and so on], often with hundreds or thousands of values.

The useful property is that distance in this number space reflects similarity in meaning. The embeddings for "car" and "automobile" sit close together, while "car" and "banana" sit far apart. This is what powers semantic search and recommendations: instead of matching exact keywords, an application embeds the user's query, then finds stored content whose vectors are nearest to it. Those vectors are typically kept in a vector database built to store embeddings and find the closest matches quickly.

Consider a support assistant. Each help article is chunked, and each chunk is turned into an embedding and stored. When a customer asks a question, the question is embedded too, and the system retrieves the chunks whose vectors are closest, meaning the passages most similar in meaning, to ground the answer. For Task 2.1, remember the chain in order: text is chunked, each chunk becomes an embedding, an embedding is a vector, and vectors are compared by closeness to find related meaning.

GenAI comes in several model types, and the exam expects you to recognize each by what it produces. A transformer is the neural-network architecture behind most modern language models; its key strength is an attention mechanism that lets the model weigh how much each token relates to the others, which is why transformer-based LLMs handle long, context-rich text so well. You do not need the internal math, just associate "transformer" with the architecture that powers today's LLMs.

A multi-modal model works with more than one type of data at once, for example accepting an image and a text question together, or generating an image from a text description. A modality means a kind of data (text, image, audio, video), so multi-modal simply means several kinds. A diffusion model is a model type used mainly for image generation. It learns to create an image by starting from random noise and gradually refining it into a clear picture that matches the prompt. When a question describes generating pictures from text, diffusion is the concept being tested.

The table below summarizes the foundational terms from this lesson so far. Treat these as definitions to recognize, not systems to build.

Task 2.1 asks you to recognize where generative AI applies. Because a foundation model can produce and transform content, its use cases span text, images, audio, video, and code. Common examples include:

• Content generation — drafting marketing copy, emails, or reports from a short prompt.
• Summarization — condensing long documents, meetings, or threads into a short brief.
• Image, video, and audio generation — creating pictures from descriptions, generating short videos, or synthesizing speech.
• AI assistants and customer-service agents — conversational helpers that answer questions and handle routine requests in natural language.
• Translation — converting text between languages while preserving meaning.
• Code generation — writing, explaining, or fixing code from a plain-language request.
• Search and recommendation — semantic search that matches meaning rather than keywords, and recommendation engines that suggest relevant items.

A concrete example ties several together. An online retailer builds a customer-service assistant: it summarizes a shopper's past orders, answers questions in a natural conversation, translates replies for international buyers, and suggests related products through recommendations. One foundation model, adapted through prompts and context, covers all of these jobs. The exam pattern is to describe a business need, such as "condense support tickets," "draft product descriptions," or "turn text into a picture," and ask which GenAI capability fits. Map the verb in the scenario (summarize, generate, translate, recommend) to the matching use case and the answer follows.

The foundation model lifecycle describes the stages a model moves through from idea to ongoing operation. The exam may ask you to place these stages in order or match a description to a stage, so learn the sequence.

Two points are worth remembering. Pre-training is where the model gains its broad general knowledge, and it is expensive and slow because it uses huge datasets and heavy compute; this is why most teams use an existing pre-trained foundation model rather than building one. Fine-tuning comes after pre-training and adapts that general model to a specific domain or task using far less data. The feedback stage makes the lifecycle a loop rather than a straight line, since results from real use feed back into evaluation and further tuning. The deeper mechanics of fine-tuning belong to Domain 3; here you only need the ordered stages and what each one does.

Most foundation model services charge by the token, not by the request. You pay for the input tokens (everything in your prompt, including any context you attach) plus the output tokens (everything the model generates in reply). Because both sides are counted, a longer prompt or a longer answer costs more, and a short question with a short answer costs less. This is the essence of the token-based pricing model: cost scales with the amount of text in and out.

This has direct effects on cost and performance. Sending a large document as context on every call multiplies input tokens and raises the bill; asking for a verbose answer raises output tokens. Longer inputs can also add latency, because the model has more tokens to process. Teams therefore trim prompts, summarize context, and cap output length to control spend. A quick example: putting a 20-page policy into every prompt is far more expensive than retrieving only the two most relevant paragraphs and sending those.

That trade-off leads to context engineering: the practice of giving the model the right information, in the right amount, at the right time so it produces accurate, relevant answers. A foundation model only knows what it learned in training plus what you put in the prompt, so supplying good context, such as relevant documents, examples, and instructions, is often what makes an application work. Context engineering balances usefulness against token cost: enough context to answer well, but not so much that it wastes tokens or overflows the context window.

An AI agent is a system that uses a foundation model as its reasoning engine and can take actions to accomplish a goal, not just return text. Task 2.1 asks only for foundational definitions of the agentic building blocks; the design details come later.

• Tool usage — an agent can call external tools and functions, such as looking up a record, running a calculation, or searching a database, to get information or perform an action.
• Memory management — an agent keeps track of relevant information across steps or conversations, so it can remember earlier context instead of starting fresh each time.
• Workflow orchestration — coordinating a sequence of steps, deciding what to do next, and chaining actions together to complete a multi-step task.
• Multi-agent systems — a pattern where several specialized agents work together on a complex problem, each handling part of it.
• Multi-agent communication — the patterns those agents use to share information and coordinate with one another.

The Model Context Protocol (MCP) is a standard way to connect agents to external systems and data sources. Rather than building a custom integration for every tool, MCP gives agents a common interface for reaching outside information and services; its role is connecting agents to external systems. A simple example: a travel-planning agent uses tools to check a calendar and search flights, remembers your stated budget through memory, and orchestrates the booking steps in order. For the exam, recognize each term's meaning and know that MCP is the protocol that links agents to external systems.