Generative AI Concepts: Tokens, Embeddings, and Foundation Models (AIF-C01)

Generative AI is artificial intelligence that generates new content instead of only sorting or scoring existing data. Give it a prompt and it produces an original response: a paragraph, a summary, an image, a snippet of code, or a spoken reply. This ability to create is what separates it from traditional machine learning, which typically predicts a label or a number.

The engine behind modern GenAI is the foundation model (FM). A foundation model is a large, general-purpose model pre-trained on a very broad dataset so that a single model can be adapted to many different tasks, such as answering questions, translating, and drafting. Because the FM already knows a great deal about language or images from pre-training, you can often use it directly or adapt it lightly, rather than training a new model from scratch for every job.

A large language model (LLM) is a foundation model specialized for text. It has been trained on enormous amounts of written material so it can predict and generate language fluently. For Task 2.1 you only need recognition-level understanding: know that GenAI creates content, that a foundation model is the reusable, broadly trained base, and that an LLM is the text-focused kind of foundation model. The deeper mechanics of customizing and prompting these models belong to Domain 3, not here.

Before a language model can work with your text, it breaks that text into tokens. A token is a small piece of text, often a word, part of a word, or a punctuation mark. As a rough guide, one token is about four characters of English, so a short sentence might be a dozen tokens. Models do not read letters or whole documents directly; they read sequences of tokens, and everything the model does, from reading your prompt to writing its answer, is measured in tokens.

Two facts matter for the exam. First, every model has a maximum number of tokens it can consider at once, called its context window, which limits how much text you can send plus how much it can generate in one call. Second, because tokens are the unit of work, they are also the unit of billing, which this lesson covers later.

The context window explains a common failure in chat applications. A model does not remember earlier turns by itself, so a chat app usually resends the whole conversation history with every new message. Each turn therefore sends more input tokens than the last, even when the newest message is short, until the total exceeds the context window and the request is rejected as too long. That is a size limit on one request, not throttling: throttling rejects requests because too many arrive too quickly, regardless of their length. Apps avoid the overflow by trimming or summarizing older turns, which is a form of context engineering.

Chunking is a related but separate idea. It means splitting a large document into smaller, manageable pieces before feeding it to a model or storing it for later retrieval. If you have a 200-page manual, you cannot fit it all into one context window, so you divide it into chunks, for example a few paragraphs each. Chunking makes long content searchable and lets an application pull only the most relevant pieces into a prompt. It also improves retrieval precision: when each chunk is embedded separately, its vector represents one focused topic, so it can match a specific question closely, whereas a single vector for a whole document blends every topic in it and matches nothing sharply. Sending only the matching chunks, instead of the whole document, also keeps input tokens down. Keep the pair straight: tokenizing splits text into the tiny units a model reads, while chunking splits a large document into sections an application can manage.

Computers cannot compare meaning directly, so GenAI applications convert text or images into numbers that capture meaning. An embedding is a numerical representation of a piece of content, such as a word, sentence, or document, produced by a model so that items with similar meaning end up with similar numbers. An embedding is expressed as a vector: an ordered list of numbers, such as [0.12, -0.44, 0.98, and so on], often with hundreds or thousands of values. An embedding model always produces vectors of the same fixed length (its dimension), whether the input is one word or a full paragraph, so the numbers are not one-per-word and the order of the original tokens is not stored.

The useful property is that distance in this number space reflects similarity in meaning. The embeddings for "car" and "automobile" sit close together, while "car" and "banana" sit far apart. Similarity is judged by how close two whole vectors are, measured as a distance or as the angle between them (cosine similarity is a common measure), not by whether the vectors share identical numbers; two vectors for similar meanings will rarely contain any exactly matching values. This is what powers semantic search and recommendations: instead of matching exact keywords, an application embeds the user's query, then finds stored content whose vectors are nearest to it. Those vectors are typically kept in a vector database built to store embeddings and find the closest matches quickly.

Consider a support assistant. Each help article is chunked, and each chunk is turned into an embedding and stored. When a customer asks a question, the question is embedded too, and the system retrieves the chunks whose vectors are closest, meaning the passages most similar in meaning, to ground the answer. Storing a new chunk in the vector database does not retrain or change the foundation model; it only adds something the application can retrieve. For Task 2.1, remember the chain in order: text is chunked, each chunk becomes an embedding, an embedding is a vector, and vectors are compared by closeness to find related meaning.

GenAI comes in several model types, and the exam expects you to recognize each by what it produces. A transformer is the neural-network architecture behind most modern language models; its key strength is an attention mechanism that scores how much each token should influence the others, which is why transformer-based LLMs handle long, context-rich text so well. Attention is what lets a model tell, for example, that "bank" in a sentence about a river means a riverbank rather than a lender, because the surrounding words shape its meaning. Because attention looks at all tokens together, a transformer also adds positional encoding, which tells the model each token's position so word order is not lost. Keep the two apart: positional encoding supplies order; attention supplies context, scoring which tokens matter to each other. You do not need the internal math, just associate "transformer" with the architecture that powers today's LLMs.

A multi-modal model works with more than one type of data at once, for example accepting an image and a text question together, or generating an image from a text description. A modality means a kind of data (text, image, audio, video), so multi-modal simply means several kinds. A diffusion model is a model type used mainly for image generation. It learns to create an image by starting from random noise and gradually refining it into a clear picture that matches the prompt. When a question describes generating pictures from text, diffusion is the concept being tested.

The table below summarizes the foundational terms from this lesson so far. Treat these as definitions to recognize, not systems to build.

Prompt engineering is the practice of shaping the input to a foundation model, its instructions, context, and examples, so that the model produces the output you want, without changing the model's weights. Rewording an instruction, stating the required format and tone, or including one sample of a good reply in every request are all prompt engineering. The model itself is untouched; only what you send it changes.

That makes prompt engineering the first and cheapest lever: it needs no labelled training data, no training job, and takes effect on the very next request. Changing the model itself, by fine-tuning on labelled examples, is slower and costlier and is used when prompting alone cannot reach the required behaviour. The specific techniques, such as zero-shot, few-shot, and chain-of-thought prompting, prompt templates, and prompt risks, belong to the Task 3.2 lesson.

Do not confuse prompts with inference parameters, the settings sent alongside a prompt. Temperature controls how random the output is (lower is more consistent, higher is more varied), and the maximum output tokens setting caps how long the reply can be. Neither one adds knowledge the model lacks, and neither sets a layout or a tone; those come from the prompt. Inference parameters are covered in the Task 3.1 lesson.

Task 2.1 asks you to recognize where generative AI applies. Because a foundation model can produce and transform content, its use cases span text, images, audio, video, and code. Common examples include:

• Content generation — drafting marketing copy, emails, or reports from a short prompt.
• Summarization — condensing long documents, meetings, or threads into a short brief.
• Image, video, and audio generation — creating pictures from descriptions, generating short videos, or synthesizing speech.
• AI assistants and customer-service agents — conversational helpers that answer questions and handle routine requests in natural language.
• Translation — converting text between languages while preserving meaning.
• Code generation — writing, explaining, or fixing code from a plain-language request.
• Search and recommendation — semantic search that matches meaning rather than keywords, and recommendation engines that suggest relevant items.

A concrete example ties several together. An online retailer builds a customer-service assistant: it summarizes a shopper's past orders, answers questions in a natural conversation, translates replies for international buyers, and suggests related products through recommendations. One foundation model, adapted through prompts and context, covers all of these jobs. The exam pattern is to describe a business need, such as "condense support tickets," "draft product descriptions," or "turn text into a picture," and ask which GenAI capability fits. Map the verb in the scenario (summarize, generate, translate, recommend) to the matching use case and the answer follows.

Equally important is recognizing what is not a generative task. If the output is a label, a number, or a grouping of existing records rather than new content, the job is traditional (predictive) machine learning, even when it is useful and data-driven.

Recommendation and search appear on both sides: GenAI adds semantic matching with embeddings and natural-language explanations, while classic recommenders are predictive models. The Task 1.2 and Task 2.2 lessons cover when a traditional model is the better choice.

The foundation model lifecycle describes the stages a model moves through from idea to ongoing operation. The exam may ask you to place these stages in order or match a description to a stage, so learn the sequence.

Two points are worth remembering. Pre-training is where the model gains its broad general knowledge, and it is expensive and slow because it uses huge datasets and heavy compute; this is why most teams use an existing pre-trained foundation model rather than building one. Fine-tuning comes after pre-training and adapts that general model to a specific domain or task using far less data. The feedback stage makes the lifecycle a loop rather than a straight line, since results from real use feed back into evaluation and further tuning. Notice which stages change the model and which only measure it. Evaluation and feedback tell you how good the model is and where it falls short, but they do not alter a single weight; running more evaluation on a weak model produces the same weak model with better documentation. To close a quality gap that feedback reveals, you loop back to a stage that changes the model. For a gap in one narrow domain, such as specialist terminology, that is usually fine-tuning on curated examples from that domain, which is far cheaper than repeating pre-training; afterwards you evaluate again before redeploying. (A gap caused by missing or outdated facts is often closed without changing the model at all, by supplying the facts as context, covered below.) The deeper mechanics of fine-tuning belong to Domain 3; here you only need the ordered stages and what each one does.

Most foundation model services billed on demand charge by the token, not by the request. You pay for the input tokens (everything in your prompt, including any context or conversation history you attach) plus the output tokens (everything the model generates in reply). Because both sides are counted, a longer prompt or a longer answer costs more, and a short question with a short answer costs less. This is the essence of the token-based pricing model: cost scales with the amount of text in and out.

Input and output tokens are usually priced at different rates, and output tokens typically cost several times more per token than input tokens. That does not mean trimming the output always saves most. The saving from any change is the number of tokens it removes multiplied by the rate for that side. With illustrative rates of $0.002 per 1,000 input tokens and $0.010 per 1,000 output tokens, a request that sends 5,000 input tokens and receives 300 output tokens costs $0.010 for input and $0.003 for output. Halving the answer saves $0.0015, while cutting 2,000 tokens of attached context saves $0.004, because input dominates the volume. Always compare tokens removed times rate, not the rates alone.

Token-based pricing is not the only billing model, and the alternatives are easy to confuse with it:

The cost trade-offs between on-demand and Provisioned Throughput on AWS are covered in the Task 2.3 lesson; for Task 2.1, recognize that a bill that grows with the length of the prompt and the answer is token-based pricing.

Token volume also affects performance. The model has to process every input token and generate every output token, so sending a large document as context on every call raises cost and can add latency, and a long answer takes longer to generate. Shortening a prompt lowers cost and, if anything, makes the reply arrive sooner. Teams therefore trim prompts, summarize context, and cap output length to control spend. A quick example: putting a 20-page policy into every prompt is far more expensive than retrieving only the two most relevant paragraphs and sending those.

That trade-off leads to context engineering: the practice of giving the model the right information, in the right amount, at the right time so it produces accurate, relevant answers. A foundation model only knows what it learned in training plus what you put in the prompt, so supplying good context, such as relevant documents, examples, instructions, and recent conversation, is often what makes an application work. If a model keeps quoting a product price list that was revised after it was trained, the fix is to place the current price list in the context sent with each request; no retraining is needed, because the model answers from the context it is given.

Context engineering balances usefulness against token cost: enough context to answer well, but not so much that it wastes tokens or overflows the context window. A model with a larger context window raises the limit on how much you can send, but it does not choose what is relevant; filling a bigger window with a whole handbook still costs more per request and can bury the passage that matters. Selecting the relevant pieces, typically by retrieving the closest chunks from a vector store, is what solves both the cost and the overflow. Retrieval-augmented generation (RAG), which automates this, is covered in the Task 3.1 lesson.

An AI agent is a system that uses a foundation model as its reasoning engine and can take actions to accomplish a goal, not just return text. Task 2.1 asks only for foundational definitions of the agentic building blocks; the AWS services for building agents are covered in the Task 2.3 lesson.

• Tool usage — an agent can call external tools and functions, such as looking up a record, running a calculation, or searching a database, to get information or perform an action.
• Memory management — an agent keeps track of relevant information within a task or conversation (short-term memory) and, where designed to, across sessions (long-term memory), so it can remember earlier context such as a stated preference instead of starting fresh each time.
• Workflow orchestration — coordinating a sequence of steps, deciding what to do next, and chaining actions together to complete a multi-step task.

A simple example: a travel-planning agent uses tools to check a calendar and search flights, remembers your stated budget through memory, and orchestrates the booking steps in order. What makes it an agent rather than a scripted workflow is that the model decides which step and which tool come next.

A multi-agent system splits a complex task across several specialized agents, each with its own instructions and tools, instead of asking one agent to do everything. Specialization keeps each agent simpler and lets the system grow by adding agents.

The most common pattern is the supervisor (orchestrator) pattern: one coordinating agent receives the request, breaks it into parts, decides at run time which specialist sub-agents to call, and combines their results into one answer. Amazon Bedrock's multi-agent collaboration follows this supervisor-and-collaborators model. A sequential pipeline is the alternative for fixed work: every item goes through the same agents in the same order, each passing its output to the next, and no agent chooses the route.

How the agents exchange messages is the communication pattern:

The deciding question is usually whether the route is fixed or depends on the request. If every request follows the same path, a sequential pipeline is simplest. If the path depends on what each request asks for, a supervisor that routes at run time fits better.

The Model Context Protocol (MCP) is an open standard for connecting AI agents and applications to external tools, data sources, and systems. It uses a client-server model: an MCP server exposes a tool or data source (for example, a ticketing system, a CRM, or a database) through the protocol, and any MCP client, such as an agent or an AI assistant, can discover and use what that server offers.

Its value is removing one-off integrations. Without a standard, every agent needs its own custom code for every system, so four agents and five systems mean twenty integrations to maintain. With MCP, each system is exposed once through an MCP server, and every MCP-capable agent connects to it as a client.

It also helps to know what MCP is not:

• It is not tied to one model provider or to the programming language of the agent; it is a vendor-neutral, openly published protocol.
• It connects an agent to tools and data. Agent-to-agent messaging, how agents in a multi-agent system talk to each other, is a separate concern with its own patterns and protocols.
• It reaches the live system, so an agent can read and update current records; data does not have to be copied into a vector store first, and the model is not retrained.

For the exam, recognize MCP as the standard way for agents to reach external systems, with servers exposing capabilities and agents connecting as clients.