Foundation Model Design: Selection, RAG, and Customization (AIF-C01)

Choosing a foundation model means matching the model to the job, because no single model is best for every application. The selection criteria the exam names are summarized below.

Some criteria are hard limits, others are trade-offs. Modality, language support, and the context window are pass-or-fail: a text-only model cannot accept a photo, however cheap or fast it is, and a model whose context window is smaller than your document cannot read it in one request. Check those first and discard models that fail. Only then weigh the trade-offs among the models left: cost, latency, and size and complexity, where a smaller model is usually faster and cheaper per token and a larger one handles harder reasoning.

Keep each criterion to its own job. Price per token, latency, and parameter count tell you how much a request costs, how fast it returns, and how capable the model is, but only the context window decides whether a long document fits into a single request. A scenario usually stresses one constraint: fast responses for a chatbot point to latency, keeping spend low at high volume points to cost and a smaller model, users in several languages point to multi-lingual support, and image input points to modality.

The context window, the output cap, and prompt caching are three different things that are easy to confuse: the context window limits how much one request can hold, the maximum output tokens setting limits how long the reply can be, and prompt caching makes a repeated part of the prompt cheaper and faster to process.

The context window is a run-time limit

The context window is the total number of tokens a model can work with in one request: the prompt (instructions, conversation history, any retrieved or pasted documents) plus the output it generates must fit inside it. If the input alone exceeds the window, Amazon Bedrock rejects the request with a validation error saying the input is too long; nothing is generated. A common way to hit this is a chat application that resends the whole conversation history on every turn: each turn makes the prompt longer until requests start failing. The fixes are to trim or summarize older turns, to retrieve only the relevant passages instead of sending everything, or to choose a model with a larger window. A larger window does not shorten answers or reduce cost; it only raises the ceiling on what one request can carry.

The output cap truncates instead of failing

The maximum output tokens setting caps how many tokens the model generates. When a reply reaches the cap, generation simply stops, so a cap set too low returns answers cut off mid-sentence, while a lower cap is also how you bound both the length and the output-token cost of each reply. Compare the two symptoms: a request rejected as too long is a context-window problem on the input side; a reply that stops mid-sentence is an output-cap problem.

Prompt caching reuses a repeated prefix

Prompt caching stores the processed form of a prompt prefix that many requests share, such as a long system prompt, a set of examples, or a reference document, so later requests that start with the same prefix read it from the cache instead of processing it again. On Amazon Bedrock, cached input tokens are billed at a reduced rate and the model returns its first token sooner, so caching lowers input cost and latency for live requests while keeping the same model and the same prompt. Know its limits:

• It works only on the identical prefix. Prompts that share no common opening content gain little or nothing.
• It applies to input only. Output tokens are billed as normal, and on some models the first request that writes the cache costs more than a normal request.
• It changes nothing about the model. Weights, answer quality, and behaviour stay the same.
• It does not enlarge the context window. Cached tokens still count toward the window, so a document too big to fit is still too big when cached.
• It is supported only on some models, so check before relying on it.

That last pair of points is why caching is not a scaling answer for a growing document set. Caching makes a large, repeated prompt cheaper, and a larger-window model moves the limit further out, but in both cases every call still carries the whole document. Retrieval Augmented Generation (RAG), covered below, is what keeps prompts small as the content grows.

Inference parameters are per-request settings that steer how a model produces its answer without changing the model itself. Inference is the act of generating an answer from a prompt, and these settings adjust that generation. The exam focuses on a few key levers.

The most tested lever is temperature. To make output more consistent, such as for a support answer or data extraction, lower the temperature. To make output more creative and varied, such as for brainstorming marketing ideas, raise it. Top-p and top-k are alternative ways to control randomness. Temperature, top-p, and top-k change which words are chosen; they do not set or cap the length. They can shift length indirectly, because the end of the reply is itself a sampled token, but only the output cap puts a hard limit on length.

Repeatable is not the same as correct. A low temperature makes the model give the same answer each time, but if the model does not know a fact, such as your company's current return policy, it will repeat the same wrong answer consistently. Hallucination caused by missing knowledge is fixed by grounding the model in the right content (RAG), not by sampling settings. These settings change behavior per request; they do not retrain or permanently alter the model.

Retrieval Augmented Generation (RAG) is a technique that retrieves relevant information from an external knowledge source at query time and adds it to the prompt, so the model answers from current or proprietary data instead of only what it learned during training. The flow is simple: the user asks a question, the system searches a knowledge source for the most relevant passages, those passages are inserted into the prompt as context, and the model generates an answer grounded in them. The model's weights never change.

RAG matters because a foundation model's built-in knowledge is frozen at training time and does not include your company's private documents. Without RAG, a model asked about your internal policy either cannot answer or may hallucinate, meaning it produces a confident but wrong answer. Grounding the response in retrieved facts reduces hallucination and keeps answers current, and because the answer comes from specific passages, the application can cite the source documents it used.

Why not just paste everything into the prompt?

Putting all your source material directly into the prompt, called in-context learning or prompt stuffing, works for a few short documents and needs no extra infrastructure. It stops working as the collection grows, for two reasons: the content no longer fits in the model's context window, and every request pays for every token of it, even the parts irrelevant to the question. RAG sends only the few passages that match each question, which is what keeps prompts small as the document set grows into thousands of pages.

RAG trade-offs

• It adds tokens. Retrieved passages are extra input, so a RAG request costs more than the bare question; RAG saves money compared with stuffing everything into the prompt or with retraining, not compared with no context at all.
• Passages must still fit. The retrieved content plus the question and the answer must fit within the context window.
• It reduces hallucination but does not eliminate it. If retrieval returns the wrong passage, or none, the model can still answer badly.

Amazon Bedrock Knowledge Bases

On AWS, Amazon Bedrock Knowledge Bases is the managed RAG capability. You connect a data source, such as an Amazon S3 bucket, and it handles chunking, creating embeddings, storing them in a vector store, retrieving the relevant pieces at query time, and passing them to the model, so you do not build the retrieval pipeline yourself. A knowledge base does not watch the source for changes on its own: after documents are added, changed, or deleted, the data source must be synced, which runs an ingestion job that you start from the console or API (StartIngestionJob). Knowledge Bases has no built-in sync schedule; to automate syncing, build your own trigger, for example an Amazon EventBridge schedule or an S3 event that calls StartIngestionJob through AWS Lambda. Syncing re-indexes content; it is not training, and the model is unchanged.

Do not confuse Knowledge Bases with Amazon Bedrock Guardrails. Guardrails' contextual grounding check evaluates whether a response is supported by the source content supplied with the request and relevant to the query, and can block one that is not. It checks answers; it does not find or retrieve the content. Retrieval is the knowledge base's job.

Business applications are broad: internal knowledge assistants that answer from company wikis, customer-support bots grounded in product manuals, and research tools that cite source documents.

A vector database stores embeddings and finds the ones closest in meaning to a query, which is how RAG retrieves passages by meaning rather than exact keywords. Each chunk of your documents is converted into an embedding, a numerical representation of meaning expressed as a vector. When a user asks a question, the question is embedded too, and the database returns the stored chunks whose vectors are closest, meaning the passages most similar in meaning.

The AWS services the exam names for storing embeddings are:

Being a data service is not enough. Amazon Athena (SQL queries over S3), Amazon Kinesis (streaming), the AWS Glue Data Catalog (metadata), and AWS Lake Formation (data-lake permissions) store, move, or govern data but do not perform vector similarity search. You do not need to configure any of these services for the exam; the point is recognizing which ones can hold embeddings for similarity search. Amazon Bedrock Knowledge Bases can use such a store behind the scenes as its vector database.

Customization approaches range from cheap and quick to expensive and involved, and the right choice is the lightest one that meets the need. Learn the ladder.

Model distillation sits in the table for comparison, but it is not a step on the adaptation ladder. The ladder ranks ways to change what a model knows or how it behaves; distillation is a cost-reduction lever that makes an already adequate task cheaper and faster to run, so its cost is not ranked against RAG or fine-tuning.

Start at the top and move down only if you must. Fine-tuning teaches a task or style and needs labeled examples. Continuous pre-training on unlabeled text teaches domain vocabulary and language, not a specific task. Neither keeps answers current as documents change; that is RAG's job.

Model distillation: teacher and student

Model distillation transfers a large model's skill on a specific task into a smaller, cheaper, faster model. The large model is the teacher; the small model is the student, which is trained on the teacher's responses until it handles that task nearly as well. Because the teacher writes the responses, you do not need a labeled dataset: Amazon Bedrock Model Distillation accepts your prompts, or your existing invocation logs, generates the teacher's responses, and fine-tunes the student. The result is lower inference cost and latency for that task, at the price of some quality and a training step, and the application then calls a different, smaller model.

Cutting running cost without customizing

Some cost levers keep the same model and the same prompt. Compare them by what they cost you:

Batch inference is priced lower than on-demand inference, but because it is asynchronous it suits work nobody is waiting for and makes interactive response time worse. Prompt caching is the lever that lowers both cost and latency for live requests, but only when there is a repeated prefix.

An AI agent is an application that uses a foundation model as its reasoning engine and can take actions to complete a goal, not just return text. An agent combines three ingredients: the FM for reasoning, tools it can call (such as looking up an order, querying a database, or triggering an API), and orchestration that plans the steps, reads each result, and decides what to do next. This lets an agent interpret a free-form request, break it into steps, call the right tool at each step, and adapt when a result changes the plan.

In Amazon Bedrock Agents, the tools are defined as action groups: each action group describes the API operations the agent may call (with an API schema or function details) and the AWS Lambda function or application code that carries them out. An agent can also be given a knowledge base so it can answer from documents as well as act. Tools can also be connected through open standards such as the Model Context Protocol (MCP), which defines a common way for agents to discover and call tools.

Consider a customer-service example. A shopper asks an agent to change a delivery address. The agent interprets the request, calls a tool to look up the order, calls another tool to update the address, confirms the change, and replies in natural language. A RAG chatbot could only explain how to change the address; the agent actually does it.

Agent or predefined workflow?

Business applications of agents include automating customer-service tasks, handling booking, order, or IT requests that span several systems, retrieving and combining information from those systems, and assisting employees with routine operations. The defining idea is that an agent is a foundation model plus tools plus orchestration that takes actions. Deeper agentic-AI design belongs to other tasks; here you need the role and the business value.

Task 3.1 questions usually describe a need and ask which design choice fits, and the method is to find the scenario's constraint and pick the lightest approach that satisfies it. Suppose a company wants an assistant that answers employee questions from its constantly changing internal HR documents, gives consistent factual replies, keeps cost low, and needs no data-science team.

Work through the decisions. Because the answers must come from current, private documents that are far too large to paste into every prompt, the right technique is RAG, not fine-tuning, since RAG adds your data without retraining and stays up to date as documents change. On AWS, Amazon Bedrock Knowledge Bases provides this as a managed service, storing the document embeddings in a vector database such as Amazon OpenSearch Service; after HR updates a document, the data source is synced so the change becomes retrievable. To keep replies consistent, set a low temperature, remembering that grounding, not temperature, is what keeps them correct. For model selection, check that the model handles the needed languages and fits the retrieved passages in its context window, then favor a smaller, lower-cost, low-latency model with adequate quality. If every request starts with the same long system prompt, prompt caching cuts its cost and latency. If the assistant also needs to take actions, such as submitting a leave request, add an agent with the right action groups.

Notice the order of preference throughout: reach for the cheapest, least invasive option that meets the need. Prompt engineering and RAG come before fine-tuning; fine-tuning comes before continuous pre-training; and pre-training from scratch is almost never the answer for a business application.