Training and Fine-Tuning Foundation Models

When you train a foundation model, you are adjusting the model's internal parameters (its weights) so it captures patterns in data. The exam asks you to recognize four key elements of training and where each sits on an effort-and-cost spectrum. You almost never pre-train your own model; instead you take a pre-trained FM from a provider and customize it. Knowing which element applies to a scenario is the skill being tested.

Read a question by matching the goal to the element: broad new knowledge points to pre-training, task-specific behavior points to fine-tuning, and a cheaper, faster model points to distillation. Each changes the model's weights, unlike prompt engineering, which you will contrast later in this lesson.

Pre-training is where a foundation model learns language, reasoning, and general knowledge from scratch. It consumes enormous, unlabeled datasets and large amounts of compute, so it is the most expensive and time-consuming step. Providers such as Amazon and its partners do this for you; on the job you consume the result rather than repeat it. Recognize pre-training in a question by phrases like "build broad knowledge from the ground up" or "train a general model on a massive corpus."

Continuous pre-training takes an already pre-trained model and keeps training it on additional, unlabeled data from your domain. You would use it when a model needs to absorb the vocabulary and concepts of a specialized field, for example medical or legal text, but you do not yet have labeled task examples. It deepens subject knowledge without starting over, so it costs less than full pre-training but more than fine-tuning.

The distinction the exam wants is simple: pre-training creates general capability from nothing, while continuous pre-training extends an existing model with fresh domain data. Both are unlabeled-data processes, which separates them from fine-tuning, where you supply labeled input-output examples that teach a specific task.

Fine-tuning takes a pre-trained foundation model and trains it further on a smaller set of labeled examples so it performs your specific task or matches your style. Because it updates the model's weights, fine-tuning genuinely changes the model's behavior, not just a single response. This is the phrase to remember for the exam: fine-tuning adapts the model's behavior to your data.

You choose fine-tuning when prompt engineering alone cannot get consistent results and when the change you need is about behavior rather than fresh facts. Typical goals include matching a brand's tone, following a fixed output format, classifying support tickets into your categories, or answering in a domain's style. You supply pairs of inputs and desired outputs, and the training process nudges the weights toward producing those outputs.

Fine-tuning costs more than prompting and requires good labeled data, but far less than pre-training because you start from a capable model. It also carries a maintenance cost: if your task or data changes, you may need to fine-tune again. Weigh that ongoing effort against simply adding context at request time with retrieval, which you will compare shortly.

The exam lists several fine-tuning methods. They overlap, so focus on the goal each one serves and when you would pick it.

Instruction tuning makes a model good at following directions rather than one narrow task. Domain adaptation specializes a general model for a subject area. Transfer learning is the broad principle behind fine-tuning: you reuse the general features learned during pre-training instead of starting over, which is why fine-tuning needs far less data than pre-training. Match the scenario to the method: specialized field points to domain adaptation, better instruction-following points to instruction tuning.

Model distillation transfers the knowledge of a large, capable "teacher" model into a smaller "student" model. The large model produces outputs, and the smaller model is trained to reproduce that behavior. The result is a compact model that keeps much of the quality while being cheaper and faster to run.

You choose distillation when a big model performs well but costs too much or responds too slowly for production, especially at high request volumes or on constrained hardware. Because inference cost scales with model size, a smaller distilled model can cut cost per request and latency significantly. The trade-off is that the student may lose some capability compared with the teacher, so you evaluate whether the smaller model still meets your quality bar.

For the exam, connect distillation to the cost-and-effort spectrum. Pre-training and continuous pre-training are about adding knowledge, fine-tuning is about adapting behavior, and distillation is about reducing size and cost while preserving as much performance as possible. When a question emphasizes "smaller, cheaper, faster model that mimics a larger one," the answer is distillation.

Fine-tuning is only as good as the data behind it, so the exam expects you to know how to prepare it. Several ideas matter.

• Data curation: gather and clean high-quality, relevant examples, and remove noise, duplicates, and errors.
• Governance: control who can access the data, respect privacy and licensing, and track where it came from so your use is compliant.
• Size: provide enough examples to teach the task, but remember that quality usually matters more than raw volume.
• Labeling: attach accurate, consistent labels or desired outputs, because the model learns exactly what you show it.
• Representativeness: make sure the data reflects the real range of inputs and users the model will serve.

Representativeness deserves special attention. If your fine-tuning data over-represents certain groups, phrasings, or scenarios, the model inherits that skew and produces biased results. For example, support transcripts drawn only from one region may teach the model habits that fail other customers. Curating balanced, representative data is a direct defense against bias. Governance and labeling quality reinforce this: careful, well-documented data leads to a fairer, more reliable fine-tuned model.

Reinforcement learning from human feedback (RLHF) is a technique that aligns a model with human preferences. Human reviewers look at several model outputs for the same prompt and rank them from best to worst. Those rankings train a reward model that captures what people prefer, and the foundation model is then adjusted to produce outputs that score higher against that reward. The exam phrasing to lock in is: humans rank outputs to align the model.

RLHF matters because raw pre-trained models predict likely text, which is not the same as being helpful, honest, and safe. By learning from human judgments, the model becomes better at following instructions, refusing harmful requests, and matching the tone people expect. It is a major reason modern chat assistants feel cooperative rather than merely fluent.

Recognize RLHF as a data-and-alignment step layered on top of fine-tuning: instead of only showing correct answers, you also teach the model human preferences about which answers are better. When a scenario describes people scoring or ranking responses to steer a model toward desired behavior, the answer is RLHF.

The exam contrasts three ways to shape a foundation model's output. Keep the core distinction clear: only fine-tuning changes the model.

Concrete example. Suppose a company wants its assistant to reply in a warm, on-brand support tone. Fine-tuning on past well-written replies bakes that tone into the model's behavior. Now suppose the assistant must answer questions about the company's latest product manuals. Fine-tuning is a poor fit because the manuals change often; instead RAG retrieves the right pages from the docs and adds them to the prompt so the model answers from current information without retraining. If you only need the answer as a short bulleted list, prompt engineering handles that by asking for it. Many real applications combine all three.