Training, Hyperparameter Tuning and Fine-Tuning Models on AWS (MLA-C02)

Pick a built-in algorithm by the shape of the problem and whether you have labels: each built-in is tuned for one problem type, and the wrong family either needs labels you do not have or ignores structure (time, pairs, sparsity) that matters. Built-in algorithms run in AWS-managed containers, emit their metrics automatically and need no training code, which is why they are the quickest route when one fits.

Reading the scenario for two words settles most choices: labeled or not (supervised XGBoost/Linear Learner versus unsupervised RCF/k-means) and what the output must be (a class, a number, a score, a cluster, a forecast). When no built-in fits, or the team already has code in a common library such as scikit-learn, PyTorch, TensorFlow or Hugging Face Transformers, use script mode instead (below). Choosing between built-ins, managed AI services and foundation models in the first place belongs to Task 2.1.

A training job reads each input channel from Amazon S3 in one of three input modes and with one of two distribution types, and built-in algorithms also expect a specific file format. Getting the format wrong makes a job fail or train on garbage; getting the mode and distribution wrong makes it slow to start.

Format. Built-in algorithms that accept CSV (XGBoost, Linear Learner and others) expect the target in the first column and no header row, with content type text/csv. A header row is read as data and a label in the last column is treated as a feature. Declaring a different content type (for example application/x-recordio-protobuf) does not convert the file; the declared type must match the real format.

Mode and distribution solve different problems and are often combined: FastFile removes the up-front copy, ShardedByS3Key stops every instance from receiving every file. Sharding is only safe when each instance should train on its own portion and the script trains on whatever appears in its channel; a script that already picks its own slice of the full file list would shard twice. Only the input mode changes how long the up-front copy takes; storage size and pricing options leave it as it is.

Script mode runs your own training script, unchanged or nearly so, inside an AWS-managed framework container: you choose a framework estimator (PyTorch, TensorFlow, Hugging Face, scikit-learn, XGBoost), point entry_point at the script and set a supported framework_version. AWS builds and patches the container and provisions the managed training instances, so you get custom code without maintaining images. Building your own image and using the generic Estimator (bring your own container) is the alternative when no managed container fits; a notebook instance is an interactive server, not managed training. These are the SageMaker Python SDK v2 names; SDK v3 replaces the estimator classes with ModelTrainer plus a SourceCode configuration (source directory, entry script, requirements file), while the container conventions below stay the same.

The container and the script communicate through fixed conventions:

• Hyperparameters set on the estimator (hyperparameters={'max_depth': 6, 'dropout': 0.3}) arrive as command-line arguments (--max_depth 6 --dropout 0.3), so the script parses them with argparse. Automatic model tuning passes each trial's values the same way, which is why hard-coded values defeat tuning.
• Input data: each channel is mounted under /opt/ml/input/data/<channel>, exposed as SM_CHANNEL_<NAME> (for example SM_CHANNEL_TRAINING).
• Model output: save the trained model to SM_MODEL_DIR, which is /opt/ml/model. SageMaker AI packages that directory as model.tar.gz and uploads it to the estimator's output_path. A model saved anywhere else is lost, and the job still reports Completed with an empty archive.
• Checkpoints: /opt/ml/checkpoints synced to checkpoint_s3_uri is for resuming (for example after a Spot interruption), not for the final model.
• Extra packages: put a requirements.txt in source_dir (the folder that holds the entry point). The framework container pip-installs it before running the script, so you keep the managed image. This is the lightest way to add a dependency: the image stays AWS-managed and patched, and nothing else in the job changes.

Automatic model tuning (AMT) runs many training jobs over ranges you define and keeps the one with the best objective metric; the strategy decides how each next combination is picked. Choose the strategy from the budget and from whether results should guide the search.

Parallelism matters for Bayesian search. MaxParallelTrainingJobs sets how many jobs run at once. If all jobs launch together, none can learn from another and Bayesian search degrades to random sampling. With a fixed total (MaxNumberOfTrainingJobs), lower parallelism usually finds a better result, at the cost of wall-clock time.

Ranges and scaling. Ranges are continuous, integer or categorical. The scaling type controls how values are sampled: Linear spreads them evenly, so in a range like 0.0001 to 1 about nine samples in ten land above 0.1 and the small values are barely explored; Logarithmic samples each order of magnitude equally and suits learning rates and regularization strengths (positive values spanning several powers of ten); ReverseLogarithmic concentrates samples near 1 and suits values such as momentum in the range 0 to just under 1.

Objective metric. Built-in algorithms publish their metrics automatically. A script-mode job must print the metric to its logs in a consistent format (for example test_rmse: 4.21), and the estimator or tuning job needs a metric definition with a regex that captures it; you then name that metric as the objective with its direction (Maximize or Minimize). The tuner reads only what that regex finds in the job's logs.

Warm start reuses earlier tuning jobs (parents) instead of starting from scratch. IdenticalDataAndAlgorithm requires the same data and training image, though ranges can change. TransferLearning allows different data, changed ranges and a different algorithm version, so it is the type for a dataset that has grown since the parent job.

Training time falls in two ways: stop work that will not pay off (early stopping) and spread the work that will across more hardware (distributed training). Faster input modes (FastFile, sharding) help the start-up phase, covered above.

Early stopping in a tuning job. Setting TrainingJobEarlyStoppingType to Auto applies a specific rule: after each epoch, AMT compares the current job's objective metric with the median of the running averages of the earlier jobs' objectives up to that same epoch, and stops the job if it is worse than that median. That is aggressive: roughly half of the weaker jobs can be cut early. It needs no code change for built-in algorithms that emit per-epoch metrics; a custom script (for example PyTorch) must log the objective metric after every epoch for Auto to work. Caps on runtime or job count are blunt instruments by comparison, because they stop good and bad jobs alike. Hyperband has its own built-in early stopping, so AMT requires TrainingJobEarlyStoppingType to be Off when you choose the Hyperband strategy.

Early stopping inside one job. Many algorithms stop themselves when a validation metric stops improving. For built-in XGBoost, provide a validation channel and set early_stopping_rounds: training ends after that many rounds without improvement, so the stopping point follows the data instead of a round count guessed in advance. Without a validation channel there is nothing to watch, and early stopping cannot trigger.

Distributed training. Choose by where the bottleneck is.

Data parallelism does not help an out-of-memory error caused by model state, because it replicates that state on every GPU; techniques that shrink the per-step batch only reduce activation memory, not the memory taken by the model state itself. Conversely, splitting a model that already fits adds communication overhead for no benefit. A bigger single instance is a scale-up, not distributed training. Managed Spot training with checkpoints lowers cost, not time.

Diagnose first from the gap between training and validation metrics, then apply the fix that matches: overfitting needs less capacity or more data, underfitting needs more capacity or better signal, and catastrophic forgetting needs fine-tuning that disturbs the base model less.

Catastrophic forgetting is overfitting's cousin for foundation models: full fine-tuning on a small, narrow dataset for several epochs overwrites weights that encoded general skills. Debugging a model that will not converge at all, and the evaluation metrics themselves, belong to Task 2.3.

An epoch is one full pass over the training data; a step (iteration) is one optimizer update on one batch; the batch size is how many examples go into each update. So steps per epoch = records ÷ batch size, and total steps = steps per epoch × epochs. With 120,000 records, a batch of 400 and 10 epochs, that is 300 steps per epoch and 3,000 steps in total, the numbers a learning-rate scheduler or evaluation interval needs.

• Batch size and memory. GPU memory per step grows with the per-device batch size. To keep a large effective batch on limited memory, use gradient accumulation: run several small batches and update once (16 batches of 64 behave like one batch of 1,024). Memory per step is set by the model and the per-device batch; settings that change how long you train or how data reaches the instance do not change it.
• Batch size and the learning rate. In data-parallel training the global batch is the per-GPU batch times the number of GPUs, so each epoch has far fewer updates. Leaving the single-GPU learning rate in place usually costs accuracy. A widely used heuristic (not an AWS feature) is to scale the learning rate roughly in proportion to the global batch and add a warmup that ramps it up over the first part of training so the larger rate does not destabilize early updates.
• Epochs. Too few underfit, too many overfit (and, for foundation models, encourage forgetting); early stopping picks the point from validation data rather than a guess.

Combine models for accuracy with an ensemble that fixes the specific weakness, and combine them for cost with a cascade or router that sends easy requests to a cheap model and only hard ones to an expensive model.

Cascades for cost. When a small, cheap model is reliable on most traffic and its confidence scores are well calibrated, score everything with it first and escalate only low-confidence cases to the large model. The expensive model then sees a fraction of traffic, while hard cases keep its accuracy. The saving depends on routing by difficulty: any scheme that sends every request to both models, or splits traffic without looking at difficulty, keeps either the full cost or the small model's errors.

Amazon Bedrock intelligent prompt routing is the managed version for foundation models: a prompt router sends each request to one of several models in the same model family, predicting which can answer well at lower cost, so you do not build or maintain your own request classifier. Capacity options change how much traffic one model can serve, not which model answers a request.

Start with task-specific prompt engineering and move to training only when prompting cannot deliver or costs too much. Prompting changes the input; fine-tuning, continued pre-training and distillation change the model.

Prompt engineering needs no data and no training job: clear instructions, a system prompt for role and rules, the exact output format or JSON schema, and a few input/output examples (few-shot). It is the fastest, cheapest fix when the model already knows the task but behaves inconsistently, for example when the format drifts between calls. Sampling parameters can make output less random, but they cannot specify a structure the prompt never asked for.

Fine-tuning moves repeated examples out of every prompt and into the model, so calls become short. A custom Bedrock model needs an inference option: on-demand deployment where the model supports it, otherwise Provisioned Throughput, whose fixed cost pays off only at sustained volume. RAG is for knowledge that changes, not for a fixed style.

Parameter-efficient fine-tuning (PEFT) for open-weight models in SageMaker AI (for example from SageMaker JumpStart): LoRA freezes the base weights and trains small low-rank adapter matrices, which removes most gradient and optimizer memory, limits catastrophic forgetting and produces a small artifact per task instead of a full model copy. Loading the frozen base in quantized form while training adapters cuts memory further so the job fits on a smaller GPU: QLoRA specifically quantizes the base to 4-bit (NF4), and 8-bit loading with LoRA adapters is a related option. The memory saving comes from what is trained and how the base is stored, not from how data is spread across GPUs.

Retrieval quality in a RAG system depends on three tunable parts: the embedding model that turns text into vectors, the chunking that decides what each vector represents, and the search type that matches queries to chunks. Diagnose which one causes the failure before changing anything; choosing RAG as the architecture is Task 2.1, and reranking and knowledge base setup are covered in Domain 3.

Embedding models. Amazon Titan Text Embeddings V2 lets you choose 256, 512 or 1,024 output dimensions (and whether to normalize vectors). Fewer dimensions mean smaller vectors, a smaller index and faster search for a small loss of accuracy. When a domain's terms are confused (different compound or product names landing close together), fine-tune the embedding model on query and relevant-passage pairs with a contrastive loss; only the embedding model decides where texts land in the vector space, so changes elsewhere in the pipeline cannot pull confused terms apart. Every vector in an index must come from the same model and dimension: after changing either, re-embed the whole corpus, re-embed queries with the same model, and rebuild the index. Mixing old and new vectors breaks similarity.

In Amazon Bedrock Knowledge Bases, chunking (default, fixed-size with overlap, hierarchical, semantic, none, or a custom Lambda transformation) is fixed when the data source is created and cannot be edited afterwards, so changing it means creating a new data source (or knowledge base) with the new configuration and ingesting the documents again; the search type (hybrid or semantic) can be overridden per query on supported vector stores such as Amazon OpenSearch Serverless. Very small chunks create more boundaries and lose context, and returning more results adds noise without fixing either cause.