The AI/ML Development Lifecycle & MLOps: Pipeline, Models & Metrics

An AI/ML pipeline is the ordered set of steps that carries a problem from raw data to a running model that keeps earning its keep. The exam wants you to name the stages and know what each one does, not to implement them. The order matters, because each stage depends on the output of the one before it.

Where labeling sits varies by source. AWS's Well-Architected Machine Learning Lens groups labeling with data collection (label, ingest, aggregate), while many textbooks file it under preparation. Either way, labeling happens before training, and the activities that unambiguously belong to preparation are cleaning, de-duplicating, imputing missing values and engineering features.

Notice that the pipeline is a loop, not a line: monitoring feeds retraining, and a retrained model goes back through evaluation before deployment — you never ship a new version you have not tested. A model is never finished the day it ships. When a question describes a symptom — messy inputs, poor test scores, a live model going stale — map it to the stage that owns that work.

You do not always train a model from scratch. For generative AI in particular, the exam expects you to know three sources of a foundation model (FM), which sit on a spectrum from buy to build.

• Use a provider's managed FM. A cloud provider hosts large, ready-trained models and lets you call them through an API. This is the fastest, lowest-effort option — you write no training code and run no servers. On AWS, Amazon Bedrock is the managed FM service.
• Adapt an open-source pre-trained model. Many capable models are published openly with their weights. You download one that someone else already pre-trained, inspect and fine-tune it on your own data, and run it in an environment you control — even moving it between clouds. You skip the months of pre-training, but you take on the work of hosting, scaling, patching and maintaining it.
• Train a custom model. You build and train a model on your own data, either from scratch or by heavily adapting a base model. This gives the most control and the tightest fit to your problem, but it demands the most data, expertise, time, and cost.

The trade-off is consistent: moving from buy toward build increases control and customization but also increases cost, effort, and required expertise. A startup validating an idea leans toward a managed FM; an organization with unique data, strict requirements, and a data-science team may justify training its own. The exam rewards matching the source to the situation, not always picking the most powerful option.

Once you have a model, you must make it available to applications. There are two broad methods, and the exam frames the choice as operational overhead versus control.

A managed API service means the provider hosts the model and exposes it as an endpoint you simply call. You send input, you get a prediction back, and you never touch servers, scaling, patching, or capacity planning. Amazon Bedrock is the classic example for foundation models. The upside is low operational overhead and fast time to value; the trade-off is less control over the underlying infrastructure and model internals, plus usage-based pricing.

A self-hosted API means you deploy the model onto infrastructure you manage and expose your own endpoint. You choose the hardware, control the environment, can keep everything inside your own network, and can customize the model freely. The cost is that you now own the operational burden — provisioning, scaling, security patching, availability, and monitoring all fall to you. Amazon SageMaker AI sits between these poles: it hosts models on managed endpoints, so you keep more control than a pure API while offloading much of the undifferentiated infrastructure work.

The rule of thumb the exam tests: choose a managed API when speed and low maintenance matter most, and self-hosting when you need maximum control, customization, or data isolation and have the team to run it.

At this exam's recognition level you should be able to point at a pipeline stage and name a fitting AWS service or feature. You are not expected to configure them.

Why Amazon S3 is the data lake

Amazon S3 holds virtually unlimited data of any format — logs, images, CSV exports, Parquet — durably and cheaply, and training jobs read from it directly. The near-misses each fail one requirement: S3 Glacier storage classes are archive tiers built for rarely accessed data, with retrieval fees and (for the colder classes) retrieval delays, so they suit old datasets you must keep, not data training jobs read often. Amazon DynamoDB and Amazon RDS are databases for an application's structured records and transactions; they are not built to hold large volumes of images or mixed raw files for bulk reading.

Feature Store: the same values for training and inference

When two teams each compute "90-day spend" in their own code, the value a model learned from in training can differ from the value computed at prediction time — a training/serving mismatch that quietly degrades predictions. SageMaker Feature Store fixes this by defining each feature once and serving it from two stores: an offline store that keeps the history used to build training datasets, and an online store that returns the latest values with low latency to a live model. Data Wrangler prepares data; Ground Truth labels it; Feature Store shares and serves the finished features.

SageMaker AI vs Amazon Bedrock

The two anchors to memorize are SageMaker AI for the build-train-deploy lifecycle of your own models and Bedrock for calling ready-made foundation models through a serverless API. The boundary has one subtlety: SageMaker AI can also host foundation models — SageMaker JumpStart lets you pick a pre-trained or open model, fine-tune it, and deploy it to a SageMaker endpoint whose instance type and scaling you choose and pay for. So "use a foundation model" alone does not mean Bedrock; "use a foundation model with no infrastructure to provision" does, while "host and control the endpoint yourself" points to SageMaker AI.

AI productivity tools: Amazon Q, Amazon Quick and Kiro

These three are assistants you use, not places where you build or train a model. Tell them apart by the user they serve.

In short: Q Developer helps people who build and run AWS workloads understand their AWS resources; Q Business serves employees over company knowledge; Amazon Quick serves business teams who research, analyse data and automate work; Kiro serves engineers who turn feature ideas into specifications and code. None of them stores training data — that job belongs to Amazon S3.

MLOps (machine learning operations) applies the discipline of software operations to the ML lifecycle. Its goal is to move a model from a one-off experiment on someone's laptop to a reliable, maintainable production system. The exam tests the concepts, not the tooling.

• Experimentation. Data scientists try many datasets, algorithms, and settings. MLOps tracks each experiment — data version, code, parameters, metrics — so results are reproducible and comparable.
• Repeatable, automated processes. Data prep, training, evaluation and deployment run as automated pipelines rather than by hand, so the same inputs reliably produce the same outputs.
• Scalable systems. The infrastructure must handle growing data volumes and prediction traffic without breaking.
• Managing technical debt. ML systems accumulate hidden complexity that works today but makes every future change risky (examples below).
• Production readiness. A model is production-ready only when it is reliable, monitored, secure, and can be updated — and rolled back — safely.

Speed and tuning are not MLOps. Making training faster or squeezing a little more out of a test score — for example, with bigger GPU instances — does not make the work reproducible or tell you when a live model fails. Only versioned data, code and models, run through automated pipelines and watched by monitoring, fix reproducibility.

What ML technical debt looks like

• Brittle data dependencies: a model silently relies on an upstream table or export whose format can change without warning.
• Undocumented manual steps: one person exports a file by hand and runs a notebook nobody else understands, so the process cannot be repeated if they are away.
• Unversioned models and data: nobody can say which model file is live, what data trained it, or how to rebuild it.
• Duplicated feature code: the same feature calculated differently in several places, drifting apart over time.

None of these break anything on day one — that is what makes them debt. The repayment is automation (pipelines instead of hand steps) and versioning (of data, code and models).

Making a model production-ready

A good test score is necessary but not sufficient. Before go-live a team needs:

• Versioned releases with rollback. Register every model build in a model registry (on AWS, SageMaker Model Registry) before deploying it, so you always know which version is serving and can redeploy an earlier one without retraining.
• Monitoring against a baseline. Capture what normal input data and prediction quality look like at launch, track the live model against that baseline, and raise alerts when it slips — SageMaker Model Monitor does this on AWS — so a decline is caught before customers complain.
• Security and scaling appropriate to the traffic and data the model will handle.

A better test score, however it is obtained, does not make a release traceable or tell you when the live model starts to slip; versioning and monitoring do.

Model monitoring watches a live model for two different kinds of trouble, and the fix depends on which one you see. Retraining is the fix for decaying prediction quality; it is not a fix for every problem.

The diagnostic clue is timing and type. A model that passed evaluation and then gradually worsened with nothing changed in its code points to drift, because overfitting and bad labels would have shown up in testing. A latency problem is about capacity, so retraining will not help it — and scaling out will not restore a model whose quality has drifted. On AWS, SageMaker Model Monitor watches endpoints for data and model-quality drift against a baseline; endpoint latency and error metrics appear in Amazon CloudWatch. Together, monitoring and retraining are why the pipeline is a loop.

To judge a classification model you compare its predictions to the known correct answers. Four terms come from that comparison, and the exam loves to test which one fits a scenario. Picture the four possible outcomes: true positives (correctly flagged), true negatives (correctly cleared), false positives (flagged by mistake), and false negatives (missed). Every metric below is built from those counts.

A quick way not to swap the definitions: precision looks at what the model flagged ("how many of my alarms were real?"); recall looks at what was really there ("how many real cases did I catch?").

Accuracy can lie. If only 1 in 1,000 transactions is fraud, a model that predicts "never fraud" is 99.9% accurate and completely useless — the majority class dominates the count and hides that it catches nothing. That is why precision and recall matter on imbalanced problems.

The clearest way to remember precision versus recall is two examples. For a spam filter, a false positive sends a real, important email to the junk folder, which users hate — so precision matters most. For a cancer screening model, a false negative means telling a sick patient they are healthy — so recall matters most, even at the cost of a few false alarms.

Putting it together: imbalance plus two costly errors

Consider an insurer screening claims, where roughly 3% are fraudulent. Paying a fraudulent claim is expensive (a false negative), but wrongly holding up an honest customer's claim is also expensive (a false positive — complaints, churn, investigation cost). Work through the options: accuracy is out, because the 97% honest majority inflates it; precision alone would let the model flag almost nothing and miss fraud; recall alone would let it flag nearly everything and swamp honest customers. When the classes are imbalanced and both errors matter, and you want one number to compare candidate models, F1 score is the fit, because it falls sharply if either precision or recall is weak.

A model can score beautifully on precision and recall and still be a bad decision for the business. That is why the exam pairs technical metrics with business metrics — the numbers a leader uses to decide whether a model earns its cost.

• Cost per user (or per prediction or per interaction): what it costs to serve one unit of demand. A model that is accurate but expensive per call may not scale.
• Development cost: the time, talent, and compute spent building and training the model in the first place.
• Customer feedback: satisfaction scores, ratings, and complaints that reveal whether the model actually helps real users.
• Return on investment (ROI): the value the model generates — revenue gained or cost saved — measured against what it cost to build and run.

Model metrics tell you whether the model is good at its task; business metrics tell you whether it is worth doing.

Worked example: per-unit ROI

Compare value and cost on the same unit. Suppose each document a model classifies correctly saves $0.50 of manual handling.

The more accurate model destroys value on every document, because its running cost exceeds what its extra accuracy saves. Compare value and cost on the same unit, and check any minimum quality requirement first.

Walk one realistic system through every stage to see the lifecycle as a whole. A bank wants to flag fraudulent card transactions in real time.

Data collection: the bank lands years of past transactions — amount, location, time, merchant — in Amazon S3, each labeled fraud or legitimate. Data preparation: the team removes duplicates and errors, fills gaps, and engineers features such as "distance from the last purchase" and "transactions in the past hour," publishing them to SageMaker Feature Store so training and live scoring use identical values. Model training: using Amazon SageMaker AI, an algorithm learns the patterns that separate fraud from normal spending; each experiment's data, parameters and metrics are tracked. Evaluation and tuning: fraud is rare and both missed fraud and blocked customers are costly, so the team compares candidates on F1 score while checking recall and precision individually.

Deployment: the chosen version is registered in the model registry and hosted behind an API so the payment system can score each transaction in milliseconds. Monitoring: CloudWatch tracks latency, and Model Monitor tracks prediction quality against the launch baseline. During a holiday spike latency climbs, so the team scales the endpoint; months later, fraudsters change tactics, live transactions drift away from the training data, and recall starts to fall. Retraining: the automated pipeline refreshes the model on recent transactions, evaluates it, and redeploys it — with the previous version still in the registry for rollback. Meanwhile the business checks ROI: fraud losses prevented per transaction versus the cost of scoring it. Every Task 1.3 idea — pipeline stages, deployment method, MLOps, model metrics, and business metrics — appears in this one story.