The AI/ML Development Lifecycle & MLOps: Pipeline, Models & Metrics

An AI/ML pipeline is the ordered set of steps that carries a problem from raw data to a running model that keeps earning its keep. The exam wants you to name the stages and know what each one does, not to implement them. The order matters, because each stage depends on the output of the one before it.

Notice that the pipeline is a loop, not a line: monitoring feeds retraining, which sends a new model back through evaluation and deployment. A model is never finished the day it ships. When a question describes a symptom — messy inputs, poor test scores, a live model going stale — map it to the stage that owns that work.

You do not always train a model from scratch. For generative AI in particular, the exam expects you to know three sources of a foundation model (FM), which sit on a spectrum from buy to build.

• Use a provider's managed FM. A cloud provider hosts large, ready-trained models and lets you call them through an API. This is the fastest, lowest-effort option — you write no training code and run no servers. On AWS, Amazon Bedrock is the managed FM service.
• Adapt an open-source pre-trained model. Many capable models are published openly. You download one that someone else trained, optionally customize it, and run it yourself. You gain control and portability but take on the work of hosting and maintaining it.
• Train a custom model. You build and train a model on your own data, either from scratch or by heavily adapting a base model. This gives the most control and the tightest fit to your problem, but it demands the most data, expertise, time, and cost.

The trade-off is consistent: moving from buy toward build increases control and customization but also increases cost, effort, and required expertise. A startup validating an idea leans toward a managed FM; an organization with unique data, strict requirements, and a data-science team may justify training its own. The exam rewards matching the source to the situation, not always picking the most powerful option.

Once you have a model, you must make it available to applications. There are two broad methods, and the exam frames the choice as operational overhead versus control.

A managed API service means the provider hosts the model and exposes it as an endpoint you simply call. You send input, you get a prediction back, and you never touch servers, scaling, patching, or capacity planning. Amazon Bedrock is the classic example for foundation models. The upside is low operational overhead and fast time to value; the trade-off is less control over the underlying infrastructure and model internals, plus usage-based pricing.

A self-hosted API means you deploy the model onto infrastructure you manage and expose your own endpoint. You choose the hardware, control the environment, can keep everything inside your own network, and can customize the model freely. The cost is that you now own the operational burden — provisioning, scaling, security patching, availability, and monitoring all fall to you. Amazon SageMaker AI sits between these poles: it hosts models on managed endpoints, so you keep more control than a pure API while offloading much of the undifferentiated infrastructure work.

The rule of thumb the exam tests: choose a managed API when speed and low maintenance matter most, and self-hosting when you need maximum control, customization, or data isolation and have the team to run it.

At this exam's recognition level you should be able to point at a pipeline stage and name a fitting AWS service. You are not expected to configure them.

The two anchors to memorize are SageMaker AI for the traditional build-train-deploy lifecycle and Bedrock for calling ready-made foundation models. SageMaker AI is a family: Ground Truth labels data, Data Wrangler prepares it, Feature Store manages features, and its endpoints deploy models. Amazon Q and Kiro are productivity tools — Q answers questions and assists users, Kiro helps developers build software — rather than places you train a model. If a question asks where data lives before any of this begins, the answer is almost always Amazon S3.

MLOps (machine learning operations) applies the discipline of software operations to the ML lifecycle. Its goal is to move a model from a one-off experiment on someone's laptop to a reliable, maintainable production system. The exam tests the concepts, not the tooling.

• Experimentation. Data scientists try many datasets, algorithms, and settings. MLOps tracks these experiments so results are reproducible and comparable.
• Repeatable, automated processes. Steps such as data prep, training, and deployment are automated into pipelines rather than run by hand, so the same inputs reliably produce the same outputs.
• Scalable systems. The infrastructure must handle growing data volumes and prediction traffic without breaking.
• Managing technical debt. ML systems accumulate hidden complexity — brittle data dependencies, undocumented steps, stale models. MLOps keeps this debt visible and under control.
• Production readiness. A model is production-ready only when it is reliable, monitored, secure, and can be updated safely.

Two production concepts appear on the exam constantly. Model monitoring watches a live model for problems, especially drift — when the real-world data flowing in gradually stops resembling the data the model was trained on, so accuracy quietly falls. Model retraining is the fix: because the world changes (new fraud tactics, new customer behavior, new products), models decay over time and must be refreshed on newer data and redeployed. Together, monitoring and retraining are why the pipeline is a loop.

To judge a classification model you compare its predictions to the known correct answers. Four terms come from that comparison, and the exam loves to test which one fits a scenario. Picture the four possible outcomes: true positives (correctly flagged), true negatives (correctly cleared), false positives (flagged by mistake), and false negatives (missed). Every metric below is built from those counts.

Accuracy can lie. If only 1 in 1,000 transactions is fraud, a model that predicts "never fraud" is 99.9% accurate and completely useless — it catches nothing. That is why precision and recall matter on imbalanced problems.

The clearest way to remember precision versus recall is two examples. For a spam filter, a false positive sends a real, important email to the junk folder, which users hate — so precision matters most. For a cancer screening model, a false negative means telling a sick patient they are healthy — so recall matters most, because you must catch every true case even at the cost of a few false alarms. F1 is the compromise when both errors hurt.

A model can score beautifully on precision and recall and still be a bad decision for the business. That is why the exam pairs technical metrics with business metrics — the numbers a leader uses to decide whether a model earns its cost.

• Cost per user (or per prediction or per interaction): what it costs to serve one unit of demand. A model that is accurate but expensive per call may not scale.
• Development cost: the time, talent, and compute spent building and training the model in the first place.
• Customer feedback: satisfaction scores, ratings, and complaints that reveal whether the model actually helps real users.
• Return on investment (ROI): the value the model generates — revenue gained or cost saved — measured against what it cost to build and run.

The exam framing is simple: model metrics tell you whether the model is good at its task, while business metrics tell you whether it is worth doing. A fraud model that blocks $2 million in losses but costs $3 million to operate has excellent recall and negative ROI. Sound AI decisions weigh both sides.

Walk one realistic system through every stage to see the lifecycle as a whole. A bank wants to flag fraudulent card transactions in real time.

Data collection: the bank streams years of past transactions — amount, location, time, merchant — into Amazon S3, each labeled fraud or legitimate. Data preparation: the team removes duplicates and errors, fills gaps, and engineers features such as "distance from the last purchase" and "transactions in the past hour." Model training: using Amazon SageMaker AI, an algorithm learns the patterns that separate fraud from normal spending. Evaluation and tuning: because missing real fraud is dangerous, the team prioritizes recall, then checks precision so genuine customers are not constantly blocked, and settles on an F1 score that balances the two.

Deployment: the model is hosted behind an API so the payment system can score each transaction in milliseconds. Monitoring: dashboards track latency and accuracy — and months later, fraudsters change tactics, real transactions drift away from the training data, and recall starts to fall. Retraining: the pipeline automatically refreshes the model on recent transactions and redeploys it, closing the loop. Meanwhile the business checks ROI: fraud losses prevented versus the cost of running the system. Every Task 1.3 idea — pipeline stages, deployment method, MLOps monitoring and retraining, model metrics, and business metrics — appears in this one story.