SAP-C03 CI/CD and DevSecOps Pipelines: CodePipeline V2, Cross-Account Promotion, Scanning, Signing, Bedrock Evaluations and Rollback

AWS CodePipeline orchestrates a release as an ordered set of stages, each holding actions that run in sequence or in parallel, and passes artifacts between them through an S3 artifact store. CodePipeline itself builds and deploys nothing: each action hands work to a provider and waits for the result.

V2 pipelines and triggers

Architect-level features (triggers with filters, pipeline-level variables, the QUEUED and PARALLEL execution modes, stage conditions and stage rollback) belong to the V2 pipeline type. For source actions that use a connection, a V2 pipeline can define push and pull request triggers filtered by:

• Branches (for example main or feature/**).
• File paths, so a pipeline in a monorepo starts only when its own directory changes (for example services/payments/**).
• Git tags, so a release starts only when someone pushes a tag that matches a pattern such as v*.

Each filter takes glob include and exclude patterns, and when a name matches both, the exclude wins, which is how you carve one name (for example a v0-scratch tag) out of a broad include. A branch filter never matches a tag, and a pull request trigger reacts to pull request events, not to tag pushes. Filters choose which events start a run; they do not change how runs interact once started.

Passing values between actions

• Pipeline-level variables are declared in the pipeline with an optional default and can be overridden when an execution starts, for example a target environment name chosen by whoever starts the run, with a default that applies when a run starts without one (an event or schedule). Actions reference them as #{variables.name}.
• Action output variables are values an action produces (a CodeBuild exported variable, a source action's CommitId or BranchName) and are referenced through the action's namespace, for example #{SourceVariables.BranchName}.
• A source revision override at start time pins which commit or image a run uses; it is not a way to pass arbitrary values.

Execution modes: SUPERSEDED, QUEUED or PARALLEL

Choose the execution mode by how concurrent runs must behave: SUPERSEDED (the default) lets the newest run win, QUEUED processes every run one at a time in order, and PARALLEL runs every execution independently at once. QUEUED and PARALLEL need a V2 pipeline.

Keep the two levers apart: trigger filters decide whether an event starts a run, and the execution mode decides what happens to runs that overlap. A requirement about which changes should start a pipeline is never solved by the execution mode, and a requirement about lost or out-of-order runs is never solved by a filter.

Gate a release with stage conditions when a machine can decide (an alarm state, a time window, a variable, a Lambda check) and with a manual approval action when a named person must decide and the decision must be recorded. Both stop a run without custom orchestration.

Stage conditions (V2)

A condition holds one or more rules, and every rule in a condition must succeed for the condition to be met. Where the condition sits decides when it runs and which results it can produce:

Automatic retry of a failed stage is a separate stage setting, not an on-failure condition result; the CodePipeline documentation notes that automatic stage retry cannot be combined with on-failure conditions.

The rule types to know:

• CloudWatchAlarm: checks the state of named alarms, for example keeping production deployments from starting while the error-rate alarm is in ALARM.
• DeploymentWindow: a cron expression with an optional time zone, for example a change window on Tuesday and Thursday mornings in America/Chicago.
• VariableCheck: compares a pipeline or action variable (including a source action's namespaced output, such as a branch name or commit message) with a value. In an entry condition with a Skip result, it lets a run bypass a stage that applies only to some runs, and the stage shows as Skipped rather than Failed.
• LambdaInvoke: runs your own function for checks no built-in rule covers.

Several rules in the same entry condition combine as an AND: the stage starts only when every rule succeeds, for example inside the change window and while the named alarm is not in ALARM. Where a rule sits matters as much as what it checks: a time window placed in an on-success condition does not gate entry, because it is evaluated after the stage has already deployed.

Manual approval

An approval action pauses the run until an IAM identity with permission approves or rejects it, optionally with a comment, and can notify reviewers through Amazon SNS. The decision, the approver and the comment become part of the execution history, which is what a change-management regulation asks for. An approval that receives no response within seven days fails. Approvals are the wrong tool when a requirement says "no people" or "least operational overhead" and the check can be automated.

Disabled transitions

Disabling the transition into a stage holds runs before it. It records no decision and needs someone (or code) to enable it again, so it is an operational lever, not an approval mechanism.

Stage rollback runs a failed stage again with the source revisions and variables of the last successful execution, returning the environment to known-good code; retry runs the stage again with the same revisions that just failed, which only helps with transient failures.

• Rollback can be started manually from the console or CLI, or automatically by an on-failure condition with a ROLLBACK result. An on-failure rollback needs no rules and no custom code.
• The target must be a successful execution started in the current pipeline structure version. Editing the pipeline definition (for example adding a stage) creates a new structure version, so earlier successes no longer qualify; automatic rollback only becomes possible again after a run succeeds under the new definition.
• Rollback is not supported in PARALLEL mode. A pipeline that must keep every commit in order and roll back automatically uses QUEUED (SUPERSEDED would drop waiting runs).
• An on-success condition with a CloudWatchAlarm rule and a Rollback result can revert a stage that finished, but only if the alarm is already in ALARM when the rule is evaluated. The rule's wait time applies only to an alarm that is already in ALARM; an alarm in OK makes the rule succeed at once. It is not a post-deployment bake monitor, so watching metrics over time after a release belongs to the deployment provider (CodeDeploy or ECS alarms) instead.

A cross-account pipeline keeps CodePipeline in a central tooling (pipeline) account and runs individual actions in target accounts by having each action assume a role in that account, while the target account reads artifacts from the tooling account's store with a customer managed KMS key. No long-term credentials for the target account exist in the tooling account.

The four pieces

1. Customer managed KMS key on the artifact store. The default encryption uses the AWS managed aws/s3 key, whose key policy you cannot edit and which another account can never use. Create a customer managed key in the pipeline's Region, grant the pipeline service role and the target accounts in its key policy, and set it as the store's encryption key.
2. Artifact bucket policy that lets the target accounts' roles read (and, where they produce output, write) artifacts.
3. A cross-account role in each target account that trusts the pipeline's service role (or tooling account), set as the action's own roleArn. The action then runs in the target account with temporary credentials.
4. For CloudFormation actions, a CloudFormation service role in the target account, set in the action's RoleArn configuration property. CloudFormation uses it to create, update and delete the stack's resources, so those permissions are scoped to the stack rather than to the action role.

Key policy delegation

When a key policy grants another account's root principal, it delegates the decision to that account: a role there can use the key only if its own identity policy also allows actions such as kms:Decrypt on that key's ARN. A KMS AccessDenied with a correct key policy and bucket policy usually means the target role's identity policy is missing this permission.

Where artifacts may be produced

An action in another account can consume an artifact only if it was produced in the pipeline account. CodePipeline cannot pass an artifact built in one target account to an action in a different target account. If a hardening or packaging step must feed several accounts, run it in the tooling account, or have the target-account step publish its output to a registry or bucket that the next account reads outside the artifact chain.

Multi-account infrastructure rollout with CloudFormation StackSets is a separate pattern covered in the Task 5.1 infrastructure-as-code lesson.

A pipeline deploys to several Regions by declaring one artifact store per Region and setting each action's region; CodePipeline copies artifacts into the store of the action's Region. Artifact promotion means building once and moving that same immutable artifact through every environment.

Regional artifact stores

• The pipeline definition uses artifactStores, a map keyed by Region, instead of the single artifactStore field; a pipeline cannot use both. The console creates the extra stores for you, but CLI, SDK and CloudFormation definitions must list them, and an update that adds a cross-Region action without them is rejected.
• KMS keys are Regional, so each store needs its own key. For a cross-account deployment, each Region's store needs a customer managed key in that Region that grants the target role; a Region whose store has no such key cannot be read by the target account, even when every other Region works.
• An existing single-Region key cannot be converted to a multi-Region key. Multi-Region keys must be created that way, and you don't need one here: separate Regional keys work.
• You do not replicate the artifact bucket yourself (S3 Cross-Region Replication) and you do not need a second pipeline per Region: CodePipeline does the copy.

Build once, promote the same artifact

Rebuilding per environment, even from the same commit, produces different artifacts because base images and dependencies change between builds. Promotion keeps production identical to what was tested:

• Container images: build and push once to a central Amazon ECR repository, and deploy by image digest (repo@sha256:…), not by tag. A digest identifies exactly one image even if someone later reuses or moves a tag.
• Organization-wide pull access: a repository policy that allows ecr:BatchGetImage and ecr:GetDownloadUrlForLayer to principals with the organization's aws:PrincipalOrgID covers every current and future account without edits. ecr:GetAuthorizationToken is granted on the pulling side (identity policy), not in a repository policy.
• Replication trade-off: ECR cross-account or cross-Region replication puts a copy near each consumer (useful for Regional independence and pull latency), but each new destination is configuration to maintain and every copy needs its own access controls.
• Infrastructure-as-code packages: build the package once and give each CloudFormation deploy action its own template configuration file (or parameter overrides) for per-environment values such as memory size, log level or table names. Settings are passed at deploy time, so the package never changes.

Make vulnerability scanning a gate by running it as a pipeline step that fails when findings exceed a threshold, so later stages never start. In CodePipeline the managed way is the InspectorScan action; outside AWS-hosted pipelines it is the Amazon Inspector SBOM Generator with the Inspector Scan API.

InspectorScan action

• SourceCodeScan mode scans the source artifact (for example open source dependencies declared in the repository) and can sit right after the source stage, so a rejected change never spends build minutes.
• ECRImageScan mode scans an image in Amazon ECR after the build and before deployment.
• The action generates an SBOM, scans it through the Inspector Scan API (the pipeline role needs inspector-scan:ScanSbom) and compares finding counts with CriticalThreshold, HighThreshold, MediumThreshold and LowThreshold. When the count goes beyond a threshold, the action fails. A critical threshold of 0 fails on any critical finding.

External CI (Jenkins, GitHub Actions, TeamCity)

When images are built outside AWS and must never leave the build host if they are vulnerable, run the Inspector SBOM Generator on the host to produce a CycloneDX SBOM and send it to the Inspector Scan API, typically through the Inspector plugin for that CI tool. Findings come back during the build, so the job fails before any push.

Why registry scanning alone is not a gate

Amazon ECR basic and enhanced scanning run on images that are already in the registry, so they report after the push, asynchronously. Turning a registry finding into a stop needs extra wiring (EventBridge plus code, or a person). Inspector's EC2 scanning examines servers, and GuardDuty Runtime Monitoring watches running workloads; neither scans an image before deployment. The choice between ECR basic and enhanced scanning is covered in the Task 1.2 containers lesson, and organization-wide Inspector setup in the Task 2.3 governance lesson.

Artifact signing proves who produced an artifact and that it has not changed since; on AWS you sign with AWS Signer profiles and verify at deployment time, so an unsigned or wrongly signed artifact cannot run.

Container images: ECR managed signing

• Configured once per registry: up to 10 signing rules, each naming a Signer signing profile and optional repository filters (for example prod*). Images pushed to matching repositories are signed automatically, with no signing tools in any pipeline.
• The signature is made with the pushing identity's credentials, so pushing roles need signer:SignPayload on the profile.
• Cross-account profiles are supported: a central security account can own the profiles and share them with workload accounts.
• Cross-Region signing is not supported: the profile must be in the same Region as the registry. A multi-Region estate needs a profile in each Region (which can still live in the security account).
• Signing manually instead means installing the Notation CLI with the AWS Signer plugin in each pipeline; it works, but it is tooling to maintain everywhere.

Signatures are stored as artifacts next to the image in ECR and are checked at deploy time by Notation-compatible verification (for example an admission controller on Amazon EKS).

Lambda: code signing configurations

• A code signing configuration lists the allowed publishers (Signer profiles) and an untrusted artifact policy, UntrustedArtifactOnDeployment.
• Warn (the default) lets a deployment with an invalid, expired or wrong-profile signature succeed and logs a warning to CloudTrail. Enforce blocks it.
• Revoking a profile or a signing job invalidates signatures made with it; it is how you retire a compromised profile, not how you tighten enforcement.

Three principles explain most signing failures:

• Listing allowed publishers is not enforcement. A code signing configuration only blocks what fails validation when its policy says so; anything else is a logged warning.
• Signing profiles are Regional resources. Account boundaries can be crossed by sharing a profile; Region boundaries cannot, so check the profile's Region whenever signing works in some places and not others.
• Signing uses the caller's permissions. Because the pushing identity signs, a signing problem limited to one pipeline points at that pipeline's role, not at the registry rule.

Amazon Bedrock Evaluations scores a model or a retrieval-augmented generation (RAG) application against a prompt dataset before release; pick the job type by what you need to measure and who (or what) does the scoring.

Models and retrievers outside Bedrock

Evaluation jobs can use your own inference responses. If the model is hosted on a SageMaker AI endpoint or elsewhere, or the retriever is a self-built OpenSearch index, generate the responses (and retrieved passages) yourself and put them in the dataset with the prompts and ground truth; the job then scores them. Nothing has to move into Bedrock, and a job cannot call an arbitrary external index on its own.

Evaluation as a pipeline gate

• Evaluation jobs are asynchronous and write results to Amazon S3. A pipeline step (for example CodeBuild) starts the job on a curated dataset, waits for completion, reads the scores and exits with a failure when a metric falls below a threshold or below the current release, which stops promotion automatically.
• Choose the metric that matches the risk: faithfulness and correctness need a judge model; robustness is programmatic.
• Evaluation is pre-release validation. Run-time protection is a different control: Amazon Bedrock Guardrails filters prompts and responses in production. Match the filter to the risk: sensitive information filters act on personal data patterns (and custom regex patterns) whether or not the text appears in the source documents, while contextual grounding checks look for answers that are not supported by the retrieved text or are irrelevant to the query, so they target hallucination rather than data leakage. Guardrail design is covered in the Task 2.5 responsible AI lesson, and production AI monitoring in Task 5.3.

Choose the strategy by how much traffic may see a new version before you know it is good and how fast you must get back: canary and linear shift traffic gradually, blue/green keeps a complete old environment to switch back to, rolling/in-place updates existing capacity in batches, and all at once has no exposure window.

AWS Lambda with CodeDeploy

• CodeDeploy shifts weights on a function alias between versions. Configurations are named for their shape: CodeDeployDefault.LambdaCanary10Percent10Minutes (10% for 10 minutes, then the rest), LambdaLinear10PercentEvery10Minutes (10% more every 10 minutes) and LambdaAllAtOnce.
• CloudWatch alarms listed in the deployment group (or SAM DeploymentPreference) make CodeDeploy shift the alias back to the original version when one fires during the deployment.
• Hooks: BeforeAllowTraffic runs a validation function after the new version is ready but before any weight moves; reporting Failed stops the deployment with nothing shifted. AfterAllowTraffic runs after the shift is complete.
• In AWS SAM, AutoPublishAlias on the function is required for DeploymentPreference, whose Type, Alarms and Hooks (PreTraffic, PostTraffic) map to the settings above.

Amazon EC2 with CodeDeploy

• In-place deployments pace themselves by minimum healthy hosts: the number (HOST_COUNT) or percentage (FLEET_PERCENT) of instances that must stay healthy, not the batch size. For example, with 12 instances that must never fall below 9 serving, a custom configuration of minimum 9 healthy hosts (or 75%) lets CodeDeploy update up to three at a time. OneAtATime keeps all but one healthy and is slow on a large fleet, HalfAtATime keeps half, and AllAtOnce keeps none, so check each built-in configuration against the stated floor.
• Blue/green provisions a replacement fleet, either by automatically copying the Auto Scaling group or from manually provisioned instances identified by tags. Traffic can be rerouted immediately or wait until someone chooses to reroute (for example after QA signs off, within a waiting period of up to two days). The original instances are either kept running or terminated after a wait you set; keeping them for a short time gives a fast switch back and then stops the cost, while terminating immediately turns rollback into a redeployment.
• Scale-out during a deployment: a new instance receives the last successfully deployed revision, so it can come up on the old version. By default CodeDeploy then runs a follow-on deployment to update these outdated instances; a deployment group set to ignore them leaves them behind. Suspending Auto Scaling processes avoids the race but stops scaling.
• Automatic rollback on deployment failure or when a listed alarm fires redeploys the last good revision.

Amazon ECS rolls back automatically in three ways: the deployment circuit breaker catches revisions that never become healthy, deployment alarms catch healthy revisions that hurt a metric, and blue/green keeps the old revision running so traffic can return to it at once.

Rolling updates

• minimumHealthyPercent and maximumPercent control how many tasks are replaced at a time; they pace the update but roll nothing back.
• The deployment circuit breaker with the rollback option detects a deployment that cannot reach steady state (for example tasks that crash at startup or fail health checks) and returns the service to the last completed deployment instead of retrying for hours.
• CloudWatch alarms in the service's deployment configuration, with rollback turned on, revert a deployment whose tasks are healthy but whose metrics go bad, such as p99 latency. Health checks can't see this kind of failure, and scaling only adds capacity.

ECS native blue/green

With the blue/green deployment strategy, ECS itself launches the green revision next to blue, can send test traffic through a test listener or listener rule, shifts production traffic, and keeps blue running for a bake time so rollback is immediate. The deployment passes through these lifecycle stages in order: RECONCILE_SERVICE, PRE_SCALE_UP, SCALE_UP, POST_SCALE_UP, TEST_TRAFFIC_SHIFT, POST_TEST_TRAFFIC_SHIFT, PRE_PRODUCTION_TRAFFIC_SHIFT, PRODUCTION_TRAFFIC_SHIFT, POST_PRODUCTION_TRAFFIC_SHIFT, BAKE_TIME and CLEAN_UP. Lifecycle hooks (Lambda functions) can be registered at every stage except SCALE_UP, BAKE_TIME and CLEAN_UP. Integration tests through the test route belong at POST_TEST_TRAFFIC_SHIFT: green receives the test traffic while blue still serves every user, so a failed hook rolls back before anyone is affected.

ECS blue/green with CodeDeploy

The older pattern uses the CodeDeploy deployment controller with a deployment group, a test listener, canary/linear/all-at-once ECS configurations and a wait before the original task set is terminated. It works, but it is a separate deployment service to configure; when a requirement says ECS should manage the deployment itself, the native strategy fits.