Securing ML and AI Workloads on AWS: IAM, VPC Isolation, Auditing and Bedrock Guardrails (MLA-C02)

Securing ML and AI workloads means controlling who and what can reach your data, models and endpoints, keeping training and inference traffic on networks you control, recording every action for audit, and putting safeguards around what foundation models (FMs) and agents can say and do. Task 4.3 groups nine skills, and each maps to a small set of AWS mechanisms:

Monitoring model quality and drift belongs to Task 4.1 and cost and performance tuning to Task 4.2. Basic endpoint VPC configuration and CI/CD pipeline design are deployment topics in Domain 3; this lesson covers the security controls layered on top of them.

Use Amazon Inspector for both images and code, and choose the scanning point from the requirement: a scan inside the build stops a vulnerable image before it is stored, and continuous registry scanning catches new vulnerabilities in images that are no longer being rebuilt. Custom SageMaker AI training and inference containers often run in production for months, so the second point matters as much as the first.

Any registry-side scan, basic or enhanced, runs after the image has landed in ECR. If a rule says a vulnerable image must never be stored, the scan has to happen in the build: the SBOM Generator produces a CycloneDX software bill of materials from the built image, the Inspector Scan API returns the vulnerabilities, and the build step fails. A requirement for both a pre-push gate and ongoing re-evaluation needs both controls.

CodeGuru availability. The exam guide names Amazon CodeGuru as an example. Amazon CodeGuru Reviewer stopped accepting new repository associations on 7 November 2025 (existing associations keep working), and Amazon CodeGuru Security was discontinued on 20 November 2025. For a new setup, AWS points to Amazon Inspector code security. Neither CodeGuru service scanned the packages inside container images; that has always been Inspector's job. IaC scanning checks templates before anything is deployed, whereas AWS Config evaluates resources only after they exist.

Integrity controls complement scanning: ECR tag immutability ensures the image behind a scanned tag cannot be silently replaced.

Grant each job or principal only the actions it needs on only the resources it uses, and remember that encrypted and cross-account artifacts need permissions in more than one place. ML artifacts are training data, features, model artifacts (model.tar.gz), container images and Model Registry entries.

Scoping S3 to prefixes

A training pipeline that reads one prefix and writes another needs s3:GetObject on the input prefix (arn:aws:s3:::bucket/project/curated/*), s3:PutObject on the output prefix, and s3:ListBucket on the bucket ARN restricted with an s3:prefix condition. Object actions take object ARNs and ListBucket takes the bucket ARN, so a ListBucket on bucket/prefix/* grants nothing, while an unconditioned ListBucket reveals every key in the bucket, including sensitive prefixes the job never uses. Avoid s3:* even on a narrow prefix: it includes delete and permission-changing actions.

KMS-encrypted artifacts

With SSE-KMS and a customer managed key, S3 decrypts objects with the key on the caller's behalf, so the reader also needs kms:Decrypt (and a writer needs kms:GenerateDataKey). In the same account the key policy can grant it directly or delegate to IAM policies. Across accounts, both are required: the key policy in the owning account must allow the external role, and the role's identity policy must allow the key.

Sharing models across accounts

A SageMaker Model Registry model package group can carry a resource-based policy that lets principals in another account describe and use its model packages. A typical central-registry setup therefore needs three grants for a production deployment role: the model package group policy, read access to the artifact prefix (bucket policy plus identity policy), and kms:Decrypt on the artifact key (key policy plus identity policy). If the deployment role is also passed as the production model's execution role, that same role is the one that must read and decrypt the artifacts.

Execution roles and iam:PassRole

A SageMaker AI job or endpoint runs with its execution role, not with the identity of the person who created it, and the creator needs iam:PassRole to hand that role to SageMaker AI. Keeping the two identities apart solves most permission puzzles in this task.

An engineer who can open a bucket from a notebook has proved only that their identity can read it; a job that then fails on download needs the permission on its execution role. Grants for the job's data always target the execution role, because the job calls S3 and KMS with that role's credentials.

A least-privilege iam:PassRole statement names the one execution role ARN as its Resource and adds the iam:PassedToService condition with sagemaker.amazonaws.com. A wildcard Resource lets the user pass any role, including a more privileged one, which is a privilege-escalation path. iam:PassRole lives in the caller's identity policy; the execution role's trust policy only decides which service may assume the role.

Reject non-compliant ML resources at creation time with explicit Deny statements that test SageMaker AI condition keys, in identity policies or in service control policies (SCPs) for a whole organization. Preventive controls stop a bad request; AWS Config and SDK defaults only report it later or can be skipped.

Useful SageMaker AI condition keys include sagemaker:VpcSubnets and sagemaker:VpcSecurityGroupIds (VPC placement), sagemaker:NetworkIsolation, sagemaker:InterContainerTrafficEncryption, sagemaker:VolumeKmsKey (storage volumes attached to job instances) and sagemaker:OutputKmsKey (output artifacts in S3), sagemaker:InstanceTypes, and for notebook instances sagemaker:DirectInternetAccess and sagemaker:RootAccess.

Four evaluation rules decide whether such a policy works:

• An explicit Deny beats any Allow. If users keep a broad policy such as sagemaker:*, adding a narrower conditional Allow restricts nothing; only a Deny does.
• Conditions in one statement are ANDed. A single Deny that requires both "no VPC" and "no network isolation" fires only when both are missing. To catch either omission, write one Deny per requirement.
• A missing key behaves differently by operator. A negated operator such as StringNotEquals evaluates to true when the key is absent, so one Deny with StringNotEquals on sagemaker:VolumeKmsKey rejects both a wrong key and no key. A positive operator such as Bool does not match an absent key, so a Deny on sagemaker:NetworkIsolation = false misses requests that omit the setting; use BoolIfExists, or pair it with a Null check (which on its own tests only whether the key is present).
• Nothing in a member account overrides an SCP deny. Identity policies, permissions boundaries and resource policies cannot grant past it.

Attach jobs to private subnets and security groups in your VPC, give them VPC endpoints for the AWS services they call, and turn on network isolation when the container must make no outbound calls at all. These are separate settings with separate effects:

Under network isolation a script that downloads something at start-up (for example a file from a public model hub or a library from a package index) will fail. Stage those files in S3 and pass them as an extra input channel; SageMaker AI delivers channels to the container even though the container cannot fetch them itself.

A job in private subnets without a NAT gateway or an S3 endpoint has no route to S3, and the symptom is a connection timeout, not AccessDenied. An S3 gateway endpoint added to the subnets' route tables restores access privately.

Studio and Bedrock private connectivity

A SageMaker AI domain in Public internet only mode (the default) sends Studio traffic over a SageMaker-managed internet path that bypasses your VPC controls. VPC only mode routes all Studio traffic through your subnets, and because there is no managed internet path, you must create interface endpoints for the SageMaker API, SageMaker runtime, STS and the other services users need (or provide a NAT path, which reintroduces internet access).

Amazon Bedrock uses interface (AWS PrivateLink) endpoints only; there is no gateway endpoint. The service names differ by plane: bedrock-runtime for inference (InvokeModel, Converse), bedrock for the control plane, and bedrock-agent / bedrock-agent-runtime for agents and knowledge bases. With private DNS enabled, existing SDK calls use the endpoint without code changes. To make stolen credentials useless outside the VPC, add a Deny on the inference actions unless aws:SourceVpce equals your endpoint ID.

Use AWS CloudTrail to record who called which API and when, Bedrock model invocation logging to record what was sent to and returned by models, and AWS Config to evaluate whether resources stay compliant over time. Each answers a different question.

CloudTrail: management events versus data events

CloudTrail records the caller, time, source and resource, never the prompt or payload. For prompt and response text, turn on Amazon Bedrock model invocation logging (to CloudWatch Logs or S3). For SageMaker AI endpoints, data capture records payloads for monitoring, but it is not an audit of caller identity. CloudTrail Lake closed to new customers on 31 May 2026 (existing event data stores keep working), so for a new account rely on trails, event history and CloudWatch.

AWS Config: continuous compliance with history

AWS Config managed rules evaluate SageMaker AI resources continuously and keep a configuration history. Examples include sagemaker-notebook-no-direct-internet-access, sagemaker-notebook-instance-root-access-check, sagemaker-notebook-instance-inside-vpc, sagemaker-endpoint-configuration-kms-key-configured, sagemaker-model-isolation-enabled (network isolation on SageMaker AI models) and sagemaker-model-in-vpc (models attached to a VPC). Each rule checks one setting, so match the rule to the exact property the requirement names. Config is detective: it reports a non-compliant resource after it exists, while IAM and SCP conditions prevent it.

Across AWS Organizations, deploy the rules as an organization conformance pack from the management account or a delegated administrator (new accounts are covered automatically) and create an organization aggregator for one consolidated compliance view. AWS Audit Manager cannot be set up in new accounts or Regions from 30 April 2026; AWS points new users to Config conformance packs.

Start from the exact error: an AccessDenied names the action and often the policy type that refused it, and a timeout usually means a missing network path rather than a permission. Then fix the identity or resource the evidence points at, not the one you assume.

Effective permission is the intersection of what every applicable policy allows, minus any explicit deny. Adding an Allow never fixes an explicit deny, and a failed call named in CloudTrail (for example kms:Decrypt by the execution role) is the fastest pointer to the missing grant.

Applications running on AWS should call Amazon Bedrock with IAM role credentials (an EC2 instance profile, ECS task role, Lambda execution role or EKS service account role), which the AWS SDK obtains and refreshes automatically with nothing to store. Amazon Bedrock API keys exist for tools and clients that accept only a bearer-style API key.

Because a short-term key expires within hours, whatever uses it must be able to obtain a fresh one, for example by generating it from the running principal's own credentials.

Governing API keys. Long-term keys are created with iam:CreateServiceSpecificCredential; deny it when iam:ServiceSpecificCredentialServiceName is bedrock.amazonaws.com to stop creation (iam:ServiceSpecificCredentialAgeDays limits how long new keys may live). Requests made with any API key are authorized through bedrock:CallWithBearerToken; deny it when bedrock:bearerTokenType is LONG_TERM to disable existing long-term keys while short-term keys keep working. Applied as SCPs, these controls cover the whole organization.

Amazon Bedrock Guardrails is a configurable set of policies that evaluates prompts and model responses and blocks or masks content that breaks your rules. Pick the policy by what it detects:

Word filters match exact text, so they cannot recognise a topic phrased in a roundabout way, and contextual grounding judges support for an answer, not its subject or its personal data.

Prompt attacks and input tags

When you call InvokeModel with a guardrail, the prompt attack filter evaluates only content wrapped in the guardrail input tags (with the tag suffix given in the request), so that a developer's own system prompt is not flagged. If the user's message is not tagged, jailbreaks pass while other filters still work.

Sensitive information: block, mask, detect

Each PII type or regex has separate actions for inputs and outputs: Block (reject with the configured message), Mask (replace the value with a placeholder such as {EMAIL} and return the rest), or detect only. Block on input stops a prompt before the model sees it; mask on output keeps the answer usable. Company-specific identifiers with a known format need a regex filter.

Guardrails change what the model and the user see, not what is logged. Model invocation logs keep the original input, and masked or blocked content can appear in plain text there and in guardrail trace output. Protect log readers with a CloudWatch Logs data protection policy that masks the identifiers (only principals with logs:Unmask see the originals).

Guardrails outside Bedrock inference

The ApplyGuardrail API evaluates any text against a guardrail without invoking a Bedrock model. For a model that must stay on a SageMaker AI endpoint (or anywhere else), call ApplyGuardrail on the prompt, invoke the model only if it passes, then call it again on the response. Guardrails are a Bedrock feature, so ApplyGuardrail is how a model hosted elsewhere gets them.

Enforce guardrail use in IAM with the bedrock:GuardrailIdentifier condition key, and treat everything a model reads as untrusted: give agent tools least privilege, take identity out of the model's hands, and require confirmation for consequential actions.

Enforcing a guardrail with IAM

• Allow the inference actions when bedrock:GuardrailIdentifier equals the guardrail ARN with its version (for example …:guardrail/gr7xk2pq:5), and add an explicit Deny when it does not equal that value. The Deny is what refuses calls that name no guardrail or another one, especially when a broad bedrock:* allow must stay.
• A guardrail ARN without a version refers to the DRAFT version, so it will not match a numbered version.
• Converse is authorized as bedrock:InvokeModel (and ConverseStream as bedrock:InvokeModelWithResponseStream), so the same statements cover both APIs.
• AWS documents a limitation: a role that enforces a guardrail this way should not be used for RetrieveAndGenerate, InvokeAgent or InvokeInlineAgent, because those APIs make internal model calls that do not all carry the guardrail identifier and fail. Use a separate role for those workloads and keep the guardrail in their requests.

ML and AI security risks and mitigations

Prompt instructions ("ignore commands in emails") and denied topics work on text and can be bypassed, and logging only detects. Confirmation protects against actions the user did not intend; protecting other users' data from the user is an authorization check inside the tool.