Applying Authentication Mechanisms: IAM, Secrets Manager, and Networks

AWS authenticates every API call by verifying a cryptographic signature attached to the request. When you call an AWS service — from the console, the AWS CLI, an SDK, or another AWS service — the request is signed with a set of credentials, and IAM resolves those credentials to a principal: an IAM user, an IAM role session, or the account root user. Only after the principal is identified does authorization (policy evaluation) happen. Authentication asks who are you; authorization asks what are you allowed to do. The DEA-C01 exam expects you to keep those two ideas separate.

Credentials come in two flavors. Long-term credentials are an access key ID and secret access key attached to an IAM user — they never expire on their own, which makes them a standing liability if leaked. Temporary credentials are issued by AWS Security Token Service (STS) when a principal assumes a role: an access key, secret key, and session token that expire automatically after the session duration. Almost every exam scenario that mentions hard-coded keys, leaked keys, or credentials in code has the same shape of answer: replace the long-term credentials with an IAM role and temporary STS credentials.

For data engineering this matters everywhere. A Glue job, a Lambda function, a Redshift cluster loading from S3, an EMR cluster's EC2 instances — none of them should hold access keys. Each assumes a role, AWS delivers short-lived credentials to the compute automatically, and rotation is handled for you. Human access follows the same principle: prefer federated identities (through IAM Identity Center) that assume roles over IAM users with passwords and access keys.

IAM gives you three identity building blocks, and the exam tests when to use each. An IAM user is a permanent identity with its own credentials, meant for the rare cases where a specific person or legacy system truly needs long-term keys. An IAM group is a collection of users used purely to attach policies once and have them apply to every member — groups cannot be referenced as a principal in resource policies and cannot be assumed; they are a permissions-management convenience, not an identity that authenticates. An IAM role is an assumable identity with no long-term credentials at all: a trust policy defines who may assume it, and permission policies define what the resulting session can do.

Roles are the default answer for anything automated. A role's trust policy is the authentication gate: for a Lambda execution role it trusts the Lambda service principal; for cross-account access it trusts the other account; for federation it trusts an identity provider. A common gotcha: assuming a role replaces your permissions entirely — the session has the role's permissions, not the union of the role's and the caller's.

Also know the vocabulary of skill 4.1.2: you create and update IAM groups and roles for people and workloads, and you attach policies to endpoints and services — VPC endpoint policies and service-linked or service roles — so that both identities and the paths they travel are governed.

Each compute or orchestration service authenticates to other AWS services through its own role, and you should be able to name the pattern for each. AWS Lambda uses an execution role: the function assumes it at invocation, and its policies must grant exactly what the function touches — for a pipeline function, perhaps reading one S3 prefix, writing to one Kinesis stream, and fetching one secret from Secrets Manager. Lambda also needs the ability to write CloudWatch Logs, which the AWSLambdaBasicExecutionRole managed policy covers.

Amazon API Gateway plays both sides of authentication. Inbound, it can authenticate callers with IAM authorization (requests signed with SigV4), Amazon Cognito user pools, or Lambda authorizers. Outbound, an API Gateway integration role lets the API call AWS services directly — for example, an API that drops records onto a Kinesis stream without any Lambda in between assumes a role trusted by the API Gateway service principal.

The AWS CLI authenticates with whatever credentials its profile resolves: for humans, prefer IAM Identity Center sign-in, which issues temporary credentials; on EC2 or in containers, the CLI automatically picks up the instance or task role from the metadata endpoint — no keys on disk. Named profiles can also be configured to assume a role, so an engineer authenticates once and pivots into a scoped data-admin role for sensitive operations.

AWS CloudFormation deserves special attention because it is a privilege-escalation vector. By default a stack operates with the permissions of the user who launches it, but you can pass a service role that CloudFormation assumes for all stack operations. That is powerful and dangerous: anyone allowed to launch a stack with that role effectively borrows its permissions. The exam-safe framing is to scope CloudFormation service roles tightly and control who may pass them with the iam:PassRole permission — PassRole is the control that stops a low-privilege user from handing a high-privilege role to a service.

AWS Secrets Manager exists because some authentication cannot be replaced by IAM roles: databases, third-party APIs, and SaaS platforms still want usernames, passwords, and API keys. Secrets Manager stores those secrets encrypted with AWS KMS, controls retrieval through IAM policies, logs every access in CloudTrail, and — its defining feature — rotates credentials automatically on a schedule.

Rotation works two ways. Managed rotation is available for services like Amazon RDS, Amazon Aurora, Amazon Redshift, and Amazon DocumentDB: Secrets Manager coordinates with the database to change the password and update the secret with no code from you. For anything else, Secrets Manager invokes a rotation Lambda function you supply, which knows how to create and test new credentials against the target system. Two rotation strategies matter for the exam: the single-user strategy changes the password of one database user (brief risk that in-flight connections using the old password fail), while the alternating-users strategy flips between two users so one always holds valid credentials — better for high-availability pipelines. Applications should always fetch the secret at connection time (and refresh on authentication failure) rather than caching it forever, or rotation will break them.

Know the contrast with AWS Systems Manager Parameter Store: Parameter Store also holds configuration and SecureString values cheaply, but it has no built-in rotation. The exam heuristic: if the scenario says rotate, database credentials, or cross-account secret access, the answer is Secrets Manager; if it says plain configuration values at minimal cost, Parameter Store is fine. A concrete pipeline pattern: a Glue connection or Lambda function references a Secrets Manager secret for the Redshift password; its IAM role is granted secretsmanager:GetSecretValue on that one secret's ARN; rotation runs every 30 days; nothing is ever hard-coded.

The network layer is your first authentication filter: traffic that cannot reach a service cannot even attempt to log in. VPC security groups are stateful virtual firewalls attached to ENIs — instances, Lambda functions in a VPC, Redshift clusters, EMR nodes, and interface endpoints all carry them. They support allow rules only (anything not allowed is denied), and return traffic is automatically permitted. The most exam-relevant trick is security-group referencing: instead of allowing an IP range into your Redshift cluster's port, allow inbound from the security group attached to your Lambda or EMR compute. Membership in the group becomes the credential, and the rule survives IP changes. Updating security groups (skill 4.1.1) usually appears as: a new consumer needs database access, so you add an inbound rule referencing the consumer's security group on the database port.

VPC endpoints keep pipeline traffic off the public internet. Gateway endpoints exist for Amazon S3 and DynamoDB — they are free and work through route-table entries. Interface endpoints, powered by AWS PrivateLink, put an ENI with a private IP in your subnets for most other services (Kinesis, Glue, Secrets Manager, STS, Redshift Data API, and many more) and also for third-party or cross-account services. Interface endpoints accept security groups; gateway endpoints do not.

Both endpoint types accept an endpoint policy — a resource policy applied to the network path itself, which is exactly what skill 4.1.5 means by applying IAM policies to endpoints. A classic pattern: the S3 gateway endpoint policy allows access only to your data-lake buckets, and the bucket policy in turn denies requests unless aws:SourceVpce matches your endpoint. Now authentication is layered: valid IAM credentials and the approved private network path are both required.

S3 Access Points complete the picture for shared data lakes. Instead of one enormous bucket policy serving every team, you create a named access point per application or team, each with its own policy, and optionally restrict an access point to a specific VPC so the data is reachable only from inside that network. The bucket policy simply delegates access control to the access points. When an exam scenario describes hundreds of consumers with different permissions on one bucket, or a requirement that data be accessible only from within a VPC, S3 Access Points are the intended answer.

The managed/unmanaged distinction (skill 4.1.6) is about how much of the security stack — including authentication — AWS operates for you. With fully managed and serverless services such as Amazon Athena, AWS Glue, AWS Lambda, Amazon Redshift Serverless, and Amazon MSK Serverless, there are no servers you can log into: authentication is IAM end to end, AWS patches the infrastructure, and your responsibility shrinks to writing good policies and protecting your data. With unmanaged or self-managed deployments — databases on EC2, self-hosted Kafka or Airflow, an EMR cluster where you can SSH to nodes — you inherit OS users, SSH keys, database password files, and the patching burden along with them.

Partly managed services sit between the extremes. Amazon EMR on EC2 provisions and monitors the cluster, but the EC2 instances live in your VPC with security groups, instance profiles, and optional SSH access you must govern; EMR Serverless removes those hosts entirely. Amazon RDS manages the OS but leaves database-level users and passwords to you — which is why it pairs so naturally with Secrets Manager rotation.

For the exam, translate scenarios like this: if the requirement stresses minimal operational overhead or no credential management, choose the more managed option (Glue over self-managed Spark, Redshift Serverless over a database on EC2, EMR Serverless over EMR on EC2). If the scenario grants a need for OS-level control or custom software, accept the unmanaged option and expect follow-up questions about the security groups, key pairs, and instance roles you now must maintain.

Amazon SageMaker Unified Studio (part of the next generation of Amazon SageMaker) is a single environment where data engineers, analysts, and ML builders work with data, SQL, notebooks, and models — and its access model is built from three nested constructs the exam names explicitly (skill 4.1.7). A domain is the top-level boundary: it holds the user directory integration (typically IAM Identity Center for single sign-on, so people authenticate with corporate credentials rather than IAM users), the associated AWS accounts, and global policies. Domain units are an organizational hierarchy inside the domain — create one per business unit or team, delegate administration of each unit to that team's owners, and organize assets under them so authority is distributed instead of centralized.

Projects are where actual work and authentication meet. A project is a collaboration workspace with its own membership, its own tooling, and — critically — its own project IAM role. When a member runs a query, a notebook, or a job inside the project, the work executes with the project role's permissions, not with the individual's personal AWS credentials. Data and compute made available to the project are available to its members; people outside the project do not see them. Membership therefore becomes the unit of access control: add an analyst to the sales-analytics project and they can immediately query that project's assets with governed, temporary credentials.

The exam-level takeaway: domains establish who can sign in (federated authentication), domain units establish who administers what, and projects establish which role your work runs as. If a scenario asks how to give a cross-functional team shared, governed access to lakehouse data in SageMaker Unified Studio without minting IAM users, the answer is to create a project, add the members, and let the project role carry the permissions.

Put the layers together with a realistic build. Your team ingests orders through Amazon API Gateway, transforms them with AWS Lambda, stages them in Amazon S3, and loads them into Amazon Redshift — and security review demands no long-lived credentials and no public network paths.

At the API layer, API Gateway authenticates client apps with a Cognito user pool authorizer, and an integration role trusted by API Gateway writes valid requests to a Kinesis data stream. The Lambda consumer has an execution role scoped to that stream, its S3 staging prefix, and one Secrets Manager secret ARN. Because the function runs in private subnets, it reaches S3 through the gateway endpoint and Secrets Manager and STS through interface endpoints — the endpoint policies allow only this account's principals, and the bucket policy requires the request to arrive through the VPC endpoint. The staging bucket exposes an S3 Access Point restricted to the VPC, so even a leaked bucket name is unreachable from outside.

For the load, Redshift sits in private subnets with a security group allowing inbound only from the Lambda function's security group on the database port. The loader fetches the current database password with GetSecretValue; Secrets Manager rotates it on a 30-day schedule using the alternating-users strategy so loads never fail mid-rotation. Redshift's own COPY authenticates to S3 by assuming an attached IAM role — no keys in the COPY statement. Every credential in this pipeline is either a temporary STS credential or a rotated secret, which is exactly the posture DEA-C01 scenarios reward.