DEA-C01 cheat sheet — every exam topic on one page

Data Ingestion and Transformation

Data Ingestion on AWS: Kinesis, MSK, DMS, Glue, and Batch Patterns

• Kinesis Data Streams for AWS-native streaming; MSK when the scenario needs Apache Kafka; SQS for independent work items with per-message retry.
• DMS task types: full load, CDC only, or full load and ongoing replication; MySQL CDC needs binary logging (automated backups on RDS) with ROW format.
• Key-based bookmarks and watermarks catch new rows only; log-based CDC is the only way to capture updates and deletes.
• Load Redshift with COPY from S3; split compressed input into equal files (a multiple of the slice count) because one compressed file is read by one slice.
• EventBridge Scheduler supports time zones and calls Glue directly; Glue triggers and scheduled rules use UTC.
• Kinesis-to-Lambda concurrency = shards x parallelization factor; set a batching window to get bigger batches at low volume.
• Bound retries and pair bisect-on-error with an on-failure destination (S3 keeps the payload); use ReportBatchItemFailures for partial batches.
• Hot DynamoDB partitions and hot Kinesis keys need key design (sharding or suffixes), not more total capacity; DAX does not absorb writes.
• Polling consumers share 2 MB/s and five GetRecords calls per second per shard; enhanced fan-out gives each consumer its own pipe.
• Only retained streams and raw data in S3 make a pipeline replayable; keep consumers idempotent.

Transform and Process Data on AWS: Glue, EMR, Lambda, and Redshift

• Glue for serverless Spark ETL (no Hive), EMR for scale and framework control, EMR Serverless for intermittent Spark or Hive, Lambda for short per-object or per-record transforms, Redshift SQL for ELT on data already loaded.
• Job bookmarks need job.init(), job.commit() and DynamicFrame reads with a transformation_ctx; Pause never advances state, and auto scaling never makes a job incremental.
• push_down_predicate still lists every partition; for millions of partitions add a partition index and use catalogPartitionPredicate.
• Glue connection ENIs have private IPs only: give the subnet an S3 endpoint or NAT, and the security group a self-referencing all-TCP inbound rule.
• Keep JDBC passwords in Secrets Manager for built-in rotation — Parameter Store SecureString and connection properties have none.
• Convert CSV to compressed, partitioned Parquet; Firehose conversion accepts JSON only, and CREATE EXTERNAL TABLE STORED AS PARQUET converts nothing.
• Stragglers mean data skew: salt the hot key or broadcast a small side — but broadcasting a multi-GB table runs the driver out of memory.
• The Redshift Data API is asynchronous: submit, return the ID, fetch later — or set WithEvent for an EventBridge completion event; API Gateway REST API usage plans give per-client keys, throttling and quotas.
• Executor OOM means oversized partitions (bigger or memory-optimized workers, more shuffle partitions); driver OOM means collecting or broadcasting too much. Compact small files toward 128 MB–1 GB before heavy processing.

Orchestrating Data Pipelines on AWS: Step Functions, MWAA, and Glue Workflows

• AWS Step Functions is the default serverless orchestrator: direct integrations with Glue, EMR, Lambda and more, per-state Retry/Catch, and no infrastructure.
• Standard workflows are exactly-once, run up to a year, keep history, support .sync/callbacks and emit status events; Express workflows run up to five minutes and are at-least-once (async) or at-most-once (sync) — and the type is fixed at creation.
• Request-response moves on immediately, .sync waits for supported AWS jobs, and .waitForTaskToken waits for an external caller — with HeartbeatSeconds to detect a dead worker.
• Retriers match in order, States.ALL goes alone and last; scope retries to transient errors, use jitter, and use Catch with ResultPath "$.error" to keep the input alongside the error.
• Distributed Map reads items from S3 and runs up to 10,000 child executions with tolerated failures and batching; inline Map is limited by concurrency, payload and history quotas.
• S3 notifications reach SNS, SQS, Lambda or EventBridge — Step Functions only through EventBridge; EventBridge Scheduler adds time zones, one-time schedules, flexible windows and per-schedule retries/DLQs.
• Asynchronous Lambda failures go to on-failure destinations or a DLQ; SQS-polled failures go to the queue's redrive DLQ; size the visibility timeout to at least six times the function timeout.
• Amazon MWAA is for Apache Airflow DAGs and its ecosystem; scale it with maximum workers, schedulers and environment class. AWS Glue workflows orchestrate only Glue crawlers and jobs.
• Alert on Standard failures with EventBridge status change events to SNS; alert on Express failures from CloudWatch metrics or logs. Redrive resumes failed Standard runs from the failed step within 14 days.

Programming Concepts for AWS Data Engineering: IaC, CI/CD, and Lambda Tuning

• Reduce runtime by reading less: Parquet over CSV, filters on partition columns, push_down_predicate (or catalogPartitionPredicate with a partition index for huge partition counts), and built-in Spark functions instead of Python UDFs.
• Reserved concurrency caps and guarantees a function and is removed from the shared pool even when idle; provisioned concurrency pre-warms a version or alias and is not a cap.
• For SQS triggers, limit concurrency with the event source mapping's maximum concurrency (per mapping), not reserved concurrency alone; S3 notifications are asynchronous and have no event source mapping.
• CPU scales with Lambda memory up to six vCPUs at 10,240 MB, with one vCPU at about 1,769 MB; /tmp is private scratch up to 10,240 MB, and EFS (VPC, access point, /mnt/ path, NFS 2049) is shared and persistent.
• CloudFormation, CDK and SAM all deploy through CloudFormation; change sets preview, drift detection finds out-of-band edits, and StackSets with automatic deployment cover every account in an OU.
• SAM gives policy templates, connectors, sam sync for development and sam deploy change sets for releases, plus AutoPublishAlias with DeploymentPreference for canary rollouts with alarm rollback.
• CI/CD for data pipelines builds once and promotes the same artifact, gates production with change set, manual approval and execute, and rolls back by reverting the commit in Git.
• Spark parallelism equals partitions and splits: extra workers sit idle beyond the partition count, and a single gzip file is read by one task.

Data Store Management

Choosing an AWS Data Store: Redshift, DynamoDB, Aurora, S3, and MemoryDB

• Read the access pattern first: analytics → Redshift or S3 + Athena, OLTP with joins and constraints → RDS/Aurora, key/value at scale → DynamoDB, durable microsecond in-memory → MemoryDB, replayable events → Kinesis/MSK.
• RA3 managed storage scales storage without compute and is required for data sharing; consumers query shared data live with their own compute and bill.
• Redshift cost: Serverless for intermittent use, usage limits to cap RPU-hours, reservations or reserved nodes only for steady load, pause/resume only for predictable idle windows, concurrency scaling for short bursts.
• Spectrum reads S3 in place and federated queries read live RDS/Aurora rows with no copy; zero-ETL replicates into Redshift; materialized views auto-rewrite queries but auto refresh is best effort, so refresh explicitly when freshness must be guaranteed.
• DynamoDB throttling with low table consumption means a hot partition key; items cap at 400 KB (large payloads go to S3); DAX caches only eventually consistent reads.
• MemoryDB acknowledges writes only after its Multi-AZ transaction log commits them; ElastiCache replicates asynchronously, so it is a cache, not a system of record.
• HNSW suits changing data with high recall; IVF builds fast but drifts as data grows; probes and ef_search trade latency for recall at query time.
• Iceberg time travel is read-only, rollback changes the table for everyone; a DELETE only hides rows until OPTIMIZE and VACUUM purge them, and VACUUM at defaults removes history older than a few days.
• An idle open transaction holding a lock needs PG_TERMINATE_BACKEND (cancel only stops running queries); keep DDL transactions short, and prevent PostgreSQL idle blockers with idle_in_transaction_session_timeout.
• Partners pushing files: a Transfer Family server endpoint. Pulling from a partner-hosted SFTP server: a Transfer Family SFTP connector.

AWS Glue Data Catalog: Crawlers, Partitions, and the Hive Metastore

• The Glue Data Catalog is the single metadata source Athena, Redshift Spectrum, EMR, and Glue jobs all share — register a table once, query it at its source from every engine; Athena reaches DynamoDB and other non-S3 stores through Federated Query connectors.
• Redshift references lake tables through an external schema (FROM DATA CATALOG or FROM HIVE METASTORE), and its IAM role needs both catalog and S3 read access; datashares can only include lake tables already defined that way.
• On EMR, hive-site configures Hive and spark-hive-site configures Spark to use the Glue Data Catalog; a cross-account catalog needs hive.metastore.glue.catalogid plus a resource policy in the owning account. Hive transactions and Hive authorization are not supported with Glue.
• Crawlers infer schemas with classifiers (custom classifiers for unknown formats; a custom CSV classifier with Has headings for all-text headers) and group similar schemas under one include path into one partitioned table.
• Crawl new sub-folders only skips schema changes and changed objects; S3 event mode crawls only what changed. Update behavior LOG or Add new columns only protects curated schemas while partitions keep being added; delete behavior can delete, deprecate or just log.
• New S3 partitions are invisible until cataloged: crawlers, MSCK REPAIR TABLE (Hive-style only), ALTER TABLE ADD PARTITION (with LOCATION for non-Hive layouts), or a Glue job with enableUpdateCatalog and partitionKeys.
• Partition projection is Athena-only, ignores catalog partitions, and needs storage.location.template for non-Hive paths; partition indexes need the first key in the filter, allow three per table, and need partition_filtering.enabled in Athena.
• Glue connections store endpoint, credentials (Secrets Manager — the role needs GetSecretValue and kms:Decrypt), and VPC placement; keep the self-referencing rule, the database's inbound rule, and an S3 gateway endpoint straight.
• A technical catalog answers how engines read data; a business catalog (Amazon SageMaker Catalog) answers what data means, drafts AI metadata for owners to review, and grants access through publish-and-subscribe approval workflows.

Managing the Data Lifecycle: S3 Storage Tiers, Expiration, and Redshift

• COPY loads in parallel per slice: split data into roughly equal files whose count is a multiple of the slice count, because a single gzip file cannot be split.
• A COPY path is a key prefix; use a manifest to load an exact file list, and mark entries "mandatory": true if a missing file must fail the load. Auto-copy (COPY JOB … AUTO ON) loads new files continuously without reloading old ones.
• UNLOAD writes several files in parallel by default; PARALLEL OFF writes one file up to 6.2 GB, while MAXFILESIZE caps size, not count.
• IA classes have 30-day minimums and Glacier Instant Retrieval 90 days; Glacier IR is cheaper to store but costlier to read than Standard-IA. Deep Archive has no Expedited tier, and Bulk from Flexible Retrieval has no per-GB fee.
• Lifecycle filters combined in And are an intersection; transitions need 30 days into IA and by default skip objects under 128 KB, and tiny objects are a poor fit for the archive classes.
• On versioned buckets, Expiration only adds a delete marker — pair it with NoncurrentVersionExpiration, and clean up expired delete markers and incomplete multipart uploads.
• DynamoDB TTL needs a Number attribute in epoch seconds, uses no write capacity, deletes within days, and expired items stay readable until then; archive TTL deletes by filtering stream REMOVE records on the dynamodb.amazonaws.com service identity with an OLD_IMAGE view.
• Complete deletion means every S3 version, VACUUM in Redshift, OPTIMIZE plus VACUUM for Iceberg, and the copies in snapshots; Object Lock compliance mode cannot be shortened or bypassed, governance mode can with s3:BypassGovernanceRetention.
• Restores of DynamoDB tables create new tables; Redshift automated snapshots die with the cluster while manual snapshots persist; cross-Region snapshot copy of KMS-encrypted clusters needs a snapshot copy grant.

Data Modeling on AWS: Redshift, DynamoDB, Lake Formation & Schema Evolution

• Redshift tables have one DISTKEY: collocate the largest, most frequent join and use ALL for small dimensions; a skewed DISTKEY needs a new key or a deep copy, because VACUUM and ANALYZE never move rows between slices.
• Star schemas (denormalized dimensions) beat snowflakes for BI; SCD type 1 overwrites, type 2 adds effective-dated rows with surrogate keys, type 3 keeps a previous-value column; Redshift PK/FK constraints are informational only.
• DynamoDB partitions cap at 3,000 RCU and 1,000 WCU even in on-demand mode: shard hot keys with suffixes, keep items under 400 KB, use key conditions not filters, and remember LSIs exist only from table creation.
• Partition lake tables by low-cardinality filter columns, bucket high-cardinality ones, and use Athena partition projection for constantly growing partition sets; it is Athena-only.
• Use columnar Parquet or ORC (Avro, JSON and CSV stay row-oriented), sort within partitions so row-group statistics can skip data, and fix small files with larger writes and compaction.
• Hive-style Parquet tables read by name, so renames return NULL for old files; Iceberg tracks column IDs and allows only safe widenings such as int to bigint.
• Glue Schema Registry: BACKWARD for consumers-first upgrades, FORWARD for producers-first, FULL when upgrade order is uncontrolled, *_ALL to check every previous version.
• Apply converted schemas before AWS DMS (which creates only tables, primary keys and, in some cases, unique indexes) and keep them with the Do nothing or Truncate preparation mode.
• A Bedrock knowledge base fixes its embedding model and vector store at creation; changing the model means a new knowledge base and a vector index of matching dimension.
• Choose Redshift for OLAP (large scans, joins and aggregations) and DynamoDB for OLTP (single-item reads and writes by key at low latency).

Data Operations and Support

Automate Data Processing on AWS: Step Functions, MWAA, Lambda & EventBridge

• Step Functions Standard is exactly-once, priced per state transition and keeps an execution history; Express is at-least-once when started asynchronously, priced per request and duration, and logs only to CloudWatch Logs.
• A provisioned MWAA environment bills every hour it exists, idle or not; Step Functions and MWAA Serverless charge nothing while no work runs.
• EventBridge triggers and schedules work but never sequences it: multi-step logic with failure handling belongs in Step Functions, MWAA, or a Glue workflow.
• MWAA: DAG import errors are in DAGProcessing, pip output in the scheduler log group; queued tasks at maximum workers need a higher maximum, scheduled tasks with idle workers point to an Airflow concurrency limit.
• requirements.txt and plugins.zip are pinned by S3 object version — update the environment to apply an upload; with no internet access, ship wheels in plugins.zip.
• Step Functions payloads cap at 256 KiB (States.DataLimitExceeded), so pass S3 keys. Retry handles transient errors inside an execution; redrive resumes a failed Standard execution from the failed step.
• SDK habits: paginators (S3 lists 1,000 keys per call), retry modes (only adaptive throttles the client), resubmitting UnprocessedItems, Glue arguments prefixed with --, and the Redshift Data API instead of JDBC.
• Use native service automation first: Glue bookmarks, EMR steps with ActionOnFailure and transient clusters, Redshift stored procedures, scheduled queries and auto-copy.
• Consume APIs with Retry-After, jittered backoff, deterministic output keys and a checkpointed cursor; maintain them with REST API usage plans, caching and a new version for breaking changes.
• EventBridge Scheduler adds time zones, one-time schedules, a retry policy and a DLQ; Pipes connects streams and queues to targets; an archive replays only events received after it was created.
• Prepare data with DataBrew for no-code recipes, profile jobs and data quality rules; use SageMaker Unified Studio as the governed project workspace where prepared data is shared alongside queries, notebooks and ML.

Analyzing Data on AWS: Athena, Redshift SQL, QuickSight & Spark Notebooks

• Athena is serverless, pay-per-data-scanned SQL over S3 for ad hoc and spiky workloads; Redshift earns its cost on frequent, concurrent, latency-sensitive queries. Redshift Serverless can query S3 through Spectrum but bills RPU-hours, not bytes scanned.
• WHERE filters rows before grouping; HAVING filters groups after. ROLLUP, CUBE, and GROUPING SETS produce subtotals in one query, and GROUPING(col) = 1 marks a subtotal row even when the data has real NULLs.
• A rolling average needs an explicit ROWS frame, and a ROWS frame counts rows, not days: fill missing days from a calendar and COALESCE them to 0, because AVG ignores NULLs.
• ROW_NUMBER gives exactly N rows per group and deduplicates to one row per key; RANK skips after ties; DENSE_RANK returns the top N distinct values. Filter ranks in a subquery (or Redshift QUALIFY), never in WHERE.
• Redshift has native PIVOT and UNPIVOT (UNPIVOT's IN list takes column names and skips NULLs unless INCLUDE NULLS); Athena pivots with CASE-based conditional aggregation.
• Redshift materialized views refresh incrementally only for supported constructs (inner joins, SUM/COUNT/AVG/MIN/MAX), auto refresh can trail base tables briefly, and query rewrite speeds up unchanged SQL. Athena can query Glue Data Catalog materialized views but cannot create or refresh them.
• An Athena CTAS cannot overwrite an existing table or non-empty location and writes at most 100 partitions per query; query result reuse helps only identical repeated queries. A standard Athena view does not protect columns: use Lake Formation.
• Verify with SQL (COUNT(*) - COUNT(col) for nulls, duplicate-key checks, distinct-key reconciliation) and Lambda on prefix-scoped S3 events, never writing back into the triggering prefix. DataBrew profile jobs sample 20,000 rows unless set to the full dataset.
• SPICE refreshes at most every 15 minutes (incremental, Enterprise), hourly full refresh is Enterprise only, and CreateIngestion triggers a refresh when a pipeline finishes. Athena notebooks need a Spark workgroup.
• Provisioned wins at sustained utilization with tuning control (add concurrency scaling for read bursts); serverless, including EMR Serverless for intermittent Spark, wins for intermittent and unpredictable workloads.

Monitoring Data Pipelines: CloudWatch, CloudTrail and Log Analysis

• Match the telemetry source to the question: CloudWatch metrics for performance thresholds, CloudWatch Logs for application behavior, CloudTrail for who-called-which-API audits.
• Metric filters and subscription filters act only on events ingested after they are created; use Logs Insights to analyze logs that already exist.
• Subscription filters stream to Kinesis Data Streams, Amazon Data Firehose (the managed path to S3 or OpenSearch Service) or Lambda; export tasks are one-off batches to S3.
• The Infrequent Access log class discounts ingestion, not storage; for years of retention keep a short CloudWatch retention and archive to S3.
• S3 object reads and Lambda invokes are CloudTrail data events: enable them explicitly and narrow them with advanced event selectors; an organization trail covers every member account, including new ones.
• Audit-grade CloudTrail: trail to S3, Athena with partition projection, log file integrity validation to detect tampering and S3 Object Lock compliance mode to prevent deletion. CloudTrail Lake is closed to new customers.
• EventBridge state-change rules to SNS give one alert per failure with its details; alarms evaluate aggregates. Check the rule's Region, the SNS key policy and the alarm's missing-data treatment when alerts never arrive.
• Glue: job run insights for why a run failed, per-executor heap metrics for undersized versus skewed, the Spark UI for slow stages; group small files, use Auto Scaling for deadlines and Flex for non-urgent work.
• EMR: set a LogUri at launch, use persistent application UIs for terminated clusters, and add compute as task nodes (Spot, capped core nodes in managed scaling) when YARN memory is the bottleneck.

Data Quality on AWS: Glue Data Quality, DataBrew and Skew Handling

• Memorize the quality dimensions — completeness, accuracy, consistency, validity, uniqueness, timeliness, integrity — and translate a symptom into a dimension, then into the rule that enforces it.
• The Evaluate Data Quality transform adds DataQualityRulesPass/Fail/Skip and DataQualityEvaluationResult columns; route on the result to quarantine failing rows. Dataset-level rules such as RowCount are skipped at row level.
• On ruleset failure defaults to None, so failing rules leave the job Succeeded; use Fail job without loading to target data, placed before the write, for a true gate.
• IsComplete and Completeness count nulls only — CSV blanks in String columns are empty strings; catch them with ColumnValues not in [NULL, EMPTY, WHITESPACES_ONLY]. IsPrimaryKey means complete plus unique, across one or more columns.
• Dynamic rules such as RowCount > avg(last(10)) * 0.8 run in Glue ETL only; rule recommendations draft rules to review, while Data Catalog column statistics only describe data.
• Alert on the Data Quality Evaluation Results Available event (state FAILED) or the glue.data.quality.rules.failed metric — Publish metrics to CloudWatch must stay on — not on Glue job failure, which a quarantining job never triggers.
• DataBrew rulesets are validated only by profile jobs (emitting a DataBrew Ruleset Validation Result event); recipes clean data. Projects sample up to 5,000 rows (first n, last n, random); profile jobs default to 20,000 rows unless set to the full dataset.
• Row counts miss changed values — reconcile with sums or hashes and DQDL multi-dataset rules; fix at-least-once duplicates with MERGE on a unique id and check uniqueness on the target, not just the new batch.
• Samples estimate but cannot certify: run uniqueness and completeness gates on the full batch.
• More shuffle partitions never split a hot key. Use AQE skew joins (default in Glue 4.0+), a broadcast hint above the 10 MB threshold, separate handling of null keys, and salting with two-stage aggregation for hot groupBy keys.

Data Security and Governance

Applying Authentication Mechanisms: IAM, Secrets Manager, and Networks

• Authentication proves identity; authorization decides permissions — IAM resolves every signed request to a principal before any policy is evaluated.
• Prefer IAM roles with temporary STS credentials for all workloads; long-lived access keys in code or config are almost always the wrong answer.
• IAM groups attach identity policies to their members but are never a principal: a bucket or access point policy cannot name a group, and a group cannot be assumed.
• A role in the resource account keeps the whole cross-account grant and its revocation with the owner; a resource policy naming another account also needs an allow in the caller's account. Third-party roles need a provider-issued, per-customer sts:ExternalId.
• iam:PassRole belongs to the principal passing the role, targets the passed role's ARN, and is scoped with iam:PassedToService; a CloudFormation service role lets people deploy privileged stacks without holding the privileges.
• Lambda: execution role = what the function can do; resource-based policy = who can invoke it. API Gateway: IAM auth for SigV4 callers, Cognito authorizers for their own pool's tokens, Lambda authorizers for other issuers; API keys are not authentication.
• IAM Identity Center assigns permission sets across accounts from one place; CLI SSO profiles reach permission sets, and role_arn plus source_profile chains to any other role.
• Secrets Manager: managed rotation only for service-managed secrets, a rotation Lambda function for self-managed databases; alternating users keeps the previous credentials valid, single user does not; a VPC rotation function needs paths to both Secrets Manager and the database.
• Security groups are stateful and allow-only; network ACLs are stateless subnet filters with allow and deny, and neither overrides the other.
• Gateway endpoints (S3, DynamoDB) are free; interface endpoints exist for S3, DynamoDB and most other services, use private DNS so unchanged code works, and cost per hour. Endpoint policies limit where VPC traffic can go; aws:SourceVpce limits where bucket requests come from; aws:SourceIp is absent through endpoints.
• S3 Access Points give each consumer its own policy under bucket-policy delegation; a VPC network origin is fixed at creation and rejects requests from outside that VPC.
• In SageMaker Unified Studio, domains handle sign-in, domain units delegate administration, and projects run members' work under a shared project IAM role.

Applying Authorization Mechanisms: Lake Formation, IAM, and Redshift

• IAM authorizes API actions; Lake Formation authorizes data — its grants are enforced identically by Athena, Redshift Spectrum, EMR, and Glue via vended temporary S3 credentials.
• Same-account access needs an Allow in the identity or the resource policy (KMS key policies and role trust policies must allow themselves); cross-account needs both; SCPs and permissions boundaries never grant, and any matching explicit Deny wins.
• Scope S3 object actions to bucket/prefix/* ARNs and s3:ListBucket to the bucket ARN with an s3:prefix condition; use Access Analyzer unused access findings and policy generation for evidence.
• Delegate role creation safely by allowing iam:CreateRole only with an iam:PermissionsBoundary condition naming the approved boundary.
• Secrets Manager for anything that must rotate or cross accounts — cross-account reads need the secret's resource policy and a customer managed KMS key, never aws/secretsmanager.
• In Redshift, GRANT ON ALL TABLES covers existing tables; ALTER DEFAULT PRIVILEGES FOR USER the creator covers future ones. RLS is deny-by-default once on; IGNORE RLS exempts a role.
• Lake Formation grants do nothing while IAMAllowedPrincipals holds Super — revoke it, clear the default settings including the per-database option, and remove direct S3 access.
• Cross-account sharing: producer grants to the account via AWS RAM, consumer accepts, creates a resource link, and grants DESCRIBE on the link plus SELECT on its target.
• LF-Tags scale lake grants by sensitivity; a table wildcard covers new tables but ignores sensitivity. Data filters combine row and column restrictions for cell-level security. ABAC matches aws:ResourceTag to ${aws:PrincipalTag}.

Data Encryption and Masking on AWS: KMS, TLS, and Anonymization

• KMS uses envelope encryption: data keys encrypt the data, the KMS key encrypts the data keys. Encrypt takes at most 4,096 bytes; GenerateDataKeyWithoutPlaintext suits components that only store keys for later. Rotation re-encrypts no data.
• AWS managed keys (aws/s3, aws/rds, aws/redshift) can never be used across accounts — share encrypted data, RDS snapshots or Redshift snapshots with a customer managed key; an aws/rds snapshot must be copied under one first.
• Cross-account key use needs the key policy in the key's account plus IAM in the caller's account, naming the key ARN; aliases, permissions boundaries and bucket policies grant no key access.
• kms:ViaService restricts a key to a service and blocks direct calls; an encryption-context condition does not, because the caller supplies the context. Grants are removed with RetireGrant or RevokeGrant.
• S3 applies one SSE mode per object: DSSE-KMS is the only native two-layer option; SSE-C keeps keys out of AWS storage (and is blocked by default on new buckets since April 2026); imported key material is still stored in KMS.
• Default encryption is forward-only: UpdateObjectEncryption switches SSE-S3 objects to SSE-KMS in place; unencrypted, SSE-C and DSSE-KMS objects need an S3 Batch Operations copy.
• Enforce TLS for everyone with an aws:SecureTransport deny in the bucket policy and require_ssl on Redshift (default on new clusters since January 2025); client sslmode protects one client only.
• EMR security configurations cover at-rest and node-to-node TLS; Glue security configurations encrypt S3 output, CloudWatch Logs (key policy must allow the Logs principal) and job bookmarks with no script change.
• Redshift dynamic data masking shows transformed values, column GRANTs hide columns, Lake Formation filters restrict columns and rows over one copy, and Glue Detect Sensitive Data redacts or hashes PII before it lands.
• Tokens, encryption and deterministic hashes are pseudonymization — still personal data under GDPR; anonymization also removes identifiers and generalizes quasi-identifiers.

Preparing Logs for Audit: CloudTrail, CloudTrail Lake, and CloudWatch Logs

• Event history keeps 90 days of management events with single-attribute lookups and no SQL; data events (S3 object reads/writes, Lambda invocations) must be enabled on a trail or event data store in advance, at extra cost.
• CloudTrail Insights baselines API call and error rates (management events, and data events on trails) to flag unusual activity without custom rules.
• An organization trail logs every current and future member account to one bucket members cannot tamper with; keep that bucket in a dedicated Log Archive account, separate from the Security Tooling and management accounts.
• Log file validation delivers hourly signed digests: it covers only files delivered after it was enabled, lags by up to an hour, and reports deleted files as INVALID: not found.
• S3 Object Lock compliance mode is true WORM; governance mode can be bypassed with a permission and MFA Delete only adds a hurdle. Object Lock can be enabled on an existing versioned bucket but never disabled.
• For alerts within seconds on a specific API call, use an EventBridge rule on 'AWS API Call via CloudTrail' events (needs a trail) or a trail to CloudWatch Logs with a metric filter and alarm — not S3 log files.
• CloudWatch Logs retention deletes expired events permanently; stream a copy to S3 with a subscription filter and Firehose. Subscription filters target Kinesis Data Streams, Firehose, or Lambda; cross-account destinations must sit in the log group's Region.
• Athena and Logs Insights bill per data scanned, OpenSearch per capacity; Redshift Spectrum needs a running cluster; EMR (with bootstrap actions for node setup) handles very large batch log processing. CloudTrail Lake is closed to new customers since 31 May 2026.

Data Privacy and Governance: Macie, Redshift Sharing, Config, and Sovereignty

• Amazon Macie discovers sensitive data in S3 only (managed and custom identifiers, policy findings, EventBridge and Security Hub CSPM output) and never masks, blocks or moves data itself.
• Custom data identifiers are immutable: refine false matches with keywords and a maximum match distance (or ignore words inside the match), then create a replacement identifier and use it in new or copied jobs.
• Automated discovery samples the estate continuously at low cost but never proves full coverage; a one-time job at 100% sampling depth does; scheduled jobs inspect only new or changed objects after their first run (which by default includes existing objects).
• Detect-tag-enforce: Macie findings drive automation that overrides inherited LF-tags on sensitive columns; analysts are granted on the default public tag, because tag expressions cannot negate — and Athena enforces it only for registered locations with IAMAllowedPrincipals removed.
• Redshift datashares give live, zero-copy access on the consumer's own (and consumer-billed) compute; both sides need RA3 or Serverless; cross-account needs encryption on both sides, producer authorization and consumer association plus CREATE DATABASE FROM DATASHARE.
• A Region-deny SCP on aws:RequestedRegion is the preventive guardrail for member accounts (never the management account); separately deny replication and copy actions such as s3:PutReplicationConfiguration, which an allowed-Region call can aim abroad.
• CloudTrail event history shows who made a management call in the last 90 days without any trail; the AWS Config timeline shows what the configuration was before and after; Config rules with SSM Automation remediation fix drift without custom code, and an aggregator gives one organization-wide view.
• Sovereignty means the data itself stays in approved Regions — in-country keys, CloudFront geo restriction or a foreign DR replica do not satisfy it; stack SCPs, regional keys and Config recording in every Region.
• SageMaker Catalog access is a subscription request approved by the publishing owner; know the sharing patterns, and that zero-ETL, shared snapshots and replication are copy-based while Lake Formation grants and datashares are in place.