DEA-C01 cheat sheet

Data Ingestion and Transformation

Data Ingestion on AWS: Kinesis, MSK, DMS, Glue, and Batch Patterns

• Kinesis Data Streams for AWS-native streaming; MSK only when the scenario requires Apache Kafka compatibility.
• A Kinesis shard writes 1 MB/s or 1,000 records/s and shares 2 MB/s of reads; enhanced fan-out gives each consumer a dedicated 2 MB/s pipe.
• Firehose delivers and SQS deletes — only retained streams (Kinesis, MSK) and raw data in S3 make a pipeline replayable.
• Bulk-load Redshift with parallel COPY from multiple compressed files, never row-by-row INSERTs.
• Glue job bookmarks make batch runs incremental; S3 Event Notifications or EventBridge make them event-driven instead of polled.
• A failing Kinesis-to-Lambda batch blocks its shard: enable bisect-on-error, cap retries, and set an SQS on-failure destination.
• DynamoDB hot-partition throttling is fixed by key design, backoff, or on-demand mode — not by adding more total capacity.
• Stateless transforms fit Lambda; windows, joins, and sessionization are stateful and need Flink or Spark with checkpointed state.

Transform and Process Data on AWS: Glue, EMR, Lambda, and Redshift

• Glue for serverless Spark ETL, EMR for scale and framework control, Lambda for sub-15-minute event-driven transforms, Redshift for SQL ELT in the warehouse.
• Convert CSV/JSON to compressed, partitioned Parquet — Athena and Redshift Spectrum bill by bytes scanned, so columnar format is a direct cost lever.
• Cheapest compute for loose SLAs: Glue Flex, EMR Serverless, and Spot — but keep EMR core nodes On-Demand; Spot belongs on task nodes.
• Stragglers where one task runs forever mean data skew: salt the hot key or broadcast the small join side — more executors won't help.
• Executor OOM wants bigger workers or more partitions; driver OOM means a collect() pulled data onto the driver.
• Compact small files toward 128 MB–1 GB — thousands of tiny objects make Spark pay per-file overhead instead of processing data.
• The Redshift Data API runs SQL over HTTPS with IAM auth and no persistent connections — the right warehouse access path from Lambda and Step Functions.
• Use Amazon Bedrock LLMs to extract, classify, and summarize unstructured data as a pipeline stage; batch calls and validate outputs against a schema.

Orchestrating Data Pipelines on AWS: Step Functions, MWAA, and Glue Workflows

• AWS Step Functions is the default serverless orchestrator: direct integrations with Glue, EMR, Lambda and more, per-state Retry/Catch, and no infrastructure — Standard for long-running ETL, Express for short high-volume work.
• Amazon MWAA is for Apache Airflow: Python DAGs, the operator ecosystem, and migrations of existing Airflow deployments — but its environment bills while idle, so it is never the answer to a cost or least-overhead question for simple pipelines.
• AWS Glue workflows orchestrate only Glue crawlers and Glue jobs; the moment any non-Glue step appears, Step Functions takes over as the orchestrator.
• Amazon EventBridge supplies both trigger styles: cron/rate schedules and event patterns such as S3 Object Created — event-driven wins whenever data arrival times vary.
• Fault tolerance is layered: retries with exponential backoff for transient errors, idempotent steps so replays are safe, SQS with dead-letter queues to buffer and isolate failures, and Catch branches to contain errors.
• Amazon SNS pushes one message to many subscribers (alerts, fan-out); Amazon SQS queues work for one consumer (decoupling, buffering) — SNS-to-multiple-SQS is the fan-out pattern.
• Chaining Lambda functions into each other for orchestration is an anti-pattern; the fix is a Step Functions state machine that provides central state, retries, and visibility.

Programming Concepts for AWS Data Engineering: IaC, CI/CD, and Lambda Tuning

• Reduce runtime by processing less data first: Parquet over CSV, partitioned S3 layouts, predicate pushdown, and fixing the small-files problem — before adding workers.
• Lambda memory scales CPU and network proportionally; reserved concurrency caps a function to protect downstream systems, and provisioned concurrency pre-warms environments for latency-sensitive work.
• Long or heavy transformations belong in Glue or EMR, not Lambda — Lambda's execution time is capped at minutes; mount Amazon EFS when a function needs more or shared file storage.
• CloudFormation, CDK, and SAM all deploy via CloudFormation: templates are the declarative base, CDK authors infrastructure in real programming languages, and SAM is the shorthand plus local-testing toolkit for serverless pipelines.
• CI/CD for data pipelines versions code and infrastructure together, runs unit and data-quality tests automatically, deploys through parameterized identical environments, and replaces every manual console release.
• In distributed computing, shuffles and data skew — not raw data size — are the usual bottlenecks; small datasets are faster on a single node than a cluster.
• Python and SQL are the default data-engineering languages on AWS; Scala/Java serve JVM-native Spark performance, and Bash scripts the CLI for operational glue.
• Monitor pipelines with structured logs in CloudWatch, alarms on failure and throttle metrics publishing to SNS, and Glue Data Quality rules that fail jobs on bad data.

Data Store Management

Choosing an AWS Data Store: Redshift, DynamoDB, Aurora, S3, and MemoryDB

• Read the access pattern first: analytics → Redshift or S3+Athena, OLTP → RDS/Aurora, key/value at scale → DynamoDB, microsecond in-memory → MemoryDB, replayable events → Kinesis/MSK.
• S3 + Athena is the cost answer for infrequently queried data; an always-on Redshift cluster for occasional queries is the classic trap.
• Redshift: KEY distribution co-locates joins, ALL suits small dimensions, sort keys prune scans; RA3 and Serverless decouple or eliminate cluster cost.
• Spectrum queries S3 in place, federated queries reach live RDS/Aurora data, materialized views precompute repeated aggregations — none of them copy data by hand.
• MemoryDB is a durable in-memory primary database, not just a cache; HNSW indexes favor recall and query speed, IVF favors cheap builds on already-loaded data.
• Apache Iceberg adds ACID transactions, row-level deletes, schema evolution, and time travel to S3 tables — and needs compaction and snapshot expiry as ongoing maintenance.
• Blocked Redshift queries usually mean an uncommitted transaction holding an AccessExclusiveLock: find it in the system views, terminate it with PG_TERMINATE_BACKEND.
• AWS Transfer Family gives partners a managed SFTP endpoint that drops files straight into S3, where events trigger the rest of the pipeline.

AWS Glue Data Catalog: Crawlers, Partitions, and the Hive Metastore

• The Glue Data Catalog is the single metadata source Athena, Redshift Spectrum, EMR, and Glue jobs all share — register a table once, query it at its source from every engine.
• The Glue Data Catalog is a managed, serverless, Hive metastore-compatible catalog; on EMR it is the least-overhead persistent metastore choice and the one shared with Athena.
• Crawlers infer schemas via classifiers, detect partition layouts, and populate the catalog; schema-change policy, scheduling, and folder-level targeting are the settings exams probe.
• New S3 partitions are invisible to queries until cataloged: use crawlers for the general case, ALTER TABLE ADD PARTITION for pipeline-driven precision, MSCK REPAIR TABLE for Hive-style backfills, and partition projection to skip partition metadata entirely.
• Glue connections store endpoint, credentials (prefer Secrets Manager), and VPC placement; jobs reach VPC databases through ENIs, so security-group rules and an S3 endpoint or NAT route are the usual failure points.
• A technical catalog answers how engines read data; a business catalog (Amazon SageMaker Catalog) answers what data means and grants access through publish-and-subscribe approval workflows.
• Lake Formation layers database-, table-, column-, and row-level permissions on catalog resources, and cross-account sharing exposes tables without copying data.

Managing the Data Lifecycle: S3 Storage Tiers, Expiration, and Redshift

• Use Redshift COPY to load from S3 in parallel (split, compressed files, manifest, IAM role) and UNLOAD to export — never row-by-row INSERTs for bulk loads.
• Pick storage classes by access frequency and retrieval need: Standard for hot data, Intelligent-Tiering for unknown patterns, Standard-IA for monthly access, One Zone-IA only for re-creatable data, and the Glacier tiers for archives.
• S3 Lifecycle rules are the answer whenever the trigger is object age — transitions move data down the cost hierarchy and expiration actions delete it, with no Lambda or cron required.
• On versioned buckets, current-version expiration only adds a delete marker; you need noncurrent-version expiration to actually reclaim storage.
• DynamoDB TTL deletes expired items for free but not instantly — filter expired items in queries, and archive them via Streams + Lambda if you need history.
• A legal deletion in a versioned bucket means deleting every version; S3 Batch Operations handles scale, and Object Lock compliance mode makes deletion impossible until retention ends.
• IA and Glacier classes carry minimum storage durations — short-lived objects can cost more there than in Standard.
• Match resiliency tools to the failure mode: versioning/PITR for mistakes, multi-AZ for AZ loss, CRR and cross-Region snapshots for Region loss.

Data Modeling on AWS: Redshift, DynamoDB, Lake Formation & Schema Evolution

• Redshift distribution styles: KEY for collocated joins on high-cardinality columns, ALL for small dimensions, EVEN for large unjoined tables, AUTO when unsure — and skewed DISTKEYs are the classic anti-pattern.
• Compound sort keys (usually leading with date) drive Redshift block pruning; interleaved keys are rarely the right modern answer.
• DynamoDB is modeled access-patterns-first: high-cardinality partition keys, sort keys for ranges, GSIs for alternate patterns, single-table design to precompute joins — hot partitions and Scans signal a broken model.
• Redshift is OLAP (flexible SQL over columnar history); DynamoDB is OLTP (millisecond key lookups) — questions that mix the workloads are testing store choice, not tuning.
• Lake Formation schemas live in the Glue Data Catalog over partitioned Parquet in S3; Apache Iceberg adds in-place schema and partition evolution without rewriting files.
• Evolve schemas additively: nullable new columns, Glue Schema Registry compatibility checks for streams, ALTER TABLE ADD COLUMN in Redshift — but DynamoDB key schemas cannot change after creation.
• AWS DMS Schema Conversion (and AWS SCT) convert structure for heterogeneous migrations and produce an assessment report; AWS DMS then moves the data with full load plus CDC.
• Vectorization converts data to embeddings for similarity search; a Bedrock knowledge base packages the chunk-embed-store-retrieve RAG pipeline over documents in S3.

Data Operations and Support

Automate Data Processing on AWS: Step Functions, MWAA, Lambda & EventBridge

• Step Functions is the serverless default orchestrator: native integrations with Glue, EMR, Athena, and Lambda, per-state Retry/Catch, and a run-a-job pattern that waits for completion.
• Choose Amazon MWAA when the scenario says complex DAGs, backfills, existing Airflow expertise, or orchestration beyond AWS — and remember its environment is always on and always billed.
• EventBridge triggers and schedules work but never sequences it: multi-step logic with failure handling belongs in Step Functions, MWAA, or a Glue workflow.
• Troubleshoot MWAA through its CloudWatch log groups (scheduler for DAG parse errors, workers for queued tasks, requirements.txt for dependency failures) and Step Functions through its per-state execution history.
• Pass S3 object keys, not large payloads, between Step Functions states; use SDK paginators, built-in retries, and the Redshift Data API instead of hand-rolled loops and JDBC connections.
• Use native service automation first: Glue bookmarks and triggers, EMR transient clusters and EMR Serverless, Redshift stored procedures, scheduled queries, and auto-copy.
• DataBrew is no-code preparation (recipes, profile jobs, data quality rules); SageMaker Unified Studio is the governed project workspace spanning preparation, querying, and ML.
• A data API you maintain needs throttling, caching, and versioning; a data API you consume needs pagination, backoff on rate limits, and idempotent ingestion.

Analyzing Data on AWS: Athena, Redshift SQL, QuickSight & Spark Notebooks

• Athena is serverless, pay-per-data-scanned SQL over S3 for ad hoc and spiky workloads; Redshift earns its cost on frequent, concurrent, latency-sensitive queries — Spectrum bridges warehouse SQL to the lake.
• WHERE filters rows before grouping; HAVING filters aggregated groups after — and ROLLUP, CUBE, and GROUPING SETS produce multi-level subtotals in one query.
• A rolling average is a window function with an explicit frame (ROWS BETWEEN n PRECEDING AND CURRENT ROW); ORDER BY without a frame yields a cumulative running aggregate instead.
• Redshift has native PIVOT/UNPIVOT; Athena pivots via CASE-based conditional aggregation; QuickSight pivot tables rotate dimensions interactively without SQL.
• Both engines support logical views, but materialized views (with auto refresh and query rewrite) are Redshift-only — Athena's nearest equivalent is a scheduled CTAS snapshot.
• Automate verification with Lambda (in-flight checks) and Athena SQL (counts, nulls, duplicates); explore issues in notebooks and SageMaker Data Wrangler; confirm visually in QuickSight.
• QuickSight SPICE serves many viewers fast from memory on a refresh schedule; direct query trades speed for freshness. DataBrew profiles visualize data quality, not business KPIs.
• Provisioned services win at sustained high utilization with tuning control; serverless wins for intermittent and unpredictable workloads with zero idle cost.

Monitoring Data Pipelines: CloudWatch, CloudTrail and Log Analysis

• Match the telemetry source to the question: CloudWatch metrics for performance thresholds, CloudWatch Logs for application behavior, CloudTrail for who-called-which-API audits.
• CloudWatch Logs Insights queries log groups in place with no infrastructure; OpenSearch Service requires an ingestion pipeline and a domain but delivers real-time full-text search and dashboards.
• CloudTrail management events are on by default (90-day history); create a trail to S3 for long-term retention, enable data events for S3 object-level audits, and query trails with Athena.
• For pipeline failure alerts, an EventBridge rule on Glue or EMR state-change events targeting an SNS topic beats polling; use CloudWatch alarms (plus metric filters) for threshold-based conditions.
• Glue troubleshooting order: job metrics for the shape, driver and executor logs for the exception, Spark UI for the slow stage; watch for small files, data skew, bookmarks and worker sizing.
• Always configure EMR logging to S3 at launch — on-cluster logs disappear at termination; keep core nodes (HDFS) on On-Demand and scale task nodes for elasticity.
• Set explicit retention on CloudWatch log groups — the default is never-expire — and use metric filters to alarm on log patterns that have no native metric.
• Analyze logs with the cheapest tool that meets latency needs: Logs Insights (ad hoc, in place), Athena (SQL over S3 archives), EMR (heavy parsing), OpenSearch (continuous operational search).

Data Quality on AWS: Glue Data Quality, DataBrew and Skew Handling

• Memorize the quality dimensions — completeness, accuracy, consistency, validity, uniqueness, timeliness, integrity — and translate scenario symptoms into a dimension, then into the rule that enforces it.
• Run checks while processing: detect empty fields in-pipeline and choose fail fast, quarantine to a separate S3 path, or tag-and-continue based on how tolerant downstream consumers are.
• AWS Glue Data Quality evaluates DQDL rulesets (IsComplete, ColumnValues, IsUnique, RowCount, Completeness thresholds) against Data Catalog tables or inside ETL jobs, where it can route passing and failing rows separately.
• AWS Glue DataBrew profiles datasets (missing values, duplicates, distributions, outliers) and validates its data quality rules during profile jobs; recipes then fix the defects without code.
• Investigate consistency hop by hop with row-count reconciliation and aggregate checksums; watch for at-least-once duplicates and timing mismatches that manufacture phantom differences.
• Choose sampling by bias profile: random for unbiased estimates, stratified to guarantee rare segments, systematic with caution around periodic data, first-n only for quick looks — and never certify uniqueness from a sample.
• Skew shows up as one straggler task or executor OOM; fix it with salting, broadcast joins, repartitioning, Spark AQE, or isolating hot keys — and fix the upstream quality defect that created the hot key.
• Publish quality outcomes as metrics and events so EventBridge and Amazon SNS can alert when failure rates breach thresholds — silent checks protect nobody.

Data Security and Governance

Applying Authentication Mechanisms: IAM, Secrets Manager, and Networks

• Authentication proves identity; authorization decides permissions — IAM resolves every signed request to a principal before any policy is evaluated.
• Prefer IAM roles with temporary STS credentials for all workloads; long-lived access keys in code or config are almost always the wrong answer.
• IAM groups organize user permissions but cannot authenticate or be assumed; roles have trust policies that define exactly who may assume them.
• AWS Secrets Manager stores and automatically rotates database and API credentials (managed rotation or a rotation Lambda); Parameter Store stores configuration but does not rotate.
• Security groups are stateful and allow-only; reference another security group instead of IP ranges to grant compute-to-database access that survives IP changes.
• Gateway endpoints (S3, DynamoDB) and PrivateLink interface endpoints keep traffic private, and endpoint policies apply IAM controls to the network path itself.
• S3 Access Points give each team or application its own policy and can be locked to a VPC — the scalable alternative to one giant bucket policy.
• In SageMaker Unified Studio, domains handle sign-in, domain units delegate administration, and projects run members' work under a shared project IAM role.

Applying Authorization Mechanisms: Lake Formation, IAM, and Redshift

• IAM authorizes API actions; Lake Formation authorizes data — its grants are enforced identically by Athena, Redshift Spectrum, EMR, and Glue via vended temporary S3 credentials.
• Write customer managed IAM policies when managed policies are too broad; explicit Deny always overrides any Allow, making it the tool for hard guardrails.
• Build least privilege iteratively: exact ARNs, condition keys, IAM Access Analyzer policy generation from CloudTrail activity, and permissions boundaries as a ceiling.
• Secrets Manager for anything that must rotate or cross accounts; Parameter Store for cheap configuration and SecureString settings without rotation.
• Amazon Redshift layers database users, groups, and RBAC roles under IAM, with IAM-issued temporary database credentials replacing password management for analysts.
• LF-Tags scale lake authorization: grant on tag expressions, let new tables inherit database tags, and stop writing per-resource grants.
• Data filters deliver row-level and, combined with column lists, cell-level security — different principals see different slices of the same table without maintaining views.
• Choose RBAC for stable job functions, tag-based (LF-TBAC) for growing lakes, and ABAC (principal-tag to resource-tag matching) when one policy must serve many teams.

Data Encryption and Masking on AWS: KMS, TLS, and Anonymization

• KMS uses envelope encryption: data keys encrypt the data locally, the KMS key encrypts the data keys, and disabling the KMS key instantly revokes access without re-encrypting anything.
• AWS managed keys (aws/s3, aws/redshift) can never be shared across accounts — any cross-account encrypted-data scenario requires a customer managed key.
• Cross-account access to KMS-encrypted data needs both a key policy statement in the key's account and IAM permissions in the consumer account; either one alone fails.
• Enforce encryption in transit by denying requests where aws:SecureTransport is false (S3) and by setting require_ssl for Redshift; EMR in-transit encryption uses a security configuration with TLS certificates.
• S3 encrypts new objects with SSE-S3 by default; use SSE-KMS for auditable key control, S3 Bucket Keys to cut KMS costs, and client-side encryption when AWS must never see plaintext.
• DynamoDB is always encrypted at rest — the only decision is AWS owned (default), AWS managed, or customer managed key.
• Lake Formation data filters give column-, row-, and cell-level masking at query time; Redshift dynamic data masking does the same inside the warehouse; Macie finds the PII that needs protecting.
• Tokenization is reversible via a vault and preserves joins; hashing is one-way and deterministic; anonymization is irreversible and can take data out of GDPR scope.

Preparing Logs for Audit: CloudTrail, CloudTrail Lake, and CloudWatch Logs

• CloudTrail management events cover control-plane actions and are on by default (90 days in Event history); data events like S3 GetObject and Lambda invocations must be explicitly enabled on a trail or event data store, in advance, at extra cost.
• An organization trail logs every member account — including future ones — to one bucket that member accounts cannot tamper with; it is the default answer for centralized multi-account audit capture.
• CloudTrail log file validation delivers signed digest files that prove whether any log file was changed or deleted; pair it with S3 Object Lock in compliance mode for WORM immutability.
• CloudWatch Logs stores application logs with per-log-group retention; metric filters turn log patterns into alarms, and subscription filters stream events to Lambda, Kinesis, Firehose, or OpenSearch in near real time.
• CloudTrail Lake gives immutable, multi-year, SQL-queryable audit storage with zero pipeline setup — choose it over trail-plus-Athena when minimal setup or non-AWS audit events are required.
• Use CloudWatch Logs Insights for interactive queries on logs still in CloudWatch; use Athena (with partitioning and compression) for logs archived in S3; use OpenSearch for full-text search and live dashboards.
• Amazon EMR is the answer for batch-processing very large log volumes — enriching or converting archives to Parquet — not for interactive investigation.
• Export long-lived application logs from CloudWatch Logs to S3 via Firehose and tier with lifecycle policies; keeping years of logs hot in CloudWatch or OpenSearch is a cost anti-pattern.

Data Privacy and Governance: Macie, Redshift Sharing, Config, and Sovereignty

• Amazon Macie discovers sensitive data in S3 (managed and custom data identifiers, automated discovery, policy findings) but never enforces access — enforcement is Lake Formation's job.
• The detect-tag-enforce loop is the tested pattern: Macie findings trigger EventBridge automation that applies Lake Formation LF-tags, and tag-based access control restricts every table or column carrying the pii tag.
• Redshift datashares give live, read-only, zero-copy access using the consumer's own compute; cross-account sharing needs producer authorization plus consumer association, and RA3 or Serverless is required.
• Block disallowed Regions preventively with an SCP on aws:RequestedRegion, and separately deny the replication and copy actions (S3 CRR, DynamoDB global tables, snapshot and backup copies) that move data across Regions without anyone calling an API there.
• AWS Config records configuration history per resource, evaluates compliance with rules, auto-remediates via Systems Manager, and aggregates findings organization-wide — CloudTrail says who called; Config says what the configuration became.
• Data sovereignty stacks three layers: Region-deny SCPs (preventive), single-Region customer managed KMS keys (data-layer), and Config aggregators (detective).
• SageMaker Catalog projects add a publish-subscribe-approve workflow over Lake Formation and Redshift grants; Glue Data Catalog remains the technical metastore underneath.
• Know the sharing patterns by signal phrase: centralized lake (one governing team), data mesh (domain-owned data products), datashares (zero-copy warehouse sharing), Data Exchange (licensed third-party data).