DEA-C01 flashcards

Data Ingestion and Transformation

• Kinesis Data Streams for AWS-native streaming; MSK only when the scenario requires Apache Kafka compatibility.
• A Kinesis shard writes 1 MB/s or 1,000 records/s and shares 2 MB/s of reads; enhanced fan-out gives each consumer a dedicated 2 MB/s pipe.
• Firehose delivers and SQS deletes — only retained streams (Kinesis, MSK) and raw data in S3 make a pipeline replayable.
• Bulk-load Redshift with parallel COPY from multiple compressed files, never row-by-row INSERTs.
• Glue job bookmarks make batch runs incremental; S3 Event Notifications or EventBridge make them event-driven instead of polled.
• A failing Kinesis-to-Lambda batch blocks its shard: enable bisect-on-error, cap retries, and set an SQS on-failure destination.
• DynamoDB hot-partition throttling is fixed by key design, backoff, or on-demand mode — not by adding more total capacity.
• Stateless transforms fit Lambda; windows, joins, and sessionization are stateful and need Flink or Spark with checkpointed state.

• Glue for serverless Spark ETL, EMR for scale and framework control, Lambda for sub-15-minute event-driven transforms, Redshift for SQL ELT in the warehouse.
• Convert CSV/JSON to compressed, partitioned Parquet — Athena and Redshift Spectrum bill by bytes scanned, so columnar format is a direct cost lever.
• Cheapest compute for loose SLAs: Glue Flex, EMR Serverless, and Spot — but keep EMR core nodes On-Demand; Spot belongs on task nodes.
• Stragglers where one task runs forever mean data skew: salt the hot key or broadcast the small join side — more executors won't help.
• Executor OOM wants bigger workers or more partitions; driver OOM means a collect() pulled data onto the driver.
• Compact small files toward 128 MB–1 GB — thousands of tiny objects make Spark pay per-file overhead instead of processing data.
• The Redshift Data API runs SQL over HTTPS with IAM auth and no persistent connections — the right warehouse access path from Lambda and Step Functions.
• Use Amazon Bedrock LLMs to extract, classify, and summarize unstructured data as a pipeline stage; batch calls and validate outputs against a schema.

• AWS Step Functions is the default serverless orchestrator: direct integrations with Glue, EMR, Lambda and more, per-state Retry/Catch, and no infrastructure — Standard for long-running ETL, Express for short high-volume work.
• Amazon MWAA is for Apache Airflow: Python DAGs, the operator ecosystem, and migrations of existing Airflow deployments — but its environment bills while idle, so it is never the answer to a cost or least-overhead question for simple pipelines.
• AWS Glue workflows orchestrate only Glue crawlers and Glue jobs; the moment any non-Glue step appears, Step Functions takes over as the orchestrator.
• Amazon EventBridge supplies both trigger styles: cron/rate schedules and event patterns such as S3 Object Created — event-driven wins whenever data arrival times vary.
• Fault tolerance is layered: retries with exponential backoff for transient errors, idempotent steps so replays are safe, SQS with dead-letter queues to buffer and isolate failures, and Catch branches to contain errors.
• Amazon SNS pushes one message to many subscribers (alerts, fan-out); Amazon SQS queues work for one consumer (decoupling, buffering) — SNS-to-multiple-SQS is the fan-out pattern.
• Chaining Lambda functions into each other for orchestration is an anti-pattern; the fix is a Step Functions state machine that provides central state, retries, and visibility.

• Reduce runtime by processing less data first: Parquet over CSV, partitioned S3 layouts, predicate pushdown, and fixing the small-files problem — before adding workers.
• Lambda memory scales CPU and network proportionally; reserved concurrency caps a function to protect downstream systems, and provisioned concurrency pre-warms environments for latency-sensitive work.
• Long or heavy transformations belong in Glue or EMR, not Lambda — Lambda's execution time is capped at minutes; mount Amazon EFS when a function needs more or shared file storage.
• CloudFormation, CDK, and SAM all deploy via CloudFormation: templates are the declarative base, CDK authors infrastructure in real programming languages, and SAM is the shorthand plus local-testing toolkit for serverless pipelines.
• CI/CD for data pipelines versions code and infrastructure together, runs unit and data-quality tests automatically, deploys through parameterized identical environments, and replaces every manual console release.
• In distributed computing, shuffles and data skew — not raw data size — are the usual bottlenecks; small datasets are faster on a single node than a cluster.
• Python and SQL are the default data-engineering languages on AWS; Scala/Java serve JVM-native Spark performance, and Bash scripts the CLI for operational glue.
• Monitor pipelines with structured logs in CloudWatch, alarms on failure and throttle metrics publishing to SNS, and Glue Data Quality rules that fail jobs on bad data.

Data Store Management

• Read the access pattern first: analytics → Redshift or S3+Athena, OLTP → RDS/Aurora, key/value at scale → DynamoDB, microsecond in-memory → MemoryDB, replayable events → Kinesis/MSK.
• S3 + Athena is the cost answer for infrequently queried data; an always-on Redshift cluster for occasional queries is the classic trap.
• Redshift: KEY distribution co-locates joins, ALL suits small dimensions, sort keys prune scans; RA3 and Serverless decouple or eliminate cluster cost.
• Spectrum queries S3 in place, federated queries reach live RDS/Aurora data, materialized views precompute repeated aggregations — none of them copy data by hand.
• MemoryDB is a durable in-memory primary database, not just a cache; HNSW indexes favor recall and query speed, IVF favors cheap builds on already-loaded data.
• Apache Iceberg adds ACID transactions, row-level deletes, schema evolution, and time travel to S3 tables — and needs compaction and snapshot expiry as ongoing maintenance.
• Blocked Redshift queries usually mean an uncommitted transaction holding an AccessExclusiveLock: find it in the system views, terminate it with PG_TERMINATE_BACKEND.
• AWS Transfer Family gives partners a managed SFTP endpoint that drops files straight into S3, where events trigger the rest of the pipeline.

• The Glue Data Catalog is the single metadata source Athena, Redshift Spectrum, EMR, and Glue jobs all share — register a table once, query it at its source from every engine.
• The Glue Data Catalog is a managed, serverless, Hive metastore-compatible catalog; on EMR it is the least-overhead persistent metastore choice and the one shared with Athena.
• Crawlers infer schemas via classifiers, detect partition layouts, and populate the catalog; schema-change policy, scheduling, and folder-level targeting are the settings exams probe.
• New S3 partitions are invisible to queries until cataloged: use crawlers for the general case, ALTER TABLE ADD PARTITION for pipeline-driven precision, MSCK REPAIR TABLE for Hive-style backfills, and partition projection to skip partition metadata entirely.
• Glue connections store endpoint, credentials (prefer Secrets Manager), and VPC placement; jobs reach VPC databases through ENIs, so security-group rules and an S3 endpoint or NAT route are the usual failure points.
• A technical catalog answers how engines read data; a business catalog (Amazon SageMaker Catalog) answers what data means and grants access through publish-and-subscribe approval workflows.
• Lake Formation layers database-, table-, column-, and row-level permissions on catalog resources, and cross-account sharing exposes tables without copying data.

• Use Redshift COPY to load from S3 in parallel (split, compressed files, manifest, IAM role) and UNLOAD to export — never row-by-row INSERTs for bulk loads.
• Pick storage classes by access frequency and retrieval need: Standard for hot data, Intelligent-Tiering for unknown patterns, Standard-IA for monthly access, One Zone-IA only for re-creatable data, and the Glacier tiers for archives.
• S3 Lifecycle rules are the answer whenever the trigger is object age — transitions move data down the cost hierarchy and expiration actions delete it, with no Lambda or cron required.
• On versioned buckets, current-version expiration only adds a delete marker; you need noncurrent-version expiration to actually reclaim storage.
• DynamoDB TTL deletes expired items for free but not instantly — filter expired items in queries, and archive them via Streams + Lambda if you need history.
• A legal deletion in a versioned bucket means deleting every version; S3 Batch Operations handles scale, and Object Lock compliance mode makes deletion impossible until retention ends.
• IA and Glacier classes carry minimum storage durations — short-lived objects can cost more there than in Standard.
• Match resiliency tools to the failure mode: versioning/PITR for mistakes, multi-AZ for AZ loss, CRR and cross-Region snapshots for Region loss.

• Redshift distribution styles: KEY for collocated joins on high-cardinality columns, ALL for small dimensions, EVEN for large unjoined tables, AUTO when unsure — and skewed DISTKEYs are the classic anti-pattern.
• Compound sort keys (usually leading with date) drive Redshift block pruning; interleaved keys are rarely the right modern answer.
• DynamoDB is modeled access-patterns-first: high-cardinality partition keys, sort keys for ranges, GSIs for alternate patterns, single-table design to precompute joins — hot partitions and Scans signal a broken model.
• Redshift is OLAP (flexible SQL over columnar history); DynamoDB is OLTP (millisecond key lookups) — questions that mix the workloads are testing store choice, not tuning.
• Lake Formation schemas live in the Glue Data Catalog over partitioned Parquet in S3; Apache Iceberg adds in-place schema and partition evolution without rewriting files.
• Evolve schemas additively: nullable new columns, Glue Schema Registry compatibility checks for streams, ALTER TABLE ADD COLUMN in Redshift — but DynamoDB key schemas cannot change after creation.
• AWS DMS Schema Conversion (and AWS SCT) convert structure for heterogeneous migrations and produce an assessment report; AWS DMS then moves the data with full load plus CDC.
• Vectorization converts data to embeddings for similarity search; a Bedrock knowledge base packages the chunk-embed-store-retrieve RAG pipeline over documents in S3.

Data Operations and Support

• Step Functions is the serverless default orchestrator: native integrations with Glue, EMR, Athena, and Lambda, per-state Retry/Catch, and a run-a-job pattern that waits for completion.
• Choose Amazon MWAA when the scenario says complex DAGs, backfills, existing Airflow expertise, or orchestration beyond AWS — and remember its environment is always on and always billed.
• EventBridge triggers and schedules work but never sequences it: multi-step logic with failure handling belongs in Step Functions, MWAA, or a Glue workflow.
• Troubleshoot MWAA through its CloudWatch log groups (scheduler for DAG parse errors, workers for queued tasks, requirements.txt for dependency failures) and Step Functions through its per-state execution history.
• Pass S3 object keys, not large payloads, between Step Functions states; use SDK paginators, built-in retries, and the Redshift Data API instead of hand-rolled loops and JDBC connections.
• Use native service automation first: Glue bookmarks and triggers, EMR transient clusters and EMR Serverless, Redshift stored procedures, scheduled queries, and auto-copy.
• DataBrew is no-code preparation (recipes, profile jobs, data quality rules); SageMaker Unified Studio is the governed project workspace spanning preparation, querying, and ML.
• A data API you maintain needs throttling, caching, and versioning; a data API you consume needs pagination, backoff on rate limits, and idempotent ingestion.

• Athena is serverless, pay-per-data-scanned SQL over S3 for ad hoc and spiky workloads; Redshift earns its cost on frequent, concurrent, latency-sensitive queries — Spectrum bridges warehouse SQL to the lake.
• WHERE filters rows before grouping; HAVING filters aggregated groups after — and ROLLUP, CUBE, and GROUPING SETS produce multi-level subtotals in one query.
• A rolling average is a window function with an explicit frame (ROWS BETWEEN n PRECEDING AND CURRENT ROW); ORDER BY without a frame yields a cumulative running aggregate instead.
• Redshift has native PIVOT/UNPIVOT; Athena pivots via CASE-based conditional aggregation; QuickSight pivot tables rotate dimensions interactively without SQL.
• Both engines support logical views, but materialized views (with auto refresh and query rewrite) are Redshift-only — Athena's nearest equivalent is a scheduled CTAS snapshot.
• Automate verification with Lambda (in-flight checks) and Athena SQL (counts, nulls, duplicates); explore issues in notebooks and SageMaker Data Wrangler; confirm visually in QuickSight.
• QuickSight SPICE serves many viewers fast from memory on a refresh schedule; direct query trades speed for freshness. DataBrew profiles visualize data quality, not business KPIs.
• Provisioned services win at sustained high utilization with tuning control; serverless wins for intermittent and unpredictable workloads with zero idle cost.

• Match the telemetry source to the question: CloudWatch metrics for performance thresholds, CloudWatch Logs for application behavior, CloudTrail for who-called-which-API audits.
• CloudWatch Logs Insights queries log groups in place with no infrastructure; OpenSearch Service requires an ingestion pipeline and a domain but delivers real-time full-text search and dashboards.
• CloudTrail management events are on by default (90-day history); create a trail to S3 for long-term retention, enable data events for S3 object-level audits, and query trails with Athena.
• For pipeline failure alerts, an EventBridge rule on Glue or EMR state-change events targeting an SNS topic beats polling; use CloudWatch alarms (plus metric filters) for threshold-based conditions.
• Glue troubleshooting order: job metrics for the shape, driver and executor logs for the exception, Spark UI for the slow stage; watch for small files, data skew, bookmarks and worker sizing.
• Always configure EMR logging to S3 at launch — on-cluster logs disappear at termination; keep core nodes (HDFS) on On-Demand and scale task nodes for elasticity.
• Set explicit retention on CloudWatch log groups — the default is never-expire — and use metric filters to alarm on log patterns that have no native metric.
• Analyze logs with the cheapest tool that meets latency needs: Logs Insights (ad hoc, in place), Athena (SQL over S3 archives), EMR (heavy parsing), OpenSearch (continuous operational search).

• Memorize the quality dimensions — completeness, accuracy, consistency, validity, uniqueness, timeliness, integrity — and translate scenario symptoms into a dimension, then into the rule that enforces it.
• Run checks while processing: detect empty fields in-pipeline and choose fail fast, quarantine to a separate S3 path, or tag-and-continue based on how tolerant downstream consumers are.
• AWS Glue Data Quality evaluates DQDL rulesets (IsComplete, ColumnValues, IsUnique, RowCount, Completeness thresholds) against Data Catalog tables or inside ETL jobs, where it can route passing and failing rows separately.
• AWS Glue DataBrew profiles datasets (missing values, duplicates, distributions, outliers) and validates its data quality rules during profile jobs; recipes then fix the defects without code.
• Investigate consistency hop by hop with row-count reconciliation and aggregate checksums; watch for at-least-once duplicates and timing mismatches that manufacture phantom differences.
• Choose sampling by bias profile: random for unbiased estimates, stratified to guarantee rare segments, systematic with caution around periodic data, first-n only for quick looks — and never certify uniqueness from a sample.
• Skew shows up as one straggler task or executor OOM; fix it with salting, broadcast joins, repartitioning, Spark AQE, or isolating hot keys — and fix the upstream quality defect that created the hot key.
• Publish quality outcomes as metrics and events so EventBridge and Amazon SNS can alert when failure rates breach thresholds — silent checks protect nobody.

Data Security and Governance

• Authentication proves identity; authorization decides permissions — IAM resolves every signed request to a principal before any policy is evaluated.
• Prefer IAM roles with temporary STS credentials for all workloads; long-lived access keys in code or config are almost always the wrong answer.
• IAM groups organize user permissions but cannot authenticate or be assumed; roles have trust policies that define exactly who may assume them.
• AWS Secrets Manager stores and automatically rotates database and API credentials (managed rotation or a rotation Lambda); Parameter Store stores configuration but does not rotate.
• Security groups are stateful and allow-only; reference another security group instead of IP ranges to grant compute-to-database access that survives IP changes.
• Gateway endpoints (S3, DynamoDB) and PrivateLink interface endpoints keep traffic private, and endpoint policies apply IAM controls to the network path itself.
• S3 Access Points give each team or application its own policy and can be locked to a VPC — the scalable alternative to one giant bucket policy.
• In SageMaker Unified Studio, domains handle sign-in, domain units delegate administration, and projects run members' work under a shared project IAM role.

• IAM authorizes API actions; Lake Formation authorizes data — its grants are enforced identically by Athena, Redshift Spectrum, EMR, and Glue via vended temporary S3 credentials.
• Write customer managed IAM policies when managed policies are too broad; explicit Deny always overrides any Allow, making it the tool for hard guardrails.
• Build least privilege iteratively: exact ARNs, condition keys, IAM Access Analyzer policy generation from CloudTrail activity, and permissions boundaries as a ceiling.
• Secrets Manager for anything that must rotate or cross accounts; Parameter Store for cheap configuration and SecureString settings without rotation.
• Amazon Redshift layers database users, groups, and RBAC roles under IAM, with IAM-issued temporary database credentials replacing password management for analysts.
• LF-Tags scale lake authorization: grant on tag expressions, let new tables inherit database tags, and stop writing per-resource grants.
• Data filters deliver row-level and, combined with column lists, cell-level security — different principals see different slices of the same table without maintaining views.
• Choose RBAC for stable job functions, tag-based (LF-TBAC) for growing lakes, and ABAC (principal-tag to resource-tag matching) when one policy must serve many teams.

• KMS uses envelope encryption: data keys encrypt the data locally, the KMS key encrypts the data keys, and disabling the KMS key instantly revokes access without re-encrypting anything.
• AWS managed keys (aws/s3, aws/redshift) can never be shared across accounts — any cross-account encrypted-data scenario requires a customer managed key.
• Cross-account access to KMS-encrypted data needs both a key policy statement in the key's account and IAM permissions in the consumer account; either one alone fails.
• Enforce encryption in transit by denying requests where aws:SecureTransport is false (S3) and by setting require_ssl for Redshift; EMR in-transit encryption uses a security configuration with TLS certificates.
• S3 encrypts new objects with SSE-S3 by default; use SSE-KMS for auditable key control, S3 Bucket Keys to cut KMS costs, and client-side encryption when AWS must never see plaintext.
• DynamoDB is always encrypted at rest — the only decision is AWS owned (default), AWS managed, or customer managed key.
• Lake Formation data filters give column-, row-, and cell-level masking at query time; Redshift dynamic data masking does the same inside the warehouse; Macie finds the PII that needs protecting.
• Tokenization is reversible via a vault and preserves joins; hashing is one-way and deterministic; anonymization is irreversible and can take data out of GDPR scope.

• CloudTrail management events cover control-plane actions and are on by default (90 days in Event history); data events like S3 GetObject and Lambda invocations must be explicitly enabled on a trail or event data store, in advance, at extra cost.
• An organization trail logs every member account — including future ones — to one bucket that member accounts cannot tamper with; it is the default answer for centralized multi-account audit capture.
• CloudTrail log file validation delivers signed digest files that prove whether any log file was changed or deleted; pair it with S3 Object Lock in compliance mode for WORM immutability.
• CloudWatch Logs stores application logs with per-log-group retention; metric filters turn log patterns into alarms, and subscription filters stream events to Lambda, Kinesis, Firehose, or OpenSearch in near real time.
• CloudTrail Lake gives immutable, multi-year, SQL-queryable audit storage with zero pipeline setup — choose it over trail-plus-Athena when minimal setup or non-AWS audit events are required.
• Use CloudWatch Logs Insights for interactive queries on logs still in CloudWatch; use Athena (with partitioning and compression) for logs archived in S3; use OpenSearch for full-text search and live dashboards.
• Amazon EMR is the answer for batch-processing very large log volumes — enriching or converting archives to Parquet — not for interactive investigation.
• Export long-lived application logs from CloudWatch Logs to S3 via Firehose and tier with lifecycle policies; keeping years of logs hot in CloudWatch or OpenSearch is a cost anti-pattern.

• Amazon Macie discovers sensitive data in S3 (managed and custom data identifiers, automated discovery, policy findings) but never enforces access — enforcement is Lake Formation's job.
• The detect-tag-enforce loop is the tested pattern: Macie findings trigger EventBridge automation that applies Lake Formation LF-tags, and tag-based access control restricts every table or column carrying the pii tag.
• Redshift datashares give live, read-only, zero-copy access using the consumer's own compute; cross-account sharing needs producer authorization plus consumer association, and RA3 or Serverless is required.
• Block disallowed Regions preventively with an SCP on aws:RequestedRegion, and separately deny the replication and copy actions (S3 CRR, DynamoDB global tables, snapshot and backup copies) that move data across Regions without anyone calling an API there.
• AWS Config records configuration history per resource, evaluates compliance with rules, auto-remediates via Systems Manager, and aggregates findings organization-wide — CloudTrail says who called; Config says what the configuration became.
• Data sovereignty stacks three layers: Region-deny SCPs (preventive), single-Region customer managed KMS keys (data-layer), and Config aggregators (detective).
• SageMaker Catalog projects add a publish-subscribe-approve workflow over Lake Formation and Redshift grants; Glue Data Catalog remains the technical metastore underneath.
• Know the sharing patterns by signal phrase: centralized lake (one governing team), data mesh (domain-owned data products), datashares (zero-copy warehouse sharing), Data Exchange (licensed third-party data).

Yes — the whole DEA-C01 deck is free and needs no account. Create a free account only if you want to save progress and drill full practice questions and mock exams.