Data Ingestion & Transformation: Kinesis, Glue, Athena, and Lake Formation

Amazon Kinesis Data Streams is the ingestion backbone when your application needs to process streaming records in real time with its own code. Producers put records onto the stream; one or more consumer applications — AWS Lambda functions, containers running the Kinesis Client Library, Amazon Managed Service for Apache Flink — read and act on them within moments of arrival. That word consumers is the decision hinge: Data Streams gives you a durable, ordered buffer, and you bring the compute that reads it.

Three properties define it on the exam. First, retention and replay: records persist for 24 hours by default, extendable up to 365 days, and consumers can rewind and re-read — multiple independent applications can process the same stream, each at its own pace. Second, ordering: a stream is divided into shards, and every record carries a partition key that determines its shard. Records sharing a partition key land in the same shard and are read in the exact order written — so "events for each device must be processed in order" resolves to Data Streams with the device ID as partition key. Third, capacity modes, needed at recognition level: provisioned mode makes you size the shard count yourself (each shard ingests up to 1 MB or 1,000 records per second), while on-demand mode scales capacity automatically and suits spiky or unpredictable traffic — the LEAST-overhead choice when throughput is unknown.

The cost of this power is that you own the consumer fleet: writing, deploying, scaling, and monitoring the code that reads the stream. When a scenario needs none of that — the data just has to land somewhere — Data Streams is over-engineering, which sets up the next section.

The exam tests Data Streams by requirement keywords: "real-time processing," "multiple applications consume the same data," "replay events," or "strict per-key ordering" all point here, and an option that swaps in Firehose fails the moment custom consumers or replay are required.

Amazon Data Firehose (long known as Kinesis Data Firehose) answers a different question: not "how do I process a stream" but "how do I get streaming data into a destination with no code and no infrastructure." Firehose is a fully managed delivery service — you point it at Amazon S3, Amazon Redshift, Amazon OpenSearch Service, or supported third-party and HTTP endpoints, and it handles batching, retries, scaling, and delivery entirely on its own. There are no shards to size, no consumers to write, nothing to administer.

Two behaviors matter for exam decisions. First, Firehose buffers: it accumulates records until a size or time threshold is reached, then delivers the batch. That makes it near-real-time, not real-time — data arrives in the destination seconds to minutes after ingestion, which is fine for loading a data lake and wrong for a fraud check that must react instantly. Second, Firehose transforms in flight: it can invoke an inline AWS Lambda function to clean, enrich, or reshape each batch, and it offers built-in format conversion that turns incoming JSON into Apache Parquet or ORC before writing to S3 — columnar output with zero ETL jobs, a detail the exam rewards repeatedly. Firehose does not retain data for replay and does not support custom consumers; it is a pipe, not a buffer.

Two related services round out the picture. Amazon MSK (Managed Streaming for Apache Kafka) exists for one exam signal: the scenario names Kafka — an existing Kafka application, Kafka APIs, or a Kafka ecosystem tool — and the answer is MSK rather than rewriting for Kinesis. And the two Kinesis services compose: the classic pattern is Data Streams → Firehose, where real-time consumers process the stream and Firehose reads the same stream to archive everything into S3 for analytics.

On the exam, "no consumer code," "deliver to S3/Redshift/OpenSearch," "near-real-time," and "least operational overhead" are the four fingerprints of Firehose; "real-time with custom processing" or "replay" pulls you back to Data Streams, and "Kafka" pulls you to MSK.

Not all ingestion is streaming — a large share of Task 3.5 questions move files into AWS, and each transfer service owns one requirement shape.

AWS DataSync is the online migration and scheduled-copy tool: an agent reads on-premises NFS, SMB, or HDFS storage (or another cloud's object store) and copies data over the network into S3, EFS, or FSx — one time for a migration, or on a schedule for recurring transfers. It handles encryption in transit, integrity verification, and bandwidth throttling, and it also copies between AWS storage services. The keyword is transfer: the data moves, and afterwards applications use it in AWS.

AWS Storage Gateway solves ongoing hybrid access, not transfer: on-premises applications keep reading and writing through a local gateway (File Gateway exposing NFS/SMB backed by S3, Volume Gateway for block storage, Tape Gateway for backup software) while the data lives in AWS. If the scenario says on-premises systems must continue accessing the data after it lands in S3, that continuing access is what separates Storage Gateway from DataSync.

AWS Transfer Family gives external partners what they already use: fully managed SFTP, FTPS, and FTP endpoints that deposit files directly into S3 (or EFS). When a scenario says partners "upload files over SFTP" and the company wants out of the server-management business, Transfer Family is the answer — no EC2-hosted FTP server, no patching.

The AWS Snow family is the offline path: physical devices for bulk transfer when the dataset is too large or the link too slow for network transfer to finish in acceptable time.

The exam tests this as pure pattern-matching under a LEAST-overhead qualifier: each option is a real transfer mechanism, and the frequency and access clues in the scenario select exactly one.

A data lake on AWS is not a product — it is Amazon S3 holding raw and transformed data at any scale, plus a metadata catalog that makes the objects queryable as tables. S3 wins the storage role on every axis the exam cares about: effectively unlimited capacity, eleven-nines durability, lifecycle tiering for cost, and native integration with every analytics service. Data lands in the lake through the ingestion paths already covered — Firehose delivery, DataSync copies, Transfer Family uploads — typically organized into zones (raw, cleaned, curated) by prefix.

Securing the lake used to mean sprawl: bucket policies for S3, IAM policies per analytics service, and no single view of who can read which table. AWS Lake Formation exists to collapse that sprawl. It layers a centralized permissions model over the S3 data and the Glue Data Catalog: you grant users and roles access at the database, table, and column level in one place, and those grants are enforced consistently across Athena, Redshift Spectrum, Glue, and EMR. Instead of maintaining parallel bucket policies and per-service IAM statements, one Lake Formation grant answers "can this analyst see this column." Lake Formation also helps stand the lake up — registering S3 locations, crawling and cataloging data — but its exam identity is governance: fine-grained, centralized data-lake permissions.

The ingestion points themselves need securing too, and the controls are the standard ones applied deliberately: IAM policies scope which producers can write to a stream or bucket, interface VPC endpoints keep Kinesis and Firehose traffic off the public internet, server-side encryption with KMS protects streams and S3 data at rest, TLS protects data in transit, and Transfer Family authenticates partners with service-managed or directory-based credentials rather than shared server logins.

On the exam, "centrally manage fine-grained access to a data lake" or "grant column-level permissions across Athena and Redshift Spectrum" is Lake Formation by definition — options that hand-assemble the same result from bucket policies and per-service IAM are the operational-overhead distractors.

AWS Glue is the transformation layer of Task 3.5, and it earns two separate exam identities.

The first is the Glue Data Catalog: a central, persistent metadata store — table definitions, schemas, partition locations — describing data wherever it sits, most importantly in S3. Glue crawlers populate it automatically: point a crawler at an S3 prefix, and it samples the objects, infers the schema, detects partitions, and writes table definitions into the catalog. The catalog is shared infrastructure: Athena, EMR, and Redshift Spectrum all read the same catalog, so a table crawled once is instantly queryable from all of them. When a scenario says analysts "cannot query the files in S3 because there is no schema" or asks how to make S3 data available to Athena without manual table definitions, the crawler-plus-catalog pair is the answer.

The second identity is serverless ETL. Glue jobs run Apache Spark (or lightweight Python) transformations — cleaning records, joining datasets, and above all converting formats, such as CSV to Parquet — without any cluster for you to provision, size, or patch. Jobs can run on a schedule, on demand, or from event triggers, chaining crawl-transform-crawl pipelines. Because it is serverless, Glue is the default ETL answer under a LEAST-operational-overhead qualifier; standing up an EMR cluster to run the same Spark job is the classic higher-overhead distractor when nothing in the scenario demands cluster control.

The exam tests Glue at decision level, not API level: recognize the crawler as the schema-inference mechanism, the catalog as the shared metastore that links S3 to every query engine, and Glue jobs as the no-cluster way to transform data between formats. If an option has your team "writing Spark code on a self-managed cluster" for a routine transformation, Glue is almost certainly the intended answer beside it.

Once data is cataloged, three services compete to query it, and SAA-C03 separates them by workload shape rather than feature lists.

Amazon Athena is serverless, interactive SQL directly against S3, using the Glue Data Catalog for schemas. There is no cluster and no data loading — you pay per data scanned by each query. That makes Athena the answer for ad hoc, occasional, or unpredictable querying of data that lives in S3: log investigations, one-off analyses, dashboards refreshed a few times a day. Its economics also create its one design obligation — because you pay by bytes scanned, format and partitioning matter enormously, which the next section covers.

Amazon Redshift is a data warehouse: data is loaded into Redshift's own managed, columnar storage, where repeated, complex BI workloads — dashboards hit constantly, heavy joins and aggregations across billions of rows, concurrent analyst teams — run with consistently high performance. The trade is overhead and commitment: you operate a warehouse (or use Redshift Serverless) and maintain load pipelines. Redshift Spectrum blurs the edge by letting Redshift query S3 data through the same Glue catalog, joining warehouse tables with lake data in one SQL statement.

Amazon EMR is the managed big-data cluster — Spark, Hadoop, Hive, Presto — for when you need framework-level control: custom Spark applications, specific library versions, machine-learning pipelines, or long-running distributed jobs that SQL cannot express. It is the most powerful and the most operationally expensive of the three.

Visualization gets one line because the exam gives it one: Amazon QuickSight is the managed BI service that builds dashboards on Athena, Redshift, S3, and other sources — any "business users need dashboards" requirement resolves to it.

Expect the exam to qualify these questions hard: "serverless" plus "infrequent" eliminates Redshift, "consistently fast dashboards for hundreds of analysts" eliminates Athena, and a named framework like Spark with custom code eliminates both in favor of EMR — unless Glue's serverless Spark covers it with less overhead.

The single most repeatable optimization in this task statement is a format change. Row-based text formats like CSV and JSON force a query engine to read every byte of every record even when the query touches two columns. Columnar formats — Apache Parquet and ORC — store each column's values together, so Athena reads only the columns the query names; they also compress far better, because a column of similar values compresses more tightly than interleaved rows. Since Athena bills per data scanned, converting a lake from CSV to compressed Parquet routinely cuts both query cost and query time by large multiples — same data, same SQL, a fraction of the bytes touched.

Partitioning is the second lever. Organizing S3 objects by key prefix — most commonly date, as in year=2026/month=07/day=15/ — lets the catalog record each partition's location, and a query filtered on the partition column (WHERE year = 2026 AND month = 07) skips every other partition entirely. Partition on the columns queries actually filter by; a lake partitioned by date serves "last week's events" by scanning seven days, not seven years.

You already know both conversion paths. For data already in S3, a Glue ETL job reads the CSV, writes partitioned, compressed Parquet, and a crawler updates the catalog. For streaming data, Firehose format conversion writes Parquet from the start — no post-hoc ETL at all, which wins any LEAST-overhead comparison for new pipelines. Compression (Snappy is the common default for Parquet) stacks on top of either path.

This is tested as a cost/performance scenario with a giveaway sentence: "queries scan the entire dataset," "Athena costs are growing," or "analysts only query recent data." The correct option combines columnar conversion with partitioning; distractors throw capacity at the problem — moving to Redshift, adding EMR nodes — when the data layout was the actual defect.

Task 3.5 questions increasingly describe whole pipelines, and the strongest preparation is holding the reference chain in your head: ingest → transform → store → catalog → query → visualize, with two or three service candidates at each stage.

A representative streaming architecture: producers (applications, IoT devices, clickstream collectors) write to Kinesis Data Streams if real-time consumers, replay, or per-key ordering are required — otherwise straight into Amazon Data Firehose. Firehose applies an inline Lambda transform if records need cleaning, converts to Parquet, and delivers into partitioned prefixes in the S3 data lake. A Glue crawler keeps the Data Catalog current; Lake Formation governs who can query which tables and columns. Athena serves ad hoc SQL, heavy repeated BI loads into Redshift, custom Spark jobs run on EMR or serverless Glue, and QuickSight dashboards sit on top. Batch and file-based sources merge into the same lake through DataSync schedules or Transfer Family endpoints.

Reading a pipeline question, work stage by stage and eliminate per stage: does ingestion need consumers or just delivery? Does transformation need a cluster or does Glue/Firehose cover it? Does querying justify a warehouse or is pay-per-scan Athena enough? Each stage usually has exactly one qualifier word — "near-real-time," "no infrastructure to manage," "ad hoc" — that fixes the choice.

Frequency and volume clues size the ingestion stage. Continuous record-by-record arrival is streaming; hourly or nightly batches are transfer or scheduled ETL; a one-time bulk load is DataSync or Snow depending on the network. Where throughput numbers appear, they select capacity, not service internals: a provisioned Data Streams shard ingests up to 1 MB or 1,000 records per second, so stated rates translate to a shard count — and if the scenario calls the traffic spiky or unpredictable, on-demand mode (or Firehose, which scales invisibly) removes the sizing exercise altogether. You are never asked to compute precise capacity; you are asked to notice when a requirement outgrows a fixed configuration.

The exam tests this compositionally: four answer options each name a full chain, three of which contain one wrong link — Data Streams with no consumer, EMR for a trivial format conversion, Redshift for occasional queries. Verify every link, not just the first.

Scenario 1. A media company collects clickstream events from its web applications and must land them in an S3 data lake in Apache Parquet format for analysis with Athena. Near-real-time delivery (within minutes) is acceptable. The team wants a solution with the LEAST operational overhead and does not want to write or manage any consumer applications. What should the architect choose?

Reasoning: Strike the options stage by stage. Kinesis Data Streams with a Lambda consumer writing Parquet works, but it exists to serve custom real-time consumers — the scenario explicitly rules out consumer code, and the Lambda plus its error handling is precisely the overhead being minimized. MSK requires Kafka clients and cluster-adjacent management with no Kafka requirement in sight. A Glue batch job converting CSV drops on a schedule fails "near-real-time" and adds a second hop. The answer is Amazon Data Firehose with its built-in format conversion to Parquet, delivering directly into partitioned S3 prefixes: fully managed, no consumers, no shards, near-real-time by design — and its buffering delay is acceptable because the scenario said minutes are fine. Every requirement sentence maps to a Firehose property; that alignment is what a correct SAA-C03 answer looks like.

Scenario 2. An analytics team queries three years of CSV application logs in S3 using Athena. Queries have grown slow and expensive; the team reports that even queries filtered to a single week "scan the entire dataset." They want to reduce Athena cost and query time with the LEAST change to their workflow. What do you recommend?

Reasoning: The complaint names both defects. "CSV" means every query reads every byte of every row — the fix is a Glue ETL job converting the logs to compressed Parquet, so Athena scans only the referenced columns. "A single week scans everything" means the data is unpartitioned — the fix is writing the Parquet output partitioned by date and recrawling so the catalog knows the partitions; week-filtered queries then touch only seven partitions. Together the two changes cut billed bytes on both axes while the team keeps the same Athena SQL workflow. The distractors fail the qualifier: migrating to Redshift solves speed by adding a warehouse to load, operate, and pay for — a workflow change, not a data-layout fix; EMR with Presto swaps serverless for a cluster; and "increase Athena capacity" is not a lever that exists — Athena is serverless, and its cost dial is bytes scanned.

Scenario 3. Hundreds of retail partners currently upload nightly inventory files over SFTP to an on-premises server, which is being retired. Files must end up in S3 for a Glue pipeline, and partners cannot change their upload tooling. What replaces the server?

Reasoning: "Partners keep using SFTP" plus "files into S3" is the signature of AWS Transfer Family: a managed SFTP endpoint backed directly by S3, per-partner credentials, nothing to patch. DataSync is wrong-shaped — it is the company pulling data from its own storage, not partners pushing; Storage Gateway serves ongoing on-premises access to AWS-resident data, and an EC2-hosted SFTP server recreates the retired problem inside AWS. When the protocol is fixed by an external party, pick the service that speaks it natively.