Managing the Data Lifecycle: S3 Storage Tiers, Expiration, and Redshift

The COPY command is the correct way to load data from Amazon S3 into Amazon Redshift, and UNLOAD is the correct way to export query results from Redshift back to S3. COPY reads from S3 in parallel across the compute resources of the cluster, which makes it dramatically faster than looping over INSERT statements — any exam option that loads bulk data with single-row inserts is wrong.

To get that parallelism, follow the loading best practices AWS tests repeatedly: split large datasets into multiple files of roughly equal size so every slice does similar work, compress the files (gzip, and columnar formats such as Apache Parquet are also supported), and use a manifest file when you must load an exact list of objects rather than everything under a key prefix. Authentication should use an IAM role attached to the cluster (IAM_ROLE) instead of embedding access keys in the SQL. COPY can also apply automatic compression encoding analysis on an initial load into an empty table.

UNLOAD runs a SELECT query and writes the result set to S3, by default in parallel as multiple files. You can write Parquet directly with FORMAT PARQUET — a common exam answer when downstream consumers are Amazon Athena or AWS Glue — and use PARTITION BY to write the output into partition folders. Use PARALLEL OFF only when a consumer genuinely needs a single file, because it serializes the write.

One trade-off worth knowing: Amazon Redshift Spectrum lets you query data in place in S3 without loading it at all. If a scenario says data is queried rarely and loading it would waste cluster storage, querying it externally (Spectrum) beats COPY. If the data is hot and joined constantly, loading it into local tables wins.

Choose an S3 storage class by asking two questions: how often is the data accessed, and how fast must a retrieval be? S3 Standard is for frequently accessed data with no retrieval fees. S3 Standard-IA (Infrequent Access) offers lower storage cost but a per-GB retrieval charge and a minimum storage duration, so it suits data read perhaps monthly. S3 One Zone-IA is cheaper still but stores data in a single Availability Zone — it is only appropriate for re-creatable data such as secondary copies or easily regenerated derivatives, because an AZ loss can destroy it. S3 Intelligent-Tiering automatically moves objects between access tiers based on observed access patterns for a small monitoring charge, and is the exam answer whenever access patterns are unknown or changing.

The archive tiers trade retrieval speed for cost. S3 Glacier Instant Retrieval gives millisecond access to rarely touched data (think quarterly-accessed compliance reads). S3 Glacier Flexible Retrieval requires a restore before access, with retrieval options ranging from minutes (expedited) to hours (standard and bulk). S3 Glacier Deep Archive is the cheapest storage on AWS with retrievals measured in hours, built for long-term retention such as seven-to-ten-year regulatory archives.

Two gotchas: the IA and Glacier classes have minimum storage durations, so objects deleted or transitioned early still incur the minimum charge — putting short-lived objects in Standard-IA can cost more than Standard. And every class except One Zone-IA (and S3 Express One Zone) stores data redundantly across multiple Availability Zones with the same eleven-nines durability design.

S3 Lifecycle configurations automate storage-class transitions so no one has to move objects by hand. A lifecycle rule attaches to a bucket, optionally scoped by prefix, object tags, or object size, and defines transition actions (change storage class after N days) and expiration actions (delete after N days). A classic data-lake policy: keep raw landing files in S3 Standard for 30 days while pipelines and analysts touch them, transition to Standard-IA at 30 days, move to Glacier Flexible Retrieval at 90 days, and expire at the end of the retention period.

Concrete scenario: a clickstream pipeline lands compressed JSON in s3://lake/raw/, a nightly COPY loads it into Redshift, and after loading the raw files are rarely read again. The right answer is a lifecycle rule on the raw/ prefix — not a scheduled Lambda that calls CopyObject, and not manual moves. Lifecycle rules are free to configure, run inside S3, and are always the preferred mechanism when the trigger is object age.

Constraints the exam probes: transitions only move down the cost hierarchy — a lifecycle rule cannot move an object from Glacier back to Standard (you restore and copy instead). Objects must generally age a minimum number of days before transitioning into the IA classes, and very small objects are not worth transitioning because IA and archive tiers carry per-object overhead and minimum-size billing considerations. When access patterns are unpredictable, skip the hand-tuned transition schedule entirely and use S3 Intelligent-Tiering, which does the tiering per object based on actual access.

To delete S3 data automatically when it reaches a specific age, add an expiration action to a lifecycle rule — S3 then removes objects some time after they pass the age threshold, with no scheduler, no Lambda, and no risk of a forgotten cron job. This is the exam's default answer for retention requirements such as delete logs after 400 days or raw files must not be kept longer than 90 days.

Expiration behaves differently on versioned buckets, and this is a favorite trick. On a version-enabled bucket, an expiration action on the current version does not destroy data — it adds a delete marker and the old version becomes noncurrent. To actually reclaim storage you need a noncurrent version expiration action, which permanently deletes noncurrent versions after they have been noncurrent for a set number of days (optionally retaining a number of newer versions). A complete versioned-bucket policy usually pairs the two, and adds cleanup of expired object delete markers so orphaned markers do not accumulate.

Two adjacent cleanup actions are worth memorizing because they save real money: AbortIncompleteMultipartUpload deletes the invisible parts of multipart uploads that never completed, and tag- or prefix-scoped rules let you apply different retention to different datasets in the same bucket — for example, expiring tmp/ quickly while archiving curated/ for years.

S3 versioning keeps every version of an object in the bucket, protecting you from accidental overwrites and deletes: writing to an existing key creates a new version rather than replacing the data, and a simple DELETE inserts a delete marker rather than removing anything. Restoring an object is as easy as deleting the delete marker or copying an older version back as the current one. Versioning is also a hard prerequisite for S3 Replication, so resiliency designs usually turn it on anyway.

Versioning has exactly three states — unversioned (the default), enabled, and suspended. Once enabled it can never be fully turned off, only suspended, and suspension does not delete the versions you already have. Because every overwrite retains the old bytes, versioned buckets grow silently; the standard companion is the noncurrent-version lifecycle expiration from the previous section, which caps how long superseded versions live.

For an extra layer of protection against destructive deletes, MFA delete requires multi-factor authentication to permanently delete a version or change the bucket's versioning state. And when the requirement escalates from protect against accidents to make deletion impossible for a retention period, versioning alone is not the answer — that is S3 Object Lock territory, covered under deletion and legal requirements below.

DynamoDB Time to Live (TTL) deletes items automatically once a timestamp attribute you designate passes, at no cost — TTL deletes consume no write capacity. You enable TTL per table, name the attribute (for example expireAt), and store the expiry time in that attribute as an epoch timestamp in seconds. Classic use cases the exam presents: session data, one-time tokens, device-state caches, and any table where rows are only meaningful for a bounded window.

The critical behavioral detail: expiry is not instantaneous. TTL is a background process, and expired items are typically removed within a few days of their expiry time, depending on table size and activity. Until the deletion happens, expired items still appear in reads, scans, and query results — so applications with strict correctness requirements must add a filter expression that excludes items whose TTL attribute is in the past. An exam scenario that says users occasionally see expired sessions is pointing at this exact gap.

TTL deletions do appear in DynamoDB Streams as delete events (attributed to the DynamoDB service principal), which enables a tidy archival pattern: enable a stream, trigger AWS Lambda on the TTL delete records, and write the expiring items to S3. That gives you hot data in DynamoDB, automatic cleanup, and a cheap long-term archive — a three-service pattern DEA-C01 rewards. Items whose timestamp is malformed, missing, or not in epoch seconds are simply never expired, which is a common cause of tables that keep growing.

When a legal or business requirement says data must be deleted — a privacy-law erasure request, a contract ending, a retention limit — the deletion must be complete, and each store has its own definition of complete. In a versioned S3 bucket, deleting the current version is not enough: every noncurrent version must be permanently deleted too (a delete specifying the version ID), or the data still exists. For deletions at scale across millions of objects, S3 Batch Operations can run a delete or a Lambda-backed action against a manifest of objects rather than hand-rolled scripts.

The inverse requirement — data that must not be deletable — is S3 Object Lock. Compliance mode prevents any user, including the root account, from deleting or overwriting locked versions until the retention period ends; governance mode allows specially permissioned users to override; a legal hold blocks deletion indefinitely until the hold is removed. Know the interaction: if a scenario combines an erasure obligation with objects under compliance-mode lock, the design was wrong at write time — locks cannot be shortened, so data subject to erasure requests should never be written under compliance retention.

In Amazon Redshift, DELETE marks rows for deletion but the space is reclaimed and rows physically removed by VACUUM (Redshift runs automatic vacuuming, but an explicit VACUUM DELETE ONLY forces the reclaim). Also remember the copies: snapshots, backups, and replicas contain the deleted data too, so a genuine legal purge has to consider snapshot retention windows. Finding where personal data lives in S3 in the first place is a job for Amazon Macie, which discovers and classifies sensitive data such as PII.

Resiliency is about surviving failures and mistakes; the lifecycle story is incomplete without it, and task 2.3 names it explicitly. In S3, the multi-AZ storage classes are designed for eleven nines of durability, and the deliberate exception — One Zone-IA — should only ever hold data you can re-create. Beyond durability, S3 Replication copies objects to another bucket: Same-Region Replication (SRR) for aggregation and separate-account backup copies, Cross-Region Replication (CRR) for disaster recovery and compliance requirements that demand a geographically separate copy. Both require versioning on source and destination, replication is not retroactive for existing objects (S3 Batch Replication handles the backlog), and the destination can use a different storage class — replicating straight into a Glacier class is a cheap DR pattern.

For the databases: Amazon Redshift takes automated snapshots on a schedule with a configurable retention period, supports manual snapshots that persist until you delete them, and can copy snapshots to another Region for DR. Amazon DynamoDB offers point-in-time recovery (PITR), which lets you restore a table to any second within the preceding retention window (up to 35 days), on-demand backups that live until deleted, and global tables for active multi-Region availability. AWS Backup can centralize backup policies across S3, DynamoDB, RDS, and more when the scenario asks for one managed, policy-driven backup solution.

Exam heuristic: match the mechanism to the failure being described. Accidental overwrite or delete — versioning (plus PITR for DynamoDB). AZ failure — multi-AZ storage classes and replicas. Region failure or compliance distance — CRR, cross-Region snapshot copy, global tables. Malicious or accidental destruction that must be impossible — Object Lock and MFA delete.