Managing the Data Lifecycle: S3 Storage Tiers, Expiration, and Redshift

The COPY command is the way to bulk-load data from Amazon S3 into Amazon Redshift: it reads many files at once across every slice of the cluster (or Serverless workgroup), which makes it far faster than looping over INSERT statements. Authenticate with an IAM role associated with the cluster (IAM_ROLE) rather than access keys embedded in the SQL, and on an initial load into an empty table COPY can analyse the data and apply compression encodings automatically.

Why file layout matters. Each slice loads files independently, so the load is only as fast as its busiest slice. Redshift can split a large uncompressed delimited file itself, but it cannot split a compressed file such as a single gzip object — one large compressed file is read by one slice while the rest sit idle. Split the data into multiple files of roughly equal size (AWS suggests 1 MB–1 GB each after compression) and make the file count a multiple of the number of slices so every slice gets the same share. Compressed and columnar inputs (gzip, bzip2, zstd, Parquet, ORC) all reduce the bytes read from S3.

Prefixes versus manifests. The S3 path in a COPY is a key prefix, not a folder: s3://bucket/logs/2026-01-15 also matches logs/2026-01-15-backfill/ and anything else that starts with the same characters. When a load must read an exact set of files, use a manifest — a JSON file listing each object URL — with the MANIFEST option. Each entry has a mandatory flag that defaults to false: a missing non-mandatory file is silently skipped, while "mandatory": true makes the whole COPY fail if that file is absent. Two near-misses: MAXERROR tolerates a number of bad records inside files that were read, not missing files, and NOLOAD only validates the files that exist without loading them.

Continuous loading with auto-copy. When files arrive steadily and each should be loaded soon after landing, Redshift auto-copy replaces a hand-built trigger. You create an S3 event integration for the bucket, then a copy job with COPY … JOB CREATE job_name AUTO ON. Redshift detects new objects under the prefix, runs the COPY automatically and records which files it has already loaded so none are loaded twice. The bucket must be in the same Region as the cluster or workgroup. Querying data in place instead of loading it is covered in the next section.

UNLOAD runs a SELECT and writes the result set from Amazon Redshift to Amazon S3. By default it writes in parallel: each slice writes its own files, so the output is several files (one or more per slice) rather than one. Use FORMAT PARQUET when downstream consumers are Amazon Athena, AWS Glue or Redshift Spectrum, and PARTITION BY to write Hive-style partition folders those engines can prune.

So a consumer that needs exactly one file of a few gigabytes gets it from PARALLEL OFF, not from raising MAXFILESIZE.

Tiering cold warehouse data to S3. Amazon Redshift Spectrum queries data in S3 without loading it into the cluster. A common lifecycle pattern for a table where only recent rows are hot: UNLOAD the older rows to S3 as partitioned Parquet, define an external table over that location (Spectrum, using the AWS Glue Data Catalog), then DELETE those rows from the local table and VACUUM to reclaim the space. Analysts still join old and new data with SQL in Redshift (for example through a view that unions the local and external tables) while the cluster stores only the hot data. Conversely, data that is hot and joined constantly belongs in local tables loaded with COPY.

Choose an S3 storage class by asking how often the data is read, how fast a read must be, and whether it can survive the loss of an Availability Zone. S3 Standard suits frequently accessed data and has no retrieval fees or minimums. S3 Standard-IA and S3 One Zone-IA store data more cheaply but add a per-GB retrieval fee, a 30-day minimum storage duration and a 128 KB minimum billable object size; One Zone-IA keeps a single AZ copy, so it is only for re-creatable data such as secondary copies. S3 Glacier Instant Retrieval is cheaper again to store and still returns data in milliseconds, but its retrieval fee is higher and its minimum storage duration is 90 days — it wins for data read about once a quarter and loses to Standard-IA for data read monthly.

Like One Zone-IA, S3 Express One Zone stores data in a single Availability Zone (in a directory bucket); it is built for single-digit-millisecond access by latency-sensitive compute in that zone, not for cost-saving archives, so it is outside the lifecycle hierarchy above.

Archive retrieval tiers. Objects in Glacier Flexible Retrieval and Deep Archive must be restored (a temporary copy) before they can be read, and the tier you request sets both speed and price:

Expedited requests can be rejected when demand is high; buying provisioned retrieval capacity guarantees they are accepted. The general rule: pick the cheapest storage class whose fastest available retrieval tier still meets the tightest deadline you must honour. Deep Archive has no Expedited tier, so it can never meet a deadline shorter than its Standard tier; Flexible Retrieval reaches minutes only through Expedited (add provisioned capacity when that deadline is contractual); Glacier Instant Retrieval needs no restore at all but costs more to store. A large restore that can wait half a day is cheapest as a Bulk restore from Flexible Retrieval.

S3 Intelligent-Tiering monitors each object and moves it automatically between the Frequent Access, Infrequent Access (after 30 days without access) and Archive Instant Access (after 90 days) tiers — all millisecond access, with no retrieval fees. Two further tiers, Archive Access and Deep Archive Access, are opt-in through an archive configuration on the bucket; objects in them behave like Glacier objects and must be restored before reading (a read attempt fails with InvalidObjectState), and a restored object returns to Frequent Access. Removing the archive configuration later does not bring already-archived objects back — they still need a restore. Objects smaller than 128 KB are never auto-tiered; they stay in Frequent Access and are not charged the monitoring fee.

S3 Lifecycle configurations move objects to cheaper storage classes as they age, inside S3, with no code to run. A rule has a filter and one or more actions: transition actions (change storage class after N days) and expiration actions (delete after N days, covered in the next section). A classic data-lake policy keeps raw landing files in S3 Standard while pipelines read them, transitions to Standard-IA at 30 days and to Glacier Flexible Retrieval at 90 days, then expires them at the end of the retention period. When the trigger is object age, a lifecycle rule is preferred over a scheduled Lambda calling CopyObject.

Filters. A rule can match on a prefix, one or more object tags, and object size (ObjectSizeGreaterThan / ObjectSizeLessThan). Combining them inside an And element makes an intersection — only objects that satisfy every condition match. Separate rules act independently, so two rules give you a union. An empty filter applies the rule to every object in the bucket. Some actions cannot be combined with a tag filter: AbortIncompleteMultipartUpload and ExpiredObjectDeleteMarker cannot be specified in a rule whose filter includes object tags (intuitively, an upload in progress or a delete marker has no tags to match).

Transition constraints the exam probes.

• Transitions only move down the cost hierarchy (Standard → IA → Intelligent-Tiering/Glacier classes → Deep Archive). Lifecycle cannot move an object from Glacier back to Standard; you restore it and copy it.
• Transitions from Standard to Standard-IA or One Zone-IA require the object to be at least 30 days old — the lifecycle API rejects a shorter period.
• Since September 2024, lifecycle by default does not transition objects smaller than 128 KB to any storage class. You can override this by adding an explicit size filter, but it is rarely worth it: the IA classes bill a 128 KB minimum per object.
• Minimum storage durations still apply when lifecycle moves or deletes an object early, so transitioning short-lived data can cost more than leaving it in Standard.

Small objects and the archive classes. Every object archived to Glacier Flexible Retrieval or Deep Archive carries about 40 KB of extra billed storage — 8 KB of metadata charged at S3 Standard rates and 32 KB charged at the archive rate — plus a per-object transition request charge. For hundreds of millions of tiny objects, that overhead can erase the saving. Aggregate small objects into larger files (for example, compacting them into archives or Parquet files) before archiving. When access patterns are unpredictable, skip a hand-tuned schedule and use S3 Intelligent-Tiering instead.

To delete S3 data automatically when it reaches a specific age, add an Expiration action to a lifecycle rule, scoped with a prefix, tag or size filter so it touches only the dataset whose retention has ended. S3 then removes objects asynchronously after they pass the age (measured from creation), with no scheduler or Lambda. This is the default design for requirements such as delete logs after 400 days — and a Transition action is not a substitute, because it moves data rather than deleting it.

Versioned buckets behave differently. On a versioning-enabled bucket, an Expiration action on the current version does not destroy data: it adds a delete marker and the previous version becomes noncurrent and keeps being billed. To remove the bytes permanently you also need NoncurrentVersionExpiration, which deletes versions a set number of days after they became noncurrent; its optional NewerNoncurrentVersions setting keeps the most recent N noncurrent versions and expires older ones. A complete versioned-bucket retention policy therefore pairs Expiration (current versions) with NoncurrentVersionExpiration (old versions). Once all noncurrent versions of a key are gone, the remaining delete marker is an expired object delete marker; the ExpiredObjectDeleteMarker action cleans these up so they do not accumulate (it cannot be set in the same action as a days-based current-version Expiration).

AbortIncompleteMultipartUpload deletes the parts of multipart uploads that never completed — parts that are invisible in listings but billed as storage. Scope it with a prefix when only one pipeline's uploads should be cleaned. Lifecycle actions are carried out by S3 itself, so a bucket policy that denies s3:DeleteObject stops users and applications but does not stop a lifecycle expiration; only S3 Object Lock prevents lifecycle from removing a protected version.

S3 versioning keeps every version of an object in the bucket, protecting you from accidental overwrites and deletes: writing to an existing key creates a new version rather than replacing the data, and a simple DELETE inserts a delete marker rather than removing anything. Restoring an object is as easy as deleting the delete marker or copying an older version back as the current one. Versioning is also a hard prerequisite for S3 Replication and S3 Object Lock, so resiliency designs usually turn it on anyway.

Versioning has exactly three states — unversioned (the default), enabled, and suspended. Once enabled it can never be fully turned off, only suspended, and suspension does not delete the versions you already have. Because every overwrite retains the old bytes, versioned buckets grow silently; the standard companion is the noncurrent-version lifecycle expiration from the previous section, which caps how long superseded versions live.

For an extra layer of protection against destructive deletes, MFA delete requires multi-factor authentication to permanently delete a version or change the bucket's versioning state. And when the requirement escalates from protect against accidents to make deletion impossible for a retention period, versioning alone is not the answer — that is S3 Object Lock territory, covered under deletion and legal requirements below.

DynamoDB Time to Live (TTL) deletes items automatically once a timestamp attribute you designate has passed, and those deletes consume no write capacity. You enable TTL per table and name the attribute (for example expireAt). The attribute must be of the DynamoDB Number type holding a Unix epoch time in seconds. Items where the attribute is missing, is a String (such as an ISO 8601 date) or holds milliseconds are simply never expired — milliseconds read as a date thousands of years away — which is a common reason a TTL table keeps growing. Typical uses: sessions, carts, one-time tokens and device-state caches.

Expiry is not instantaneous. TTL is a background process; expired items are typically deleted within a few days of their expiry time. Until then they remain in the table and are returned by reads, queries and scans — with eventually consistent and strongly consistent reads alike, because consistency is about replication, not expiry. Applications that must never show expired data add a filter expression (or key condition) that excludes items whose TTL value is earlier than the current time.

Archiving expired items. TTL deletions appear in DynamoDB Streams as REMOVE records. A TTL delete is distinguishable from an application delete by its userIdentity field: type is Service and principalId is dynamodb.amazonaws.com; deletes made by your application have no such identity. To archive only TTL-removed items with their full contents:

1. Enable a stream with the OLD_IMAGE or NEW_AND_OLD_IMAGES view type — a KEYS_ONLY or NEW_IMAGE stream has no attributes left to archive for a deleted item.
2. Attach an AWS Lambda event source mapping with an event filter matching that service userIdentity, so the function is invoked only for TTL deletes and needs no filtering code of its own.
3. Write the old image to Amazon S3 (directly or through Amazon Data Firehose).

The result is hot data in DynamoDB, automatic cleanup, and a cheap long-term archive in S3.

When a legal or business requirement says data must be deleted — a privacy-law erasure request, a contract ending, a retention limit — the deletion must be complete, and each store defines complete differently: every S3 version, every Redshift block, every table snapshot and every backup copy.

Amazon S3. In a versioned bucket, deleting the current version only adds a delete marker; each version must be deleted by version ID, or an ongoing requirement handled by lifecycle (Expiration plus NoncurrentVersionExpiration). For a one-time deletion across millions of listed objects, note that S3 Batch Operations has no native delete-object operation: its operations include copy, tagging, ACLs, restore, Object Lock retention and legal hold, replication and checksums. Bulk deletes run through the Invoke AWS Lambda function operation over a manifest (for example an S3 Inventory report including version IDs), with a function that calls DeleteObject per key and version, and the job's completion report records which deletes succeeded. Amazon Macie helps find where personal data lives in S3 in the first place.

Data that must not be deleted: S3 Object Lock. Object Lock applies write-once-read-many protection to object versions; it requires versioning and can be enabled on new or existing buckets. It offers two controls:

Governance mode fits protect from everyone except a small admin team; compliance mode fits regulatory WORM. Lifecycle cannot permanently delete a locked version either. If data subject to erasure requests was written under compliance retention, the design was wrong at write time — the lock cannot be shortened.

Amazon Redshift. DELETE only marks rows as deleted; the blocks are physically reclaimed by vacuuming. Automatic vacuum delete runs in the background, but an explicit VACUUM DELETE ONLY reclaims the space promptly without spending time on sorting. Know the modes: VACUUM FULL (the default) reclaims and re-sorts, DELETE ONLY reclaims without sorting, and SORT ONLY sorts without reclaiming deleted-row space. Snapshots still contain the deleted rows until they age out or are deleted, and restoring a table from an older snapshot brings the rows back — a genuine purge accounts for snapshot retention.

Apache Iceberg tables queried with Amazon Athena. Athena's DELETE FROM on an Iceberg table writes position-delete files (merge-on-read); the original Parquet data files still hold the rows, and earlier snapshots still reference them for time travel. Permanent removal takes two steps: OPTIMIZE … REWRITE DATA USING BIN_PACK rewrites the affected data files without the deleted rows (set the table property optimize_rewrite_delete_file_threshold to 1, since its default of 2 skips files with a single delete file), then VACUUM expires snapshots older than vacuum_max_snapshot_age_seconds (default five days — lower it for a prompt purge) and removes the files no snapshot needs. Athena never expires snapshots unless you run VACUUM.

Resiliency means matching a protection mechanism to the failure you must survive — mistakes, an Availability Zone loss, a Region loss or deliberate destruction. In S3, the multi-AZ storage classes are designed for eleven nines of durability; One Zone-IA is the deliberate exception and should hold only re-creatable data.

S3 Replication. Same-Region Replication (SRR) copies objects to another bucket in the same Region (aggregation, separate-account copies); Cross-Region Replication (CRR) copies them to another Region (DR, geographic-separation requirements). Both need versioning on source and destination. Replication only applies to objects written after the rule exists — S3 Batch Replication backfills existing objects (and retries failed ones). A rule can set the replica's storage class directly, so replicating straight into a cheaper class needs no lifecycle step. S3 Replication Time Control (RTC) adds a 15-minute replication target backed by a service level agreement, with replication metrics and notifications. S3 Multi-Region Access Points give one global endpoint that routes requests to the nearest bucket and support failover, but they copy no data — replication rules underneath keep the buckets in sync.

Amazon DynamoDB. Point-in-time recovery (PITR) restores a table to any second within its recovery window (configurable, up to 35 days); on-demand backups persist until deleted; global tables keep active replicas in several Regions. Every PITR or backup restore creates a new table — the original table is untouched, so applications keep reading the corrupted data until they are pointed at the restored table (or data is copied back). For backups copied into a separate account or Region, use AWS Backup with its advanced DynamoDB features enabled, which add cross-account and cross-Region copy (and cold-storage tiering) on top of an organization-wide backup policy. The same AWS Backup plans also cover S3 buckets, RDS, EBS and other services, so one central, policy-driven backup schedule can protect a whole data platform.

Amazon Redshift.

To keep a retired warehouse restorable, take a final or manual snapshot with the required retention before deleting the cluster. For DR, enable cross-Region snapshot copy so automated (and manual) snapshots are copied to another Region on their own schedule; for a cluster encrypted with an AWS KMS customer managed key, you first create a snapshot copy grant in the destination Region naming a KMS key there. You can restore a whole cluster from a snapshot, or restore a single table from a snapshot into an existing cluster.

Heuristic. Accidental overwrite or delete — versioning, PITR, snapshots. AZ failure — multi-AZ storage classes and Multi-AZ deployments. Region failure or distance requirements — CRR (with RTC for a deadline), cross-Region snapshot copy, global tables. Deletion that must be impossible — Object Lock, MFA delete and backups held in a separate account.