DVA-C03 Data Stores: DynamoDB Design, Caching and Data Lifecycles

You choose a data store by writing down how the application reads and writes its data, then picking the service built for that pattern: DynamoDB for key-value lookups at any scale, Amazon RDS or Aurora for relational data with joins and multi-table transactions, Amazon S3 for large objects and files, and Amazon ElastiCache for in-memory data that must come back in under a millisecond. The questions to ask are: how is a record found (by a known key, or by ad hoc criteria)? How big is it? Must several records change together? How fast must reads be, and does the data have to survive a failure?

Two limits push designs across stores. A DynamoDB item, attribute names and values included, cannot exceed 400 KB; compressing or switching capacity mode does not change that. The standard pattern is to keep the large object in S3 and store its object key, plus searchable metadata, in the DynamoDB item, so queries stay small and the file is downloaded only when needed. In the other direction, DynamoDB transactions make a set of writes atomic, but DynamoDB still expects you to design keys and indexes for the queries you know about; it is not a substitute for a relational engine when people keep writing new join-heavy reports. Athena can run SQL over S3, but S3 has no transactions.

An S3 Lifecycle configuration is a set of rules that S3 applies to objects on its own, with no code or scheduler: transition actions move objects to cheaper storage classes as they age, and expiration actions delete them. A rule's filter decides which objects it touches: a prefix, an object tag, object size, or several conditions combined in an And element. Rules are evaluated independently, so two separate rules (one for a prefix, one for a tag) act on anything that matches either; to act only on objects that have a given prefix and a given tag, put both in one rule's And filter.

Lifecycle maps neatly onto data whose value falls with age: for example, transition to S3 Standard-IA after 30 days (the earliest a Lifecycle rule can move objects into Standard-IA or One Zone-IA) and expire after a year. Adding an archive transition keeps data you meant to delete. S3 Intelligent-Tiering moves objects between access tiers automatically but never deletes anything, so it does not replace an expiration action.

Versioned buckets and multipart uploads

Versioning changes what "expire" means. In a versioned bucket, an Expiration action on the current version does not remove data: S3 adds a delete marker, and the previous object becomes a noncurrent version that is still stored and billed. Every overwrite also creates a noncurrent version. The actions that actually free storage are:

• NoncurrentVersionExpiration — permanently deletes noncurrent versions a set number of days after they become noncurrent (optionally keeping a number of newer noncurrent versions).
• ExpiredObjectDeleteMarker — removes delete markers that have no noncurrent versions left behind them, which otherwise clutter version listings.
• AbortIncompleteMultipartUpload — cleans up the parts of multipart uploads that were started but never completed; those parts are billed until the upload is aborted, yet they never appear as objects.

A useful diagnostic: if the number of current objects is stable but billed storage keeps growing, the growth is in noncurrent versions or incomplete uploads, not in the objects you can see. Suspending versioning does not delete versions that already exist.

DynamoDB Time to Live (TTL) deletes items automatically after an expiry time you store in each item. You enable TTL on the table and name one attribute; DynamoDB treats an item as expired when that attribute is a Number holding a Unix epoch time in seconds that is in the past. Any other type, including an ISO 8601 string or an epoch value stored as a string, is silently ignored, and a value in milliseconds lands tens of thousands of years in the future, so neither ever expires. TTL deletions run in the background and do not consume write capacity, which makes TTL cheaper and simpler than a scheduled scan-and-delete job.

Expired is not the same as deleted. AWS removes expired items typically within a few days, and until then they still come back from GetItem, Query and Scan, even with strongly consistent reads. When an application must never show expired data, it filters on the TTL attribute itself (for example, a filter expression that keeps only items whose expiry is later than the current time).

TTL deletions appear in DynamoDB Streams as ordinary REMOVE records, so event name, source and old image look exactly like an application's DeleteItem. The difference is the record's userIdentity: TTL deletions carry type Service and principalId dynamodb.amazonaws.com. A stream consumer that must archive only expired items checks that field.

Retention on other stores

• CloudWatch Logs: a new log group keeps events forever by default. Each log group has a retention setting, chosen from a fixed list of periods (from one day up to ten years), after which CloudWatch Logs deletes older events itself. Metric filters turn events into metrics and subscription filters forward new events; neither removes stored events, and S3 Lifecycle rules do not apply to log groups. For cheap long-term keeping, a subscription filter to Amazon Data Firehose delivers events continuously to S3, where a Lifecycle expiration rule deletes the archive later, while the log group keeps only the short window people search.
• Cache keys: a TTL on each ElastiCache key bounds how long a cached value can live (see the caching section).
• S3 Object Lock: the opposite job, preventing deletion until a retention period ends (below).

S3 Object Lock: retention periods and legal holds

S3 Object Lock stops object versions from being deleted or overwritten for a set time or indefinitely, using a write-once-read-many (WORM) model; it requires versioning, which is enabled automatically when you turn Object Lock on. Each protected version can have a retention period in one of two modes, a legal hold, or both.

Retention modes always need a date, so an open-ended hold is a legal hold, not a mode. Legal holds and retention periods are independent: a version is protected while either is in force. Versioning plus a Lifecycle rule is a cost tool, not protection, because anyone with permission can still delete versions.

Choose ElastiCache for Valkey (or Redis OSS, which Valkey is compatible with) when the cache needs rich data types, replication, failover or persistence; choose Memcached only for a simple, multithreaded key-value cache whose contents can be lost. The table compares node-based clusters; ElastiCache Serverless caches for either engine replicate data across Availability Zones and support snapshots.

The data types are what make Valkey more than a cache. A sorted set keeps members ordered by score, so commands such as ZREVRANK and ZREVRANGE return a player's rank or the top N in sub-millisecond time even under heavy writes, which is the classic real-time leaderboard. Pub/sub channels let every web server receive each new chat message through the same store. Replication with Multi-AZ failover plus snapshots is what lets a session store survive a node failure and be restored after an accidental flush.

Scaling a node-based Valkey cluster

With cluster mode disabled there is one shard: a single primary takes every write and every node holds the whole data set. Read replicas add read capacity only, and vertical scaling stops at the largest node type. Cluster mode enabled partitions the key space across several shards, each with its own primary, which scales both write throughput and memory. For availability, give each shard at least one replica in another Availability Zone and enable Multi-AZ, so a replica is promoted automatically if a primary fails.

Use lazy loading (cache-aside) to cache only what is actually read, write-through to keep cached data fresh as it is written, and a TTL on keys with either strategy to bound staleness and free memory. Most production caches combine all three.

Staleness questions usually hinge on who writes the data. If another system that you cannot change updates the database, write-through is impossible and lazy loading never notices the change; a TTL that matches the staleness the business accepts (for example, 300 seconds for "up to 5 minutes old") is the least-code fix. If write-through fills memory with rarely read items and pushes out popular ones, add a TTL to the write-through path so unread entries expire, and add lazy loading so an expired item that is requested later is reloaded.

Eviction policies decide what goes when memory is full: allkeys-lru removes the least recently used keys, volatile-lru removes only keys that have a TTL, and noeviction makes new writes fail with an error instead. Eviction manages memory; it does not fix staleness, and neither do replicas or Multi-AZ.

Each store has a natural cache: DAX for DynamoDB, ElastiCache for relational query results, CloudFront for S3 objects and API Gateway caching for repeated API responses (table below). DynamoDB Accelerator (DAX) is a managed in-memory cache that runs in your VPC in front of a DynamoDB table and is API-compatible with DynamoDB: the application keeps the same GetItem, Query, PutItem and other calls and simply sends them through the DAX client, so read latency drops from milliseconds to microseconds without writing cache-aside logic. That is the difference from ElastiCache, which can cache DynamoDB data too but only with your own code around every call.

DAX keeps two caches:

• Item cache — results of GetItem and BatchGetItem. Writes sent through DAX are write-through: DAX writes to DynamoDB first and, once DynamoDB confirms, updates the cached item, so the application never gets success for data that is not in the table.
• Query cache — results of Query and Scan, keyed by the request parameters. Writes do not invalidate it; a cached result stays until its TTL ends. Both caches default to a 5-minute TTL, set separately in the cluster's parameter group, so lowering the query cache TTL bounds how stale Query results can be.

DAX caches eventually consistent reads only. A request with ConsistentRead set to true is passed straight through to DynamoDB and its result is not cached, so an application that sets strong consistency everywhere sees almost no cache hits and no drop in consumed read capacity. Adding nodes or raising TTLs does not change that; switching the reads that can tolerate eventual consistency does, while keeping strong reads only where read-after-write matters.

Cache by data path

What related features do, and what they do not do

• S3 Transfer Acceleration speeds long-distance transfers over the AWS network, and S3 Cross-Region Replication copies objects to a second bucket; neither caches, so every request is still served by a bucket.
• ElastiCache lives in one Region's VPC: it shortens the path between your application servers and the database, not between distant users and your Region.
• RDS read replicas spread read traffic and larger instance classes run queries faster, but both still execute the full query every time; only a cache returns a stored result without running it.
• RDS Proxy pools and shares database connections (useful for Lambda bursts); it does not store query results.
• DAX works only with DynamoDB.

Choose a partition key with high cardinality (many distinct values) that requests use evenly, because DynamoDB spreads data and throughput by partition key value and each partition supports at most 3,000 read units and 1,000 write units per second. A unique identifier such as a deviceId in a fleet-telemetry table is ideal when each request targets one device; an attribute with only a handful of values (a device model, a region, a Boolean flag), or a value that every writer shares at the same moment (the current time bucket), funnels traffic onto a few partitions, and those partitions throttle while the table's total capacity sits unused. Raising provisioned capacity or switching to on-demand mode does not lift the per-partition limit.

Patterns that spread a hot key

• Composite keys: if every read asks for one user within one tenant, a partition key such as tenantId#userId keeps the access pattern intact and spreads a large tenant across many values. Changing only the sort key leaves the hot partition key in place.
• Write sharding: append a random or calculated suffix (1 to N) to a busy key such as a date, so writes land on N different values. The trade-off is on the read side: a reader that needs everything for that key runs one Query per suffix and merges the results, which still reads only the matching items. A calculated suffix (for example, a hash of an ID) lets a reader find a single item again.

Hot keys matter on global secondary indexes too. A GSI has its own partitions, keyed by the index's partition key, and when a GSI cannot absorb its writes DynamoDB throttles writes to the base table. A GSI keyed on a low-cardinality attribute such as a device's firmware version can therefore cause PutItem throttling even when both the table and the index show spare provisioned capacity. The fix is the same: shard the index key (the attribute plus a suffix) and query each shard.

DynamoDB reads are eventually consistent unless you ask for strong consistency, are billed in 4 KB read units, and should use a Query on a key (never a Scan plus filter) for any frequent access pattern, paging with LastEvaluatedKey.

Because reads are eventually consistent by default, a read made right after a successful write can return the previous value; setting ConsistentRead to true on GetItem, Query or Scan returns the result of every write that succeeded before the read. Strong consistency is the smallest fix for a read-after-write problem, such as a library app that records a returned book and then, on the next screen, sometimes still lists it as on loan. Options on the write, such as ReturnValues, condition expressions or transactions, do not change how a later read behaves.

Read capacity units (RCUs) are counted per 4 KB, with item size rounded up to the next 4 KB:

• Strongly consistent read: 1 RCU per 4 KB.
• Eventually consistent read: half of that (one RCU covers two reads of up to 4 KB).
• Transactional read: 2 RCUs per 4 KB.

For example, a 6 KB item rounds to 8 KB, which is 2 units per strongly consistent read; 50 such reads per second need 100 RCUs, or 50 if eventual consistency is acceptable. Write capacity units work the same way per 1 KB.

Global secondary indexes support only eventually consistent reads, because they are updated asynchronously; a Query on a GSI with ConsistentRead true is rejected with a ValidationException. Local secondary indexes do support strong reads. When you must look up by a GSI attribute and see the latest data, query the GSI to find the item's key, then GetItem that key from the base table with ConsistentRead true. Changing the index's projection does not change its consistency.

Query vs Scan: reading only the items you need

A Query reads only the items that match a key condition, while a Scan reads every item in the table or index; both apply any FilterExpression after the read, so a filter shrinks the response but not the capacity consumed or the time taken.

A Query's KeyConditionExpression needs equality on the partition key and may add one condition on the sort key: =, <, <=, >, >=, BETWEEN or begins_with. Design sort keys so the key condition does the selecting: an ISO 8601 timestamp lets begins_with(sentAt, :day) read just one day, and putting a status at the start of the sort key (for example, OPEN#2026-10-29...) lets a Query select open items without a filter. A Query filter cannot reference the partition key or sort key attributes; that request is rejected. GetItem needs the complete primary key and returns one item.

When an access pattern does not use the table's key, the answer is usually an index rather than a better Scan: in a library's loans table keyed by copy ID, a GSI keyed by memberId (with the due date as sort key) turns a full-table Scan filtered on the member into a Query that touches only that member's loans. A ProjectionExpression trims the attributes returned but not the data read, and a parallel scan (Segment and TotalSegments) finishes sooner by splitting the work across workers but still reads everything.

Pagination

Each Query or Scan call reads up to Limit items or 1 MB of data, whichever comes first, and only then applies the filter. A page can therefore hold fewer items than Limit, or none, while more matches remain. The only reliable end-of-results signal is a response with no LastEvaluatedKey; to continue, send the next request with ExclusiveStartKey set to that value. Raising Limit does not remove the 1 MB cap and still pays for the items the filter discards; consistency settings do not change the order of Limit and filter.

A global secondary index (GSI) can use any partition and sort key and can be added or removed at any time; a local secondary index (LSI) keeps the table's partition key, swaps in a different sort key, and must be defined when the table is created.

These rows decide most design choices. Strong reads in a second sort order, within one partition key, point to an LSI: a telemetry table keyed by deviceId and timestamp can add an LSI sorted by alert level, provided the table has not been created yet. The 10 GB item-collection cap is the price: an LSI suits data where no single partition key value can grow that large. Because an LSI can never be dropped, a table that outgrows the cap (ItemCollectionSizeLimitExceededException) cannot be fixed in place; it needs a redesigned table, and a GSI, which has no per-key size cap, is the usual replacement where eventual consistency is acceptable.

Projections and sparse indexes

A projection (KEYS_ONLY, INCLUDE listed attributes, or ALL) decides which attributes are copied into an index, and it is fixed when the index is created. If a GSI query needs attributes it does not project, neither Select nor a ProjectionExpression can fetch them; create a new GSI with the right projection, move queries to it once it is active, then delete the old one. Projecting fewer attributes keeps an index smaller and cheaper to write.

DynamoDB writes an item to an index only when the item has the index's key attributes. If an application sets an attribute only while an item is in some state (say, an overdue flag set only on loans past their due date) and removes it afterwards, a GSI keyed on that attribute is sparse: it holds just those items, so even scanning the index reads a tiny fraction of the table.

DynamoDB protects concurrent writes by evaluating a condition expression and performing the write as one atomic step; if the condition is false, the write is rejected with ConditionalCheckFailedException. Any read-then-write sequence in application code leaves a race window, however consistent the read.

• Create only if new: PutItem replaces an existing item with the same key. attribute_not_exists(username) on the partition key makes the second of two simultaneous sign-ups fail instead of overwriting the first. ReturnValues such as ALL_OLD only reports what was overwritten, after the damage is done.
• Optimistic locking: give each item a version number; every save is a conditional UpdateItem requiring the version to equal the one the editor loaded, and the same update increments it. If someone saved in between, the condition fails and the app reloads the item so the change can be reapplied. No locks are held while users edit.
• Atomic counters: an update expression that refers to the stored value, SET views = views + :one or ADD views :one, makes DynamoDB do the arithmetic during the write, so concurrent increments are never lost. Counters are not idempotent: a retried call can count twice.

Transactions vs batches

TransactWriteItems commits up to 100 Put, Update, Delete and ConditionCheck actions all-or-nothing, and each action can carry its own condition; if any condition fails, nothing is written. A library checkout, for example, can mark a copy as on loan only if it is still available and add the loan to the member's record in the same transaction. BatchWriteItem only sends Puts and Deletes in one request for efficiency: it supports no conditions, no updates, and no all-or-nothing guarantee, because individual items can fail and come back as unprocessed.