What is caching? A plain explanation for cloud architecture

The plain idea, and why it matters

Some work is expensive to repeat — querying a database, rendering a page, calling a slow external API — while re-reading a copy of the answer you already computed is cheap. Caching exploits that gap: the first time something is requested, the system does the expensive work and keeps a copy nearby; every subsequent request for the same thing is served from that nearby copy instead.

The payoff is twofold. Responses get faster, because reading from a fast, local store beats redoing the original work every time. And the systems doing the original work — a database, an origin server, a third-party API — see less load, because a smaller share of requests actually reach them. Both effects compound: faster, cheaper, and more resilient to sudden spikes in traffic.

Where caching happens across a system

A single request in a modern web application can pass through several layers of caching before it ever reaches the real data:

• Browser cache — the visitor’s own browser stores images, scripts and other assets locally, so a second visit to the same page needs far less downloading.
• CDN / edge cache — a content delivery network keeps copies of static (and sometimes dynamic) content at locations close to visitors around the world, discussed further in our explainer on CDNs.
• Application-layer cache — the application itself stores computed results, such as a rendered page fragment or an API response, so it does not recompute them on every request.
• In-memory caches — a dedicated fast store, kept entirely in memory rather than on disk, sits between the application and slower systems for the hottest, most frequently accessed data.
• Database cache — the database engine itself keeps recently used data in memory so repeated queries against the same rows do not always hit disk.

Hits, misses, and the hard part: invalidation

A cache hit means the requested data was already in the cache and could be served immediately; a cache miss means it was not, so the system has to do the original, slower work — and usually stores the result for next time. A high hit rate is the whole point of caching; a low one means the cache is not earning its keep.

The genuinely hard part is deciding when cached data has gone stale. The underlying data changes — a price updates, a record is edited — but the cached copy does not know that on its own, so somebody has to invalidate or refresh it. Get this wrong and users see outdated information confidently served as current, which is arguably worse than being slow. This is why cache invalidation is famously cited as one of the hardest problems in computing: it is conceptually simple and practically full of edge cases.

TTLs and eviction, conceptually

Two mechanisms keep caches manageable. A time-to-live (TTL) is an expiry timer attached to each cached item — after it elapses, the item is treated as stale and refetched, which is a simple, blunt way to bound how out of date data can get without anyone having to explicitly invalidate it. Eviction is what happens when a cache is full and something new needs room — the cache discards existing entries, typically the ones least recently or least frequently used, on the reasonable assumption that data nobody has asked for in a while is the least likely to be asked for next.

Cloud caching services

Cloud providers offer managed caching so teams do not have to run their own caching infrastructure. On AWS, ElastiCache provides managed in-memory caching, and CloudFront is AWS’s CDN for edge caching of content. Azure’s equivalent in-memory service is Azure Cache for Redis. Google Cloud offers Memorystore for the same purpose. All of them handle the operational work — provisioning, patching, scaling — behind a managed service, which is precisely the appeal cloud platforms bring to a concept that predates the cloud by decades.

The honest trade-off, and caching in cert study

Caching is not free speed — it trades raw performance for added complexity and a real risk of serving stale data. Every cache is another moving part that can fail, another place data can drift out of sync with the source of truth, and another thing to reason about during an incident. Used well, that trade is clearly worth it; used carelessly, it turns "why is this data wrong" into a much harder question to answer.

Caching is a recurring pattern in the AWS Solutions Architect – Associate (SAA-C03) exam, which weighs it under both performance and cost-optimisation: a well-placed cache reduces load on expensive backend resources while improving response times, and knowing when to reach for ElastiCache versus CloudFront versus a database-level cache is exactly the kind of design judgement the exam tests.