System designCore10 min
Cache failure modes
A cache hides how much load the database would really get. The dangerous moments are when it stops hiding it: a busy key expires, requests ask for things that don't exist, many keys go at once, or one key outgrows its node.
1 · The problem
The database is sized for the misses, so losing hits multiplies its load
A live sports app is serving the cup final. At kick-off it takes 100,000 reads a second for scores, line-ups and match pages, and its cache answers 99% of them. The database sees the other 1%, 1,000 reads a second, and it was sized for about that, with some headroom.
Nothing about that database is wrong until the hit ratio moves. It sees only the misses, so a small drop in the hit ratio is a large jump in its load:
At 100,000 reads a second. Losing the cache is a 100× jump for the database, not a slowdown.
Each failure on this page is a way the hit ratio drops suddenly: for one key, for keys that never hit, for many keys at once, or for the whole cache. And a database that slows down under the extra load makes it worse, because requests that time out are often retried, adding load exactly when there is none to spare (Timeouts, retries, backoff & jitter).
2 · Failure 1
A stampede: one busy key expires and every reader rebuilds it at once
The final's live score page is one cache key, read 20,000 times a second. Its TTL runs out. Rebuilding the page takes the database about 100 ms, and every request in those 100 ms misses: about 2,000 of them. Each runs the same query and writes the same page back.
One expiry turns into 2,000 identical queries. If the database slows, the rebuild takes longer and even more requests miss.
That burst is a stampede, a cache-shaped thundering herd. It needs only a key that is read far more often than it takes to rebuild, which describes every popular page.
Fix | How it works | What it costs |
|---|---|---|
Request coalescing | Only one request per key rebuilds it; the others wait and share its result | Waiting requests are as slow as the rebuild |
A lock in the cache | The first miss takes a short lock and rebuilds; others serve the old copy or wait | A lock to expire safely if its holder dies |
Refresh early | Rebuild a little before expiry, with a chance that grows as expiry nears | Some rebuilds happen earlier than needed |
Serve stale while refreshing | Keep the expired copy, return it, and rebuild in the background | Readers see a copy a few seconds old |
Request coalescing, also called single flight, usually runs inside each app server. With 40 servers, the 2,000 queries become 40, one per server, not one. A lock held in the shared cache, or a lease like those in Cache invalidation & eviction, gets it down to one: on keys prone to this, leases cut one database's peak from 17,000 to 1,300 queries a second.
Refreshing early works best when it is random: each reader that finds the key near its expiry rebuilds it with a small chance that rises as the expiry nears, and scaled by how long the rebuild takes. One reader usually wins well before the deadline, and nobody waits.
3 · Failure 2
Requests for things that don't exist go straight through to the database
A scraper walks through match ids, most of which don't exist: /match/7341029, /match/7341030, and so on, 5,000 a second. Each one misses the cache, misses the database, and leaves nothing behind to cache, so the next request does the same. The cache's hit ratio for these requests is zero, by construction.
This is often called cache penetration. It can be an accident, a bug looping over bad ids, as easily as an attack. Two fixes, for two shapes of traffic:
- Cache the miss. Store not found for an id, with a short TTL. This is negative caching, and it stops repeated requests for the same missing id.
- Filter the ids. Random ids are each new, so caching misses doesn't help. A Bloom filter holding every real match id answers definitely not from memory for almost all of them, and only the rare false positive reaches the database.
A no from the filter answers 404 at once. Only ids that may exist go on to the cache and, on a miss, the database. Bloom filters explains the filter.
Behind both, the usual limits still apply: reject ids that are malformed or out of range before any lookup, and cap how fast one client can ask.
4 · Failure 3
An avalanche: many keys go at once, from a timer or a dead node
At 18:00 a deploy warms the cache with 100,000 match pages, all with a one-hour TTL. At 19:00, during the final, they all expire in the same second, and 100,000 rebuilds land on the database together. Many keys missing at once is an avalanche, and a shared expiry time is the most common cause.
The fix is cheap: add a random jitter to each TTL, so keys written together expire apart. With one hour plus a random 0 to 6 minutes, the 100,000 expiries spread over 360 seconds: about 280 a second instead of 100,000 in one.
The other cause is a cache node dying. With 10 cache nodes, one loss sends its tenth of the keys to other nodes that don't have them, and every one of those reads misses until the keys are rebuilt:
Misses go from 1% to 1% + 10% × 99% ≈ 10.9% of 100,000 reads: almost 11 times the database load, from losing one node in ten.
Fix | How it helps |
|---|---|
Jitter the TTLs | Keys loaded together expire apart |
Replicate cache nodes | A replica takes over a dead node's keys, already warm |
Warm before taking traffic | A new or restarted cache is filled before it serves, so it never starts cold |
A small local cache in each app server | The busiest keys survive a shared-cache failure for a few seconds |
5 · Failure 4
One hot key can overload the one cache node that holds it
The score page doesn't only stress the database when it expires. A shared cache spreads keys across its nodes, but each key lives on one node. At 20,000 reads a second, the score page's node does far more work than the other nine, and its network link and CPU become the limit for the whole page, however many nodes the cache has. A key like that is a hot key.
Without the local copies, node 3 takes all 20,000 reads a second for the page. With each server keeping it for one second, node 3 sees one read per server per second.
Two fixes, often used together:
- A local copy with a very short TTL. Each app server keeps the hot page in its own memory for a second. A score a second old is fine, and the node sees 40 reads a second instead of 20,000.
- Split the key. Store copies under
score:final#1toscore:final#8, which land on different nodes, and have each reader pick one at random. Every write has to update all eight.
The same skew hits databases and queues too; Hot partitions & skew covers it there.
6 · The defense
Design the database's protection as if the cache will fail, because it will
Every fix above lowers the odds of a sudden drop in hit ratio. None makes it impossible: nodes die, deploys flush, traffic shifts to something nobody cached. So the last layer protects the database directly, and assumes the cache is gone:
Protection | What it does when the cache fails |
|---|---|
Serve stale on error | Keep expired copies, and return them when the rebuild fails or times out |
Back off retries | Retries wait longer each time, with jitter, instead of tripling the load |
Cap the database's work | Reject or queue requests beyond what the database can serve, so the ones it accepts still succeed |
Fail fast on a struggling database | A circuit breaker stops sending requests for a while, so it can recover |
Capping the work, called load shedding, is what keeps the third row honest: a database sized for 1,000 reads a second that is sent 100,000 doesn't serve 1,000 and fail the rest: its queues grow, every request waits, and most time out, the ones it was able to serve included. Turning the excess away at the door keeps it serving what it can.
7 · In a real system
Watch a cache die at peak and the retries finish the job
URL Shortener System Design
Redis goes down; Cassandra takes every read
Run the scenario: Redis dies at peak, about 4,000 reads a second land on a Cassandra sized for 3,000, and retries without backoff push it to nearly 12,000. Then try the fixes, from backoff and coalescing to an in-process cache and pre-warming, and see which keep redirects working.
Check yourself
5 questions
Takeaways
Remember this
- The database is sized for the misses: a small drop in hit ratio is a large jump in its load, and losing the cache is 100× at a 99% hit ratio.
- A stampede is many readers rebuilding one expired key; coalesce, lock, refresh early or serve stale.
- Requests for missing ids bypass the cache: cache the miss for repeats, and filter ids for random ones.
- An avalanche is many keys missing at once: jitter TTLs, replicate and warm cache nodes.
- A hot key overloads one cache node; short local copies or split keys spread it, and the database still needs its own protection.