System designCore10 min
Caching strategies
Keep a copy of the data people read most somewhere faster, so most reads never reach the database. The hit ratio decides how much that buys you, and who writes the copy decides what it costs.
1 · The idea
A few menus get most of the reads, so a small copy answers most of them
A food delivery app lists 200,000 restaurants. At dinner time it opens a menu 50,000 times a second, and building one is a database query that joins dishes, prices and options, about 20 ms. Most of those reads are for a small set of restaurants: the busy, well-rated ones near where people are ordering.
A cache keeps copies of those menus somewhere faster than the database: a map in each app server's memory, or a shared cache such as Redis that every server asks over the network. A read that finds its data in the cache is a hit; one that doesn't is a miss, and goes to the database.
Caching works because reads are skewed. Measured web traffic follows a Zipf-like curve: the most popular item gets the most requests, the next fewer, and a long tail is rarely asked for. If menu popularity followed that curve exactly, with the second menu read half as often as the first and the third a third as often, caching a small slice would answer most reads:
Share of reads for the top k menus = (1 + 1/2 + … + 1/k) ÷ (1 + 1/2 + … + 1/200,000). The busiest 1% of menus answers 64% of reads; going from 10% to 20% of menus adds only 5 points.
Real traffic is flatter or steeper than this model, but the shape holds: the first slice of cache does most of the work, and each extra gigabyte buys less than the one before.
2 · The numbers
The hit ratio decides how fast reads are and how much the database sees
The hit ratio is the share of reads answered from the cache. Say the shared cache answers in about 1 ms, and a miss pays that 1 ms and then the 20 ms query. The average read costs hit ratio × 1 ms + miss ratio × (1 + 20) ms:
Average read time. At 90% hits, misses are a tenth of the reads but 2.1 of the 3 ms.
The database's view matters even more, because it sees only the misses. At 50,000 reads a second:
Hit ratio | Database reads per second |
|---|---|
50% | 25,000 |
90% | 5,000 |
99% | 500 |
Going from 90% to 99% sounds like a small step, and it cuts the database's read load ten times. That is also the danger: a database sized for 500 reads a second behind a 99% cache cannot survive the cache going away, which Cache failure modes takes up.
3 · The default
Cache-aside: the application checks the cache and fills it on a miss
The cache sits beside the read path, not in it: the application does every step.
- Read the cache
A hit returns the menu at once.
- On a miss, query the database
The application, not the cache, knows how to build a menu from its tables.
- Store it with a TTL
The next reader gets a hit. The TTL, time to live, is how long the copy may be served before it expires, which bounds how stale it can get.
- On a write, update the database, then delete the key
The next read rebuilds the menu. Why delete rather than overwrite, and the races that remain, are in Cache invalidation & eviction.
Cache-aside for one menu
async function getMenu(id: string) {
const key = `menu:${id}`;
const cached = await cache.get(key);
if (cached) return JSON.parse(cached); // a hit
const menu = await db.loadMenu(id); // a miss: build it
await cache.set(key, JSON.stringify(menu), { EX: 600 }); // keep 10 minutes
return menu;
}Cache-aside is the usual default, for three reasons:
- only menus someone asked for take up memory;
- the cache stays optional: if it fails, reads still work, only slower;
- any store with get and set can be the cache.
The cost is the miss. The first read of each menu, and the first after each expiry, pays the cache round trip and the full query.
4 · The alternatives
Who writes to the cache decides what you trade: freshness, write speed or safety
Cache-aside leaves everything to the application. The other patterns move the work around, and the delivery app has a use for each:
Pattern | How it works | Where the app uses it | Watch out for |
|---|---|---|---|
Read-through | The cache loads misses itself, through a loader you give it | A shared library for menus, so no service forgets the load | The cache library sits in every read |
Write-through | Each write goes to the cache and the database before it is acknowledged | A restaurant editing its menu sees the change at once | Every write pays both; data nobody reads still fills the cache |
Write-behind | Writes go to the cache and reach the database later, in batches | Counting how often each dish is viewed | Writes not yet flushed are lost if the cache fails |
Write-around | Writes go to the database only; reads fill the cache | Order receipts, written once and rarely opened | A read soon after a write is a miss |
Write-through keeps the cache current on every write, so reads right after a write hit. It is almost always combined with cache-aside: misses after an expiry or a restart are still loaded on demand. Write-around is what cache-aside does with writes anyway: they skip the cache, and only data someone reads gets cached.
Write-behind, also called write-back, earns its risk where writes are many and each one is worth little. A popular dish is viewed 2,000 times a minute; the cache adds the views up and writes one row a minute instead of 2,000 rows. Between flushes, though, the counts exist only in the cache's memory:
One database write a minute instead of thousands, at the price of losing whatever wasn't flushed. Fine for view counts; never for payments.
Persistence on the cache narrows that window without closing it. Redis can append each write to a file and sync it to disk every second, its default when the file is on, and then loses at most about a second of writes. Syncing on every write closes the gap, and gives up most of the speed write-behind was for.
5 · Where caches live
Every layer can cache, and the closer to the user, the more a hit saves
Layer | Example in the app | A hit saves |
|---|---|---|
Browser or phone | Restaurant photos, kept for a day | The whole network trip |
CDN edge | Photos and app files, near each city | The trip to your region |
Reverse proxy | Whole menu pages for visitors who aren't logged in | Your app servers' work |
In the app server | A map of the 1,000 busiest menus | A network hop to the shared cache |
Shared cache | Every menu read lately | The database query |
Database | Its own memory of recently used table pages | A disk read |
The two layers you build yourself trade differently. A map in the app server's memory answers in well under a microsecond, since a memory read is about 100 ns, against roughly half a millisecond for a round trip to a shared cache in the same data center. But each of the app's 40 servers keeps its own copy: the memory is spent 40 times, a restarted server starts cold, and a change has to reach all 40.
Two levels: the local map answers the very busiest menus with no network hop, the shared cache catches the rest, and the database sees only what both miss.
The nearer the user, the bigger the saving, and the harder it is to take a copy back: you can delete a key from your shared cache, but not from a phone. Content delivery networks (CDNs) covers caching at the edge.
6 · What to cache
Cache what is read often and changes rarely; read the rest from the source
A cache pays off when the same data is read many times between changes, and when a slightly old copy does no harm. Going through the app's data:
Data | Cache it? | Why |
|---|---|---|
Menus and photos | Yes, for minutes to a day | Read constantly, edited rarely |
Whether a restaurant is open | Briefly, for seconds | Read on every page, but a stale open sends orders to a closed kitchen |
Results for a typed search | Rarely worth it | Most searches are typed once, so their copies are never read again |
An order's status | No | Read by one person, changes every few minutes, must be current |
Payments and balances | Never | A stale answer is a wrong answer |
Two more costs to weigh before adding a cache:
- every cached item is a copy that can go stale, so each new cache is one more thing to keep fresh;
- a cache shared between users must never hand one user's data to another, so a key for personal data has to say whose data it is, or the data stays out of the shared cache.
7 · In a real system
One miss fills the cache for every click after it
URL Shortener System Design
Cache-aside on the URL shortener's read path
A short link that isn't in Redis is read from Cassandra and then written into Redis, so the next click on it is a hit. Step through the miss to see each hop.
Check yourself
5 questions
Takeaways
Remember this
- Reads are skewed, so a small cache of the most popular items answers most of them.
- The database sees only the misses: going from 90% to 99% hits cuts its read load ten times.
- Cache-aside is the default: read the cache, load misses, store with a TTL, and delete the key after a write.
- Write-through, write-behind and write-around trade write speed, freshness and durability; write-behind can lose unflushed writes.
- Cache what is read often, changes rarely and is safe to serve slightly old; keep personal data out of shared keys.