Content delivery networks (CDNs)

A news site runs its servers in Virginia. A story breaks, and readers in Sydney open it. Between them and Virginia is about 200 ms of round trip, measured between cloud regions, and nothing the site does to its servers can shorten it.

Round trips add up before a single byte of the story arrives. A browser opening a new secure connection needs:

• one round trip for the TCP handshake;
• one for the TLS 1.3 handshake;
• one for the request and the first bytes of the answer.

That is three round trips, 3 × 200 = 600 ms, for the page alone, and the page then asks for images, scripts and styles.

A content delivery network (CDN) is a fleet of servers, called edge servers, spread across hundreds of cities. Readers connect to the nearest one, and it answers from its own cache. The site's own servers, the origin, are asked only for what the edge doesn't have.

Most CDNs work this way, as pull CDNs: they fetch from the origin on demand. A push CDN gets files uploaded ahead of time instead, which suits large, predictable content such as a catalog of video files, and saves the first reader's miss.

The hit ratio is the share of requests an edge answers from its cache. The origin sees the rest, so small changes near the top matter a lot. For a million reads of the story's image:

Going from 99% to 99.9% looks like a rounding error and cuts the origin's share ten times. The ceiling is set by the lifetime: an edge holding the image for 10 minutes asks the origin no more than once every 10 minutes, however many readers it serves. With 300 edges, that is still up to 300 fetches per 10 minutes, one per city with readers.

Two features push the origin's share down further:

• A second cache tier, often called an origin shield: edges that miss ask a regional cache near the origin, and only that cache asks the origin.
• Request collapsing: when thousands of readers ask for the same uncached file in the same moment, the edge sends one request to the origin and answers all of them from it. Without it, the first second of a viral story becomes thousands of identical origin requests, a stampede like those in Cache failure modes.

An edge stores each answer under a cache key, usually the host and the full URL. Two requests share a copy only if their keys match, so anything that makes keys differ quietly lowers the hit ratio:

Check the defaults of the CDN you use. One large CDN, for example, doesn't cache HTML or JSON unless told to, and never stores a response that sets a cookie or says private, no-store, no-cache or max-age=0.

The origin says how long a copy may be reused in the Cache-Control response header, a list of directives that browsers and every shared cache on the way obey. The ones that matter on a CDN:

• max-age: how long browsers and the CDN may reuse the answer;
• s-maxage: a separate lifetime for shared caches such as the CDN, so the edge can keep a page for a minute while browsers check every time;
• stale-while-revalidate=30: after the lifetime ends, serve the old copy for up to 30 more seconds while fetching a new one in the background;
• stale-if-error: keep serving the old copy if the origin is failing;
• immutable: this URL will never change, so don't even check on reload.

A purge tells the CDN to drop a URL now, and on modern CDNs it is fast: one reports a median under 150 ms worldwide for purges by tag or prefix. But a purge reaches only the CDN. Browsers, and any other cache that stored the file under its max-age, keep their copies. That is why long-lived files get a new name for each version instead of being purged: the page simply points at the new URL, and the old copies are never asked for again.

The last row is the expensive mistake. On Christmas Day 2015, during an attack on the Steam store, a caching rule changed in a hurry served store pages built for one user to other users. For about 90 minutes, pages for roughly 34,000 people, some with billing addresses and purchase histories, could be seen by strangers. A shared cache cannot tell whose page it holds unless the key or the headers say so.

The same reach helps when traffic spikes for any reason. The CDN's network is far larger than the origin, so a crowd, or many kinds of attack, lands on the edges first, and the origin sees only the misses. That is also the reason to watch the hit ratio during a spike: a cache key that suddenly varies turns the CDN's capacity back into the origin's problem.