Load shedding & backpressure

Friday, 7 p.m. A food delivery app's order service can finish 2,000 requests a second, and on a normal evening 1,500 arrive. Then a discount lands on every phone at once, and for half a minute 3,000 a second arrive. The extra 1,000 a second have to go somewhere. With nothing to stop them, they join a queue that grows by 1,000 every second.

The phone app gives up after 2 seconds. Follow the queue (Throughput, utilization & queueing has the full arithmetic):

• Every second of the surge, the requests being answered have waited a third of a second longer.
• Six seconds in, the wait passes 2 s. From then on, every answer reaches a phone that has stopped listening.
• The server is still 100% busy, finishing 2,000 requests a second. Not one of them is still wanted.

Useful work has its own name: goodput, the requests answered correctly and soon enough to matter. Throughput counts everything the server finishes, wanted or not. Under overload the two split apart:

Real systems fall further than the chart, because overload feeds itself:

• Retries. A phone that gave up tries again, so the surge grows.
• Crashes. Waiting requests hold memory until a server dies. Its load moves to the others, which then die sooner.
• Recovery is harder than the fall. Every crash cuts capacity. A service that was healthy at 10,000 requests a second and began crashing at 11,000 will almost certainly keep crashing at 9,000.

A failure that outlives the thing that started it is called a metastable failure: the trigger is gone, but retries, crashes or a cold cache keep the load above what is left. The way out is to cut the load well below what the service used to handle. The cheaper way is never to let the queue grow in the first place.

Load shedding means refusing part of the traffic on purpose when a server nears its limit. Each refused request gets an immediate 503 Service Unavailable, ideally with a Retry-After header saying when to come back. The phone can show "Busy, try again in a moment" at once, instead of a spinner that ends in a timeout two seconds later.

In the chart, one customer in three sees that message during the surge. That sounds bad until you compare it with the alternative, where everyone waits and nobody's order goes through.

Refusing has to be cheap, or it adds to the overload:

• Refuse early: at the gateway, or in the first lines of the service, before reading the body, checking the token or touching the database.
• Refuse at more than one layer: the gateway protects everything behind it, and each service knows its own state and protects itself.
• Remember that refusing is never free: for a tiny request, saying no costs nearly as much as answering. Flood a server with enough of them and it spends most of its CPU refusing.

When should a server start saying no? Not at a fixed number of requests per second. Requests differ in cost, and a fixed number goes stale every time you add servers or the database slows down. Better signals measure the server itself:

The in-flight limit comes from Little's law: in flight = throughput × time inside. The order service finishes 2,000 requests a second at about 50 ms each, so at full speed 2,000 × 0.05 = 100 requests are inside it. Set the limit near 100 and refuse the 101st at once.

The limit also adjusts itself. If the database slows and requests take 100 ms, the same 100 slots let in only 100 ÷ 0.1 = 1,000 a second: exactly what the service can now finish. A fixed limit of 2,000 a second would keep admitting twice too many. Adaptive concurrency limits go one step further: like TCP's congestion control, they raise the limit while latency stays flat and cut it as soon as latency starts to climb.

Shedding is not the same thing as rate limiting, and you need both:

A rate limit can't save a server that thousands of clients, each within their quota, push past capacity together. Shedding can't tell a runaway script from a paying customer. Rate limiting algorithms covers the first; the rest of this page covers the second.

Refusing at random treats a customer pressing Pay the same as the app quietly fetching "restaurants you might like". They are not equal. Give every request a priority (also called criticality) where it enters the system, and shed from the bottom up:

Health checks get their own row for a reason. A busy server that ignores the load balancer's check is marked dead and taken out. Its traffic moves to the others, which are just as busy, and they fail their checks too. Shedding meant to save the fleet ends up shrinking it.

Three details make priorities work:

• The priority travels with the request. When the order service calls payments, the call keeps the order's priority, so payments sheds in the same order.
• Shed in order. Lower priorities are refused first and at lower load; critical requests only when nothing else is left.
• Degrade before you refuse. Graceful degradation answers with something cheaper instead of nothing: popular restaurants from a cache instead of a personal ranking, or a tracking map that updates every 30 s instead of every 5.

It works at scale. Netflix gives every request a priority, both at its API gateway and inside its services. During one infrastructure outage, with prefetch traffic up about 12 times, requests from people actually pressing play stayed above 99.4% available.

The worst stretch of the chart was a server working flat out for nobody. A server can't see a phone give up, unless the request carries that moment with it: a deadline, the time after which the caller no longer wants the answer.

Each hop passes on the time that is left, not a fresh timeout. If the gateway receives 2 s and spends 0.1 s, it sends 1.9 s onward. Before starting any expensive step, a server checks what is left, and refuses at once if the work can't finish in time:

Inside one server, the queue itself can drop stale work:

• Bound the queue by age. A request that has waited longer than its caller will wait is thrown away unanswered.
• Serve the newest first once a queue forms. Requests normally go in arrival order. When a queue builds, the oldest are the likeliest to be abandoned, so adaptive LIFO (last in, first out) spends the server on requests that can still succeed.

A third trick gives a standing queue little patience. Facebook runs a variant of the CoDel algorithm: a request may queue for 100 ms if the queue was empty recently, but only 5 ms if it hasn't been empty for 100 ms. Short bursts still queue; a queue that never drains is cut down fast.

LIFO is unfair on purpose: under a long overload the oldest requests may never be served. That is right for people staring at a screen, and wrong for work that must happen eventually, like sending a receipt. That kind of work needs the next idea.

Shedding throws work away. Some work must not be lost: every placed order needs its receipt email. Here the answer is backpressure: the overloaded stage tells the stage before it to slow down, and the work waits instead of disappearing. There are three ways to say it:

The delivery app sends receipts through an event log. The email provider accepts 50 emails a second, but at the dinner peak orders create 120 receipts a second. The receipt worker pulls only what it can send, so the other 70 a second wait in the log:

Backpressure has to end somewhere. It can end at a durable buffer built to hold a backlog, like the log above. Or it ends at a person, as a 503 and a "try again". What must not happen is a chain of services that each simply block: every waiting caller holds a thread and memory, so the slowdown climbs back up the chain until the front door stops too.

Callers can push back on themselves too. In client-side throttling, each client counts, over the last two minutes, the requests it sent and the ones the server accepted. Once requests pass K times the accepts (K = 2 is typical), the client refuses some new requests itself, before they cost the server anything. It refuses each new request with probability:

• p = max(0, (requests − K × accepts) ÷ (requests + 1))

A client that sent 1,000 requests and had 300 accepted refuses (1,000 − 600) ÷ 1,001 ≈ 40% of its new ones locally. As the server recovers, accepts rise and the client lets traffic back through. Retries need the same discipline: a retry budget, and an "overloaded, don't retry" answer, stop refused requests from coming straight back (see Timeouts, retries, backoff & jitter).

Shedding keeps a service alive through a surge it can't serve. It does not make the surge servable, and it brings traps of its own:

Above all, shed requests are real customers getting an error. Count them, alert on them, and treat a steady shed rate as a capacity problem to fix, not as a feature quietly doing its job.