System designCore8 min
Timeouts, retries, backoff & jitter
Retries turn brief blips into successes, and done naively they turn a struggling service into a dead one. A timeout, a retry rule, a backoff with jitter and a budget get you the first without the second.
1 · Start here
A call without a timeout can hang its caller forever
A flight search site asks an airline-availability service for seats on every search. One evening that service stops answering, without failing: connections open, and nothing comes back. The search service has 200 worker threads. Each search that calls the hung service holds a thread while it waits, and within seconds all 200 are waiting. Searches for every route, not just this airline's, now queue behind them, and the site is down because one dependency is slow.
Every remote call needs a timeout: a limit after which the caller stops waiting and treats the call as failed. Pick it from the dependency's latency, not a round number. A timeout at a high percentile of healthy calls means only that share of healthy calls are cut off by mistake:
Latency of healthy calls to the availability service, and the timeout. A 1-second timeout, just above p99.9, cuts off about 1 healthy call in 1,000 and stops a hung call from holding a thread for more than a second.
A timeout at the median would cut off half the healthy calls; a 30-second default would let a hang hold threads for 30 seconds each. Better still is a deadline passed down the chain: if the user's search has 2 seconds left when it reaches the search service, a call two hops down shouldn't start a wait the user won't be around for.
2 · Retries
Retry what is likely to work next time, and only what is safe to repeat
A retry sends a failed call again. It pays off when the failure was brief: a dropped connection, a server restarting, one overloaded instance behind a load balancer. It is useless when the request itself is wrong, and dangerous when the first attempt may have done its work:
Failure | Retry? | Why |
|---|---|---|
Connection reset, or a timeout on a read | Yes | Probably a blip, and a read is safe to repeat |
503 or 429, overloaded | Yes, after waiting; honor | Hammering it now makes the overload worse |
400, 404, a validation error | No | It fails the same way every time |
A timeout while booking a seat | Only with an idempotency key | The first attempt may have booked it; a plain retry books twice |
The last row is the one that hurts. A timeout means I don't know, not it failed: the booking may have succeeded and the answer been lost. Retrying a write is safe only if the server can recognize the repeat, which is what an idempotency key does (Idempotency).
3 · The catch
Retries multiply down the stack, and peak when the service is weakest
The search site has three layers: the web app calls the search service, which calls the availability service, which queries the seat database. If each layer tries up to 3 times, one search against a failing database turns into 3 × 3 × 3 = 27 queries, exactly when the database is struggling. With five layers it would be 3⁵ = 243.
Each layer multiplies the attempts of the layer above. One click becomes 27 queries at the bottom.
Even at one layer, retries add load in proportion to the failures. With up to 3 attempts and a failure rate f, each request costs 1 + f + f² attempts: nothing extra when all is well, and three times the load when the service is completely down, the moment it can least afford it. A surge of retries that keeps a service from recovering is a retry storm.
- Up to 3 attempts: 1 + f + f²
- With a 10% retry budget
Without a cap, retries triple the load on a dead service. A budget that lets retries add at most 10% of normal traffic keeps it at 1.1×, however bad the failure.
Two rules contain it:
- Retry at one layer. Usually the one closest to the failing service, or the one that knows the call is safe. The others pass the failure up.
- Give retries a budget. Retry only while retries are a small share of a client's traffic, such as 10%. When most calls fail, the budget runs out and the client fails fast instead of adding load.
4 · Waiting between tries
Back off exponentially, and add jitter so clients don't retry in waves
Retrying at once rarely helps a struggling service. Exponential backoff waits longer after each failure, doubling up to a cap: 100 ms, 200 ms, 400 ms, 800 ms. That gives the service room to recover. But if 1,000 clients failed at the same moment, they all wait the same 100 ms, and retry at the same moment too:
- Backoff, no jitter
- Full jitter
1,000 clients fail at once. Without jitter they all retry at 100 ms, one spike; with a random wait between 0 and 100 ms, about 100 arrive in each 10 ms.
Jitter adds randomness to each wait, so clients that failed together retry apart. The common versions:
Strategy | Wait before retry n |
|---|---|
Backoff, no jitter | min(cap, base × 2ⁿ) |
Full jitter | A random time between 0 and min(cap, base × 2ⁿ) |
Equal jitter | Half of min(cap, base × 2ⁿ), plus a random time up to the other half |
Decorrelated jitter | min(cap, a random time between base and 3 × the previous wait) |
In simulations of many clients competing for one resource, plain backoff did far more work and took far longer than any jittered version, and full jitter needed the fewest calls. It is also the simplest to write:
Backoff with full jitter, a budget and a deadline
const base = 100, cap = 10_000, attempts = 3;
for (let attempt = 0; attempt < attempts; attempt++) {
try {
return await call({ deadline });
} catch (e) {
const last = attempt === attempts - 1;
if (last || !isRetryable(e) || !retryBudget.allow()) throw e;
const ceiling = Math.min(cap, base * 2 ** attempt);
await sleep(Math.random() * ceiling); // full jitter: anywhere from 0 to the ceiling
}
}5 · Knowing when to stop
When a dependency is down, stop asking it
A budget is a rule about when not to retry, and gRPC builds one in. Each client keeps a token count, starting full. Every failed call takes away one token; every successful call adds back a fraction. While the count is above half, failed calls may be retried. When most calls fail, the count drains below half, and retries stop until successes refill it.
Mechanism | What it decides | How it recovers |
|---|---|---|
Per-request limit | At most 3 attempts for one call | Each new call starts again |
Retry budget | Retries only while they are a small share of traffic | Successes earn retries back |
Circuit breaker | Stop calling a failing dependency at all, for a while | Lets a trial call through, then resumes |
A retry budget keeps a trickle of retries going even in a bad patch, and returns to normal by itself as calls succeed. A circuit breaker cuts calls off completely and probes before reopening, which suits a dependency that is plainly down; The circuit breaker pattern covers it. A service can also tell its callers to stop: an overloaded, don't retry error lets the layer above give up instead of piling on.
6 · In a real system
A cache dies, and retries finish off the database
URL Shortener System Design
Retries without backoff triple the load on Cassandra
When the URL shortener's cache dies at peak, about 4,000 reads a second fall on a Cassandra sized for 3,000. Timed-out reads come straight back as retries, and the load climbs to nearly 12,000. Try backoff at the checkpoint and see what it trims, and what it can't.
Check yourself
5 questions
Takeaways
Remember this
- Every remote call needs a timeout from the dependency's high percentile, such as p99.9, and a deadline passed down the chain.
- Retry transient failures of calls that are safe to repeat; a timed-out write needs an idempotency key.
- Retries multiply across layers and peak when the service is down: retry at one layer, within a budget.
- Back off exponentially up to a cap, with full jitter, so clients that failed together don't retry together.
- When most calls fail, stop: a retry budget drains, a circuit breaker opens, and an overloaded service can say don't retry.