Bulkheads

An appointment-booking service is used by 4,000 clinics and salons. Each server calls three outside services through one HTTP client with a pool of 50 connections: a payment provider for deposits, an SMS provider for reminders, and a calendar-sync service. Normally they need few connections at once (in flight = arrival rate × time per call, from Throughput, utilization & queueing):

Then the SMS provider slows to 10 seconds a message. Reminder calls now want 30 × 10 = 300 connections. They can't have 300, but within a couple of seconds they hold all 50, and every payment call queues behind them. Patients can't book, because of a reminder service nobody was watching.

The same thing happens one level up. A chain of 300 salons starts a bulk import of its old appointments. The import runs through the same servers and the same database connections as live bookings, and every other customer's booking page slows to a crawl. One heavy customer degrading everyone else is a noisy neighbor.

Both are the same failure: a shared resource with no limit per user. A bulkhead is that limit. It splits the resource into compartments, so one caller's trouble can fill its own compartment and no more. Unlike a circuit breaker, which stops calling a failing dependency (The circuit breaker pattern), a bulkhead caps how much of you any one thing can use, failing or not.

Inside one server, the fix is a separate pool, or a separate concurrency limit, per dependency. Size each from its normal load with room for bursts:

When a compartment is full, the next call must not wait forever. The usual choices:

• Reject at once. Resilience4j's semaphore bulkhead, for example, allows 25 concurrent calls by default and waits 0 ms for a free slot before refusing.
• Wait briefly in a small bounded queue, then refuse.
• Defer the work. A refused reminder goes back on a queue to be sent later; nobody needs it within the second.

Two ways to build the walls:

• A semaphore counts calls in flight and refuses past the limit. It is cheap and works with any threading model.
• A thread pool per dependency runs the calls on its own fixed set of threads with a bounded queue, so the caller's threads are never the ones stuck waiting, at the cost of extra threads and a hand-off.

Bulkheads pair well with breakers. The bulkhead keeps a slow dependency from taking more than its share while the breaker decides it is failing; then the breaker stops the calls altogether.

The bulk import is a different problem: not a slow dependency, but a big consumer competing for the same servers, queues and database. Here the walls go around kinds of work, so that the work that must stay fast gets capacity nobody else can take:

Each layer has its own wall, and each wall has an ordinary tool:

Prefer the limits your platform already enforces, such as container resource limits, database connection limits per user and gateway rate limits, over rebuilding them in application code.

Pools protect against one dependency or one kind of work. But some failures hit everything a server does: a bad deploy, a corrupt configuration, a request that crashes any server that handles it (a poison pill). Against those, the largest bulkhead is a cell: a complete, independent copy of the system that serves only some of the customers.

The design rules that make cells work:

• Partition by a key that most requests already carry, here the customer ID, so a request never needs two cells.
• Keep the router thin. It only decides which cell, because it is the one piece every request still shares.
• Share no state between cells, or the shared part becomes the new single point of failure.
• Deploy cell by cell. A bad release breaks the first cell it reaches and is rolled back before it reaches the rest.

Cells don't have to mean more hardware: 30 servers can become 6 cells of 5. What they cost is the work of routing, provisioning cells and moving customers between them. The scope of impact shrinks with the number of cells, so 20 cells means a 5% blast radius. Availability zones and regions are the same idea applied by the cloud provider (Redundancy & failure domains).

Cells give each customer one compartment shared with many others. Shuffle sharding does better with the same machines. Take 8 workers. Ordinary sharding splits them into 4 fixed shards of 2, and a customer whose requests break their workers takes down a quarter of everyone. Shuffle sharding instead gives every customer its own random pair of the 8, and there are 28 different pairs (8 × 7 ÷ 2).

Suppose one clinic's malformed requests crash both of its workers. Only customers with exactly the same pair, about 1 in 28, lose both. Everyone else shares at most one worker with it and still has a healthy one, as long as their clients retry on the other worker of their shard. Shuffle sharding only works with clients that retry like this.

The numbers grow fast. Amazon Route 53 spreads its DNS service over 2,048 virtual name servers and gives each customer domain 4 of them: about 730 billion possible shards, enough that no two customer domains share more than two name servers. A flood of traffic aimed at one domain overwhelms its four name servers, and every other domain keeps at least two that work.

A shared pool of 50 connections can lend any idle connection to whoever needs it. Pools of 20, 10 and 15 can't: on a normal day the payment pool may be full while the SMS pool sits idle. Bulkheads trade efficiency for isolation, and that trade has to be sized on purpose:

• Too small a compartment rejects ordinary bursts, so you have built an outage of your own.
• Too large, and it no longer protects anything.
• Too many walls make the system hard to size, monitor and reason about.

Watch every compartment separately. A bulkhead that is full or rejecting is either doing its job during an incident or too small in normal times; per-pool usage and rejection counts tell you which (Observability: metrics, logs & traces).