The circuit breaker pattern

An online pharmacy's checkout service handles 400 checkouts a second. Each one asks an outside courier company for available delivery slots, which normally takes 50 ms. The service has a pool of 200 worker threads, shared by every endpoint it serves: checkout, order history, prescription uploads.

One evening the courier's API stops answering. It doesn't return errors; it just doesn't respond, so every call waits for the full 5-second timeout. The number of requests in flight is the arrival rate times the time each one takes (Little's law, from Throughput, utilization & queueing):

Within half a second of the stall, all 200 threads are waiting on the courier. Order history and prescription uploads never call the courier, but they queue for the same threads, so they stall too. The pharmacy's mobile app calls the checkout service and runs out of connections in turn. This is a cascading failure: one slow dependency, many failed services.

Retries make it worse. Each timed-out call that is retried holds a thread for another 5 seconds and sends the struggling courier yet more traffic (Timeouts, retries, backoff & jitter). Async servers don't escape either; instead of threads, they run out of connections, memory or queue space. Timeouts limit how long one call can wait. Nothing yet limits how many calls wait at once on something that is clearly broken.

A circuit breaker wraps every call to a dependency and tracks how recent calls went. Once failures cross a threshold, it trips and fails new calls immediately, without touching the dependency at all. The name comes from the electrical breaker that cuts the current before the wiring overheats. It has three states:

In the pharmacy, the breaker around the courier API sees half of its recent calls time out and opens. From that moment a delivery-slot request returns an error in microseconds instead of 5 seconds. The threads come back, order history and uploads work again, and the courier stops receiving traffic it can't serve.

A breaker and a retry do different jobs:

• A retry assumes the failure is brief and tries again.
• A breaker assumes the failure will last and stops trying.

They combine well when the retry sits inside the breaker and gives up immediately when the breaker reports that it is open. Retrying a call the breaker refused only adds load.

An open breaker can't know when the courier is healthy again unless something tries. After a wait, say 30 seconds, it moves to half-open and lets a small number of real calls through:

Why not reopen the floodgates at once? A service that has just come back up may cope with a trickle of requests but fall over again under the full 400 a second. The half-open state ramps it up gently. Some breakers also lengthen the open wait each time a trial fails, from seconds to minutes, so a dependency that is down for an hour isn't probed every 30 seconds.

Instead of waiting on a timer, a breaker can also probe a health-check endpoint of the dependency. Either way, every state change is worth an event and a metric: an opened breaker is one of the clearest signals that a dependency is in trouble (Observability: metrics, logs & traces).

A breaker is only as good as its idea of "failing". Most libraries keep a sliding window, either the last N calls or the last N seconds, and compute a failure rate over it. Resilience4j, a widely used Java library, is a good reference for the knobs and their defaults:

Which outcomes count matters as much as the numbers:

• Count: timeouts, connection errors, 5xx responses, and 429 or 503 responses that say the dependency is overloaded.
• Don't count: client errors such as 400 or 404. A user typing an invalid ZIP code is not a sign the courier is down, and counting it lets bad input trip the breaker for everyone.
• Trip at once when a response says so explicitly, such as "overloaded, retry in 2 minutes".

Slow calls deserve their own rule. The courier in this story never returned an error; it hung. A breaker that waits for timeouts learns of the hang 5 seconds per call late. Counting calls slower than a threshold well under the timeout, say 1 second, as failures trips it sooner, and catches a dependency that is sick but not dead.

Failing fast only helps if the caller knows what to do with the failure. The answer depends on what the dependency was for:

The last row is the important one. Some dependencies have no safe substitute, and the honest fallback is a fast, clear error or accepting the work for later. A fallback that invents an answer for a critical check is a bug that looks like resilience.

Test fallbacks on purpose. They run only during incidents, so a broken fallback is usually discovered during one. Most libraries can force a breaker open, which also gives operators a manual switch when they know a dependency is down.

A breaker makes a hard decision for a whole group of calls. If that group is too broad, healthy calls are refused along with sick ones. Suppose the pharmacy's order database is split into 10 shards and one shard fails:

• One breaker for the whole database sees 10% of calls fail, or trips and refuses all ten shards. Either way it can't help the one shard that matters.
• One breaker per shard opens only for the broken shard, and the other 90% of customers never notice.

The same applies to separate hosts, regions or endpoints of one dependency: give each part that fails on its own its own breaker.

Other limits to keep in mind:

• Each caller judges alone. In most libraries the breaker lives inside each caller's process. With 50 checkout instances, each has its own breaker and its own window, so they trip at different moments. That is usually fine, but it means a low-traffic instance may never collect enough calls to trip.
• Rarely used modes are hard to test. A breaker adds states the system is almost never in, and those paths are the least exercised code you have. A breaker that is slow to close also adds recovery time after the dependency is fine.

A breaker doesn't replace the basics. It still needs timeouts to notice slowness, and it protects only the caller. The overloaded service protects itself with Load shedding & backpressure, and separate thread pools per dependency, so one slow dependency can't take all the threads, come from Bulkheads.