System designCore10 min
Health checks & failover
A standby is only useful if something notices the failure, moves the traffic and convinces every client to follow. Health checks do the noticing, failover does the moving, and most outages that "should have failed over" went wrong in one of the steps in between.
1 · The problem
Failover is a chain of steps, and the outage lasts as long as all of them
An electric-vehicle charging network runs an app that drivers use to start a charging session. Behind it sit API servers behind a load balancer, and a database with a primary and a synchronized standby in another data center. At 18:02 the primary's host dies. The standby is up to date and ready. Drivers still can't start charging for four minutes.
Nothing was broken; every step simply took its time. These are the worst case of each step with this team's settings:
30 + 120 + 60 + 30 = 240 s. Each number is a setting or a measured worst case in this team's setup. None is unusual; the point is that they add up.
Each step is a separate design decision:
- Detect: how often to check, and how many failures count as dead.
- Decide: who is allowed to declare the old one dead, without creating two primaries (Failure detection & split brain).
- Switch: promote the standby, or stop routing to the broken copy.
- Redirect clients: DNS caches, connection pools and retries must all let go of the old address.
This page covers the checks themselves and how traffic moves at each layer. The kinds of standby, cold to hot, are in Redundancy & failure domains.
2 · Health checks
A health check must answer one precise question
"Is it healthy?" is several different questions, and each answer triggers a different action. Kubernetes makes the distinction explicit with three kinds of probe:
Probe | Question | If it fails |
|---|---|---|
Startup | Has the process finished starting? | Other probes wait; after too long, the container is restarted |
Liveness | Is the process stuck beyond repair, for example deadlocked? | The container is killed and restarted |
Readiness | Should it receive traffic right now? | It is removed from the load balancer's list, but keeps running |
The usual mistake is a liveness check that tests a dependency. The charging app's API servers once had a liveness probe that queried the database. During the four-minute failover every probe failed, so Kubernetes restarted every API server, which then spent a minute starting up after the database was back. A database outage became a longer API outage. Wrong liveness probes in Kubernetes cause exactly this kind of cascading failure.
Two endpoints, two questions
// Liveness: can this process still serve anything at all?
app.get('/livez', (req, res) => res.sendStatus(200));
// Readiness: send me traffic? Local checks only.
app.get('/readyz', (req, res) => {
const ok = startedUp && !draining && diskWritable();
res.sendStatus(ok ? 200 : 503);
});Readiness is also how a server leaves gracefully. Before a deploy stops it, it starts answering 503 on /readyz (draining), the load balancer stops sending new requests, and in-flight ones finish. Kubernetes' defaults check every 10 seconds and act after 3 failures in a row, so a server that hangs can keep receiving traffic for about 30 seconds unless those settings are tightened.
3 · How deep
Shallow checks miss sick servers; deep checks can fail them all at once
How much should a health check test? There are three depths, each catching more and risking more:
Depth | Tests | Catches | Risk |
|---|---|---|---|
Liveness | The process answers on its port | Crashed or frozen servers | Misses a server that answers but fails every real request |
Local | Resources only this server uses: disk, memory, its own worker pool | One broken server | Low: unlikely to fail on many servers at once |
Dependency | Calls the shared database, cache or payment provider | Expired credentials, a broken connection from this server | When the shared dependency is slow, every server fails together |
The last risk is the dangerous one. If every API server's check queries the database, a slow database makes every check fail at once, and the load balancer removes the whole fleet:
One shared dependency, three identical verdicts. The servers were fine and could have served cached or read-only answers, but the check took them all out.
Two safety nets exist:
- Fail open. When every target fails its checks, AWS's load balancers and Route 53 send traffic to all of them anyway, on the bet that a struggling server beats none.
- Keep fast checks local. Let the load balancer's checks test local health only, and let a slower central system judge dependency checks, with rate limits so automation never removes too many servers at once.
A partial dependency failure can slip between the two: servers flap in and out as their checks fail now and then, never all at once, so fail-open never triggers. Under overload a check can also fail just because the server is busy, and removing servers during a traffic surge shrinks the fleet exactly when it is needed. Answer health checks first, with spare workers kept for them.
4 · Gray failure
The worst failures pass their health checks
One API server's certificate for the payment provider expires. Its /health returns 200, because nothing in the check touches payments, but every charging session it handles fails at the payment step. Worse, failing fast makes it look idle, so a balancer that favors the server with the fewest requests in flight sends it more traffic. It becomes a black hole.
This is a gray failure: the component looks healthy to whatever checks it, and broken to the users of it. In large clouds, gray failure is behind most incidents. The defense is to judge servers by real traffic as well as by probes:
Passive checks watch real responses. Envoy, for example, ejects a host after 5 consecutive 5xx responses by default, ejects it for longer each time it happens again, and never ejects more than 10% of hosts at once.
Other real-traffic signals:
- Compare each server with its peers. A server whose error rate or latency is far above the fleet's is suspect, even if it can't tell on itself.
- Report the version in health responses, so a server that comes back after days offline running old code is caught.
- Use hysteresis. AWS's Application Load Balancer, by default, removes a target after 2 failed checks but returns it only after 5 good ones, so a flaky server doesn't flap in and out.
5 · Moving traffic
Failover moves traffic at every layer, and each layer has its own speed
The same idea, detect and redirect, happens at three layers with very different mechanics:
Layer | Who moves traffic | How fast | Watch out for |
|---|---|---|---|
A server | The load balancer stops sending to it | Check interval × failure count: seconds to a minute | Deep checks and gray failures, above |
A database primary | A standby is promoted; the endpoint's DNS name now points to it | Typically 1–2 minutes on Amazon RDS | Two primaries at once; apps must reconnect |
A whole region | DNS answers with the other region, or a global load balancer shifts traffic | Check interval plus DNS caching: minutes | The standby region must have capacity ready |
For a region, the charging network uses DNS failover with a primary and a secondary record. Health checkers in many locations probe the primary region, and its record is served while it is healthy:
In Route 53, the primary counts as healthy if more than 18% of its health checkers say so, so one checker location losing its network path can't fail a region over by itself.
Two details matter in the bad moments. If both primary and secondary look unhealthy, Route 53 returns the primary rather than nothing. And the switch is only as fast as clients forget the old answer: DNS caches keep it for up to the TTL, and some clients longer (DNS resolution).
6 · In practice
Clients decide when a failover really ends, and failback is a second failover
After the database's DNS name points to the new primary, the API servers are still holding pooled connections to the dead one. Existing connections must be re-established. Some Java setups cache DNS answers until the process restarts, which turns a two-minute failover into "restart every server"; the advice is a DNS cache TTL of 60 seconds or less.
Make the client side part of the design:
- Connection pools validate connections and give up on dead ones quickly.
- Timeouts and retries let a request that hits the old primary fail fast and retry on the new one (Timeouts, retries, backoff & jitter). Retried writes must be safe to repeat (Idempotency).
- Capacity is already there. A standby that must first scale up adds minutes; static stability means the failover only moves traffic.
- Writes that never replicated are lost with an asynchronous standby, which is why the charging network's standby is synchronous.
Once the old primary is repaired, moving back is a failback: a second failover, with all the same steps and risks. Do it deliberately, when the data has caught up and traffic is low, never automatically the moment a check turns green, or a flapping server bounces traffic back and forth.
- Measure each step
Know how long detection, promotion, DNS and reconnection take in your setup, and add them up.
- Test the checks
Break a dependency on purpose and confirm the fleet isn't removed and fail-open behaves as expected.
- Rehearse failover and failback
On a schedule, in production, while everything is healthy: the first real failover should not be the first failover.
Check yourself
4 questions
Takeaways
Remember this
- Failover time is the sum of detecting, deciding, switching and redirecting clients; measure each step.
- Liveness asks whether to restart, readiness whether to send traffic; never make liveness depend on a shared dependency.
- Fast load-balancer checks should test local health; dependency checks can remove a whole fleet at once, which is why balancers fail open.
- Gray failures pass their health checks, so also judge servers by real responses and against their peers.
- Clients end a failover: DNS caching, connection pools and retries decide when traffic actually moves, and failback deserves the same care.