Redundancy & failure domains

A single point of failure (SPOF) is a component whose failure, alone, stops the service. To find them, trace one request and ask at every box: what if this dies right now?

Take an online shop. A shopper's request resolves the shop's name, reaches the load balancer and lands on one of three app servers, which asks the cache for the product and, on a miss, the database. The app tier passes the test: one server dies and two carry the load. The cache is one machine. When it dies, every product read falls through to a database that was sized only for the misses.

The obvious single points are databases and caches. The dangerous ones are the things nobody draws as part of the system: the name, the config, the certificate, and the box in front of all the copies.

A 2021 outage at one of the largest internet companies shows how deep the hiding goes. A maintenance command cut its data centers off from the network, and the internal tools needed to fix that depended on the same network. Engineers had to go to the buildings.

Put two cache nodes side by side and a read fails only when both are down at the same moment. If each is down 1% of the time, and the second is no more likely to be down because the first is, both are down 1% × 1% = 0.01% of the time: availability 99.99%. In general, n copies each available a give 1 − (1 − a)ⁿ.

A shortcut: when every input is made only of nines, add the nines. Two independent three-nines copies give six nines. Now the caveat that carries this whole article. The multiplication assumes the copies fail for unrelated reasons. That is false whenever they share a rack, a power feed, a deploy, a config value, a certificate, a dependency or a bug. Everything that follows is about making the assumption a little more true.

Active-passive means one copy serves and the other waits, and how warm the waiting copy is sets both the recovery time and the bill. A cold standby is backups and a plan: you rebuild, in hours or days. A warm standby runs scaled down with data replicating to it: failover is minutes plus a scale-up. A hot standby runs at full size, in sync, taking no traffic: failover is seconds, and you pay for a copy that produces nothing.

Active-active means every copy serves all the time. There is no failover moment, only less capacity, and you have proved every day that the second copy works. The price is that writes now happen in two places.

N+1, N+2 and 2N describe spare capacity rather than pairs; the terms come from data-center power design. N is what carries the load, N+1 has one spare unit, N+2 two, and 2N is a complete second set on its own distribution path. In data-center practice N+1 is the common choice and 2N the expensive one.

A worked example: your API needs 4 servers at peak, each 99% available.

• Exactly 4: the tier is up only when all four are. 0.99⁴ = 96.1%, about 28 hours down a month.
• N+1 (five servers, any four suffice): 99.90%, 42 minutes.
• N+2: 99.998%, under a minute.
• 2N (eight servers): 99.99999%, well under a second.

The first spare bought two nines for 25% more servers.

How the switch happens, who decides the primary is dead, and how you stop it coming back as a second primary is Failure detection & split brain.

A failure domain is a set of things that fail together: one host, the rack it sits in, the building, the region. Two copies inside the same domain share every failure of that domain, and the independence you multiplied with is gone. So spread copies across domains, and pick the domain by what you want to survive and what the distance costs you.

The cloud providers define the middle two the same way. An availability zone is one or more data centers with their own power, cooling and networking; a region is a group of them, usually three or more. Treat a zone as a single failure domain within a region. Zones are far enough apart that a fire takes one, and close enough that copies can replicate synchronously.

Why do systems with three copies of everything still go down? Because the copies are not three independent coin flips. They run the same code, receive the same config, trust the same certificate, call the same dependencies and read the same clock. These are correlated failures (also called common-mode failures), and each one takes every copy out at the same moment, which is exactly what 1 − (1 − a)ⁿ assumed could not happen.

The defenses match the causes. Roll out to one copy, then one zone, and watch before continuing; the 2019 firewall rule above skipped exactly that staging. Treat config pushes as deploys, with the same staging and rollback. Know your dependencies' dependencies. And test by actually turning a whole zone off: the provider in the 2023 outage had never fully taken that facility offline, so nobody knew which services lived only there.

Every copy costs three things:

• Money. A hot standby is twice the hardware for no extra output. Active-active is twice the hardware and does give you capacity, once built.
• Complexity. Failover logic, health checks, the runbook for the day the copy is the one that is wrong. Every added component can fail too.
• Agreement. Two copies of data have to be kept the same. Synchronous replication makes every write wait for the other zone (a millisecond or two) or the other region (tens of milliseconds). Asynchronous replication lets the copies drift, and a failover can lose the last writes.

That last cost is where the rest of this track goes. Once there are two copies and a network between them, you must decide which one is right when they disagree, and what a client may see meanwhile: Consistency models, CAP & PACELC and Consensus & leader election. Redundancy is the entry ticket to distributed systems, not a way around them.