Failure detection & split brain

A city library runs its lending system on a primary database in the main building, with a standby copy in a branch across town, ready to take over if the primary dies. One afternoon a construction crew cuts the cable between the two buildings.

This is a network partition: machines are split into groups that can talk within the group but not across it. It is not a crash; everyone is still running. Partitions come in awkward shapes too:

• Partial: A can reach B and B can reach C, but A cannot reach C.
• One-way: A hears B, but B cannot hear A. A faulty network card that drops incoming packets but still sends outgoing ones does exactly this.

Partitions are not rare exotic events. Links fail every day in ordinary data centers (see Why distributed systems are hard), and any system with more than one machine has to decide what to do when one happens.

The standard way to watch another machine is a heartbeat: a small message it sends every second or so. If the heartbeats stop for longer than a timeout, you declare it dead. The library's standby does exactly this:

There is no timeout that is both fast and never wrong. A smarter detector learns how regular the heartbeats usually are, and instead of a yes-or-no answer reports a suspicion level: how unlikely this much silence would be if the machine were alive. Each system then picks its own threshold, for example "act when there is less than a 1-in-1,000 chance that it is still fine".

Two machines can simply watch each other. A cluster of 1,000 cannot: if every node heartbeats every other node, that is 1,000 × 999, about a million messages each round, and the load grows with the square of the cluster.

So large clusters use gossip instead:

• Check a random neighbor. Each round, every node pings one other node chosen at random.
• Ask for a second opinion. If it gets no answer, it asks a few other nodes to try. Maybe only its own link is bad.
• Suspect before declaring dead. A node that still fails is marked suspected, and it gets a short window to answer "I'm alive" before the verdict.
• Spread the news like a rumor. Every node passes what it has heard to a few others, who pass it on. The news reaches all 1,000 nodes in a handful of rounds, because each round the number of informed nodes multiplies.

Back at the library. The standby's timeout fires, so it promotes itself to primary, which is exactly what it was built to do. But the old primary was never dead:

This is split brain: two nodes each acting as the leader. Every system with automatic failover is exposed to it, and the damage is not a pause but conflicting data. It happens to large companies too: in 2018 a 43-second network cut made one large code-hosting site's tooling promote a new primary database in another region. Both sides then held writes the other lacked, and putting them back together took over a day.

A node that cannot prove it is still in charge must stop acting as if it is.

Two ideas, usually used together, make split brain impossible rather than unlikely.

1. Only a majority may act. Run an odd number of nodes, say three: the primary, the standby, and a small third witness somewhere else. A node may lead only while it can reach a majority, two of the three. When the cable is cut, at most one side can have two. The other side knows it is in the minority and stops accepting writes.

With only two nodes, neither side of a cut has a majority, which is why the third one matters. How nodes agree on a leader this way is Consensus & leader election.

2. Stamp every leader with a number. Each time a new leader is chosen, it gets a number higher than any before: a fencing token. It sends that number with every write, and the storage remembers the highest number it has seen and refuses anything lower. An old leader that wakes up late cannot do damage:

The bluntest version of the same idea has a memorable name, STONITH: "shoot the other node in the head". Before taking over, the new leader cuts the old one's power or network access, so it cannot write anything at all.