System designCore8 min
Why distributed systems are hard
You tap Pay for a coffee, the spinner turns, and nothing comes back. Were you charged? That question, which no amount of cleverness can answer for sure, is the heart of every distributed system.
1 · The shift
Many computers, no shared memory, no shared clock
A distributed system is several computers that work together by sending messages over a network, while trying to look like one system to the people using it. They share no memory and no clock. Everything one of them knows about the others, it learned from a message.
Why build them at all, when one computer is so much simpler?
- More capacity than any single machine can offer.
- Surviving failure: when one machine dies, the others carry on.
- Being close to users, on several continents at once.
Paying for one coffee crosses four machines run by three companies. Every arrow is a message that can be delayed or lost.
Inside one program, calling a function is simple: it returns, or the whole program crashes. Across a network, a "call" is a message out and a message back, and that changes everything.
2 · Partial failure
Some parts fail while others carry on, and you may not know which
On one computer, failures are mostly all-or-nothing. In a distributed system you get partial failure: some machines or links break while the rest keep working. Here is one way the coffee payment can go:
The money moved. Only the reply was lost. But the coffee app cannot see that.
From the coffee app's side, all of these look exactly the same, a request sent and silence after it:
What really happened | Were you charged? |
|---|---|
The request was lost before it arrived | No |
The payment service crashed before charging | No |
It charged you, and the reply was lost | Yes |
It is just slow, and will answer in a minute | Probably, soon |
3 · Living with it
Everything else is a way of acting without knowing
No clever trick removes that uncertainty. What distributed systems do instead is act sensibly in spite of it. Most of the rest of this track, and much of system design, follows from that one fact:
- Timeouts. At some point you stop waiting and assume the worst. It is a guess, and sometimes a wrong one.
- Retries. Try again, but a retry may repeat something that already happened: a second charge. So operations are made safe to repeat, with a unique key the server can recognize.
- Failure detection. Machines check on each other and decide who is dead, knowing they will sometimes be wrong. That is Failure detection & split brain.
- Agreement. Several machines agree on one answer even though some may fail mid-conversation. That is Consensus & leader election.
For the coffee, the practical answer is: retry with the same payment key, so the payment service says "already paid" instead of charging twice, and let the customer see a pending state instead of an error.
The retry carries the same key, so the payment service recognizes the repeat and answers instead of charging twice.
4 · The limit
No number of confirmations makes both sides sure
Could more messages remove the doubt? Picture two friends arranging to meet by text, on a network that sometimes loses messages. They must both turn up, or neither:
Whoever sent the last message never knows it arrived. Add another confirmation, and its sender has the same doubt.
This is the Two Generals problem, and it has been proved: when messages can be lost, no fixed number of messages guarantees both sides are certain they agree. Real systems do not try. They make disagreement survivable instead: a repeat is harmless, a timeout leads to a safe default, and a record that settles things later is kept somewhere both sides trust.
5 · The fallacies
Eight comfortable assumptions about the network, all false
A famous list from the 1990s names the assumptions that quietly break distributed designs. Each one is easy to make, because inside a single computer it is true:
The assumption | The reality | What it breaks |
|---|---|---|
The network is reliable | Links and switches fail every day, even in well-run data centers | Any design with no plan for lost messages |
Latency is zero | Every round trip costs time, far more across continents | Chatty designs that make many small calls |
Bandwidth is infinite | Links fill up | Sending large payloads that could have been small |
The network is secure | Traffic can be read or forged, even inside | Unencrypted internal calls |
Topology doesn't change | Machines and routes come and go | Hard-coded addresses |
There is one administrator | Many teams and companies own parts of the path | Assumptions about settings you do not control |
Transport cost is zero | Encoding data takes processor time, and moving it costs money | Budgets (see Estimation, capacity & cost) |
The network is homogeneous | Different hardware, versions and protocols meet | Code that works only on the machines it was tested on |
The first one is the most expensive to get wrong, and the evidence against it is plentiful. A large company's data centers see, at the median, about 40 link failures a day, and the network's built-in redundancy recovers only about 40% of the traffic those failures would cost.
6 · Failure models
Decide which failures you are designing for
You cannot defend against everything, so every design starts from a failure model: a statement of how its parts are allowed to fail.
Model | How a machine may fail | What defending against it costs |
|---|---|---|
Crash-stop | Stops and never comes back | The simplest: replace it |
Crash-recovery | Stops, then restarts, having lost what was only in memory | Write important state to disk before saying "done" |
Omission | Keeps running, but some messages are lost | Timeouts and retries |
Byzantine | Anything at all: wrong answers, even deliberate lies | Many more machines: at least 3f + 1 to survive f liars |
Most systems inside one company assume crash-recovery machines on a network that loses and delays messages. They do not assume machines lie, because they all run the company's own code. Guarding against liars costs so much that it is saved for settings where the parties do not trust each other, such as open networks of strangers, or controls where a faulty sensor must never be believed.
Check yourself
5 questions
Takeaways
Remember this
- A distributed system is many machines with no shared memory or clock, cooperating only by messages.
- Failures are partial: some parts break while others carry on.
- You cannot tell slow from dead, or a lost request from a lost reply; timeouts, safe retries, failure detection and consensus all follow from that.
- No number of confirmations makes two sides certain over a lossy network, so design for disagreement to be survivable.
- Pick a failure model: most internal systems assume crash-recovery machines and a lossy network, not liars.