System designDeep dive9 min
Coordination services
Every distributed system has a few facts that all its nodes must agree on: who leads, who is alive, who owns which piece of work, which configuration is current. Coordination services such as ZooKeeper and etcd keep those facts in a small, strongly consistent store, tie them to the liveness of the processes that wrote them, and tell everyone when they change.
1 · The idea
A coordination service holds the few facts every node must agree on
A log-ingestion pipeline has 30 workers sharing 512 partitions of incoming logs. One process, the coordinator, decides which worker reads which partitions. The pipeline needs answers that every machine agrees on: who the coordinator is right now, which workers are alive, and which worker owns partition 214. Two workers both reading it means duplicate logs; none means a gap.
The logs themselves never touch the coordination service. It holds kilobytes: one entry per live worker, one per assignment, the current leader and a config version.
Each of these could be built from scratch with Consensus & leader election, but it's hard to get right, and every team would build it again. Google's Chubby replaced exactly that: before it, most of Google's systems used ad hoc primary election. Afterwards GFS and Bigtable used it to pick their master and to store a small amount of metadata. Kubernetes keeps its whole cluster state in etcd.
2 · The store
It's a small, replicated store where every write goes through consensus
Underneath, a coordination service is a key-value store, a tree of paths in ZooKeeper and flat keys in etcd, replicated on a few servers that agree on every write through consensus: Zab for ZooKeeper, Raft for etcd. It stays available while a majority of its members is up. Every value carries a version number, so a client can write "set this only if the version is still 7", the compare-and-set that the recipes below rely on.
Member failures each size survives. A quorum is a majority, so a fourth member adds a machine that can fail without adding tolerance. Seven is a practical maximum, and five, which survive two failures, are enough in most cases; each extra member is one more copy every write must reach.
It's built for small data. ZooKeeper is built for values in the byte to kilobyte range and by default refuses to transfer more than about 1 MB at once. etcd limits a request to 1.5 MiB by default and the whole store to 2 GiB, with 8 GiB as the suggested maximum. Adding members makes writes slower, not faster, since each is one more copy to make. The pipeline writes a few hundred small entries; it would never store a log line here.
3 · Sessions
Sessions and leases tie an entry to a process that's still alive
Knowing who's alive is the core service. In ZooKeeper, each client holds a session kept alive by heartbeats. A worker registers by creating an ephemeral node, /workers/w17, which exists only as long as the session that created it. If the cluster hears nothing from the worker within the session timeout, the session expires and the node is deleted, without anyone having to clean up:
Membership maintains itself: a dead worker's entry vanishes when its session expires, and the coordinator learns about it from a watch instead of polling.
etcd does the same with leases: a lease has a time to live, the client keeps renewing it, and keys attached to it are deleted when it expires. Kubernetes uses this pattern for node health: every node's kubelet heartbeat is an update to that node's Lease object, and the control plane judges the node by when the lease was last renewed (Failure detection & split brain covers choosing the timeout).
4 · Watches
Watches push changes to clients, but a client must re-read after each one
The coordinator writes the assignment table under /assignments, and every worker needs to know when its share changes. Polling 30 workers' worth of reads every second is wasteful and still slow to react. Instead, a client sets a watch while reading, and the service notifies it when that data changes.
The details differ, and they matter:
- ZooKeeper's classic watch is a one-time trigger. After it fires, the client must read again and set a new watch; changes in between aren't delivered one by one, so the client reads the current state rather than replaying events. Since version 3.6, persistent recursive watches stay in place.
- etcd's watch streams every change in order of revision, and a client can resume from a revision it already saw, as long as the history hasn't been compacted.
Aspect | ZooKeeper, classic watch | etcd watch |
|---|---|---|
Fires | Once, on the next change | On every change, in revision order |
After it fires | Read again and set a new watch | Keeps streaming |
Changes in between | Not delivered one by one; read the current state | Delivered; resume from a revision unless compacted |
Watches also create a classic trap. If all 30 workers watch the same node and it changes, 30 notifications and 30 reads hit the service at once. ZooKeeper's recipes call this the herd effect and are designed to avoid it: each client watches only what it needs, often just one neighbor.
5 · Leader election
Leader election is a short recipe, and fencing finishes it
Coordination services provide primitives, and patterns such as locks and elections are built on them as client-side recipes. ZooKeeper's election recipe uses an ephemeral node that also gets a sequence number:
ZooKeeper's leader election recipe, for each coordinator candidate
me = create("/election/c_", EPHEMERAL | SEQUENTIAL) # -> /election/c_0000000007
loop:
kids = getChildren("/election")
if me is the lowest in kids:
become leader
else:
prev = the child just below me
if exists(prev, watch = true):
wait for the watch to fire, then loopThe lowest number leads. If it crashes, its node vanishes with its session, and only the next candidate, which was watching it, wakes up and checks. Kubernetes does the same for its scheduler and controller manager in high-availability setups, using a Lease object to decide which instance is active while the others stand by.
One gap remains. A leader that freezes past its session timeout has lost the election but doesn't know it yet, and may still write. Its writes must carry a number that only grows, such as ZooKeeper's transaction ID or etcd's revision for the leadership key, so the systems it writes to can reject the old leader (Distributed locks covers fencing tokens in detail).
6 · Consistency
Writes are strictly ordered, but reads may be stale unless you ask
Both services order every write through a single leader, and ZooKeeper also applies each client's requests in the order that client sent them. Reads differ. ZooKeeper answers reads from whichever server the client is connected to, without consensus, which is why it is fast for read-heavy workloads, around 10 reads to each write. The price is that a read can return an older value; a client that needs the latest calls sync() first. etcd makes reads linearizable by default, through Raft, and offers faster serializable reads that may be stale.
Aspect | ZooKeeper | etcd |
|---|---|---|
Data model | A tree of znodes | Flat keys, with every revision kept until compaction |
Liveness | Sessions and ephemeral nodes | Leases with keep-alives |
Reads | Served locally, may be stale; sync() to catch up | Linearizable by default; serializable optional |
Notifications | Watches, one-time or persistent | Watch streams from any retained revision |
Conditional writes | Version checks | Transactions comparing values or revisions |
For the pipeline, staleness matters in one place. A worker that reads an old assignment could read partition 214 alongside its new owner. The assignment carries a generation number that the log brokers check, the same fencing idea as above, so a stale reader is refused instead of trusted.
7 · In practice
Keep it small and off the hot path
A coordination service is critical infrastructure: when it's unavailable, no leader can be elected and no worker can join. It should therefore do little. Clients should cache what they read, such as the current leader's identity, and rely on watches to learn when it changes, instead of asking on every request.
- Store facts, not data
Leaders, members, assignments and config versions; kilobytes, never payloads.
- Run 3 or 5 members
Odd sizes, spread across failure domains.
- Tie liveness to sessions or leases
Entries for dead processes then remove themselves.
- Watch narrowly
Avoid herds; re-read after each notification.
- Fence every leader
Carry a growing number into every write the leader makes.
- Cache and stay off the request path
Clients cache what they read; requests shouldn't wait on the service.
Check yourself
4 questions
Takeaways
Remember this
- Coordination services store the small facts all nodes must agree on: leaders, members, assignments and configuration.
- They replicate a small key-value store with consensus on 3 or 5 members, with limits measured in kilobytes per value and gigabytes in total.
- Sessions and leases make entries vanish when their process dies, and watches push changes instead of polling.
- Leader election and locks are client recipes over ephemeral, sequential entries, and still need fencing tokens.
- Writes are strictly ordered; ZooKeeper reads can be stale unless synced, while etcd reads are linearizable by default.