System designCore12 min

Distributed locks

Three servers, one job that must run once. A lock in a shared store looks like the answer, and for saving work it is. For keeping data correct, a lock that can expire while its holder is still working needs one more piece: a check at the resource itself.

Watch it evolve1 / 7

The problem

Three job servers, one nightly job

An online store runs its background jobs on three identical job servers, so one can fail without the jobs stopping. At midnight all three start the same job: charge this month's subscription renewals.

Start

Stage 1

Stage 2

Final

1 · The problem

A lock in one server's memory means nothing to the other two

An online store runs its background jobs on three identical job servers, so that one can fail without the jobs stopping (Redundancy & failure domains). Every server runs the same scheduler. So at midnight all three start the same job: charge this month's subscription renewals.

On a single server you would protect the job with a lock in memory, and only one thread could run it (Concurrency). That lock lives inside one process. The other two servers can't see it, so it stops nothing. The lock has to live somewhere all three can reach:

One named lock in a store every server can reach. Whoever acquires it runs the job; the others skip it or wait.

That is a distributed lock: a named lock kept in a shared store, which a server acquires only if nobody holds it and releases when done. Before choosing how to build one, ask what a double run would cost. The same store runs two jobs, and the answers differ:

Job	If two servers run it at once	The lock is for
Rebuild the search index, hourly	The same index is built twice; some processor time is wasted	Efficiency: an occasional double run is harmless
Charge subscription renewals, nightly	Customers are charged twice	Correctness: a single double run is an incident

Keep that distinction in mind. An efficiency lock may fail once in a while. A correctness lock may not, and the rest of this page shows why no lock can promise that on its own.

2 · Leases

Every distributed lock is a lease, because holders crash

Server 1 takes the renewals lock, then crashes halfway through. If the lock never expired, nobody would charge renewals again until a person noticed and deleted it. So every practical distributed lock is a lease: it is valid for a fixed time, its TTL (time to live), and then it frees itself. A holder that needs longer renews the lease while it works.

With Redis as the lock store, acquiring is one command:

Acquire the renewals lock for 30 seconds

SET lock:renewals 7f3a9c1e NX PX 30000

NX         only if nobody holds it
PX 30000   expires after 30,000 ms
7f3a9c1e   a random value, new for every acquire

The random value is how a holder recognizes its own lock. Releasing must mean "delete the key, but only if it still holds my value", never a plain delete. Here is what a plain delete does after a slow job:

Server 1

Redis

Server 2

SET lock NX PX 30000

35 s of work

30 s: key expires

SET lock NX PX 30000

done: DEL lock

server 2's lock gone

Server 1 released a lock that was no longer its own, so a third server can now take it while server 2 is working. Deleting only when the value matches server 1's random value would have left server 2's lock alone.

Recent Redis versions do the compare-and-delete in one command (DELEX with IFEQ); older ones use a three-line script. Renewal follows the same rule: extend the TTL only if the value is still yours. Then pick the TTL, which is a trade:

TTL	After the holder crashes	While the holder works
5 seconds	The job resumes within 5 s	Must renew every couple of seconds, or the lease ends mid-job
5 minutes	The job can stall for up to 5 minutes	Plenty of room, but still a limit

Renewing looks like it closes the gap: keep extending the lease while you work, and it never runs out. It closes the gap only while the holder is running. A holder that is frozen can't renew.

3 · The catch

A lease can run out while its holder is still working

From inside server 1, everything looks fine: it took a 30-second lease 10 seconds ago, so it has 20 left. Then, between checking and acting, the whole process stops. Common causes:

Garbage collection that pauses every thread, for seconds in a large heap.
The operating system stopping the process, or swapping its memory to disk.
The network holding the next request in a queue long after it was sent.
The clock jumping, when expiry is computed from wall-clock time. Redis expires keys by wall-clock time, so a clock change on the Redis server can free a lock early.

Here is a 40-second garbage-collection pause during the renewal run:

Server 1

Lock store

Server 2

Payment provider

acquire, 30 s lease

granted

GC pause: 40 s

30 s: lease expires

acquire

granted

charge customer 42

wakes: charge customer 42

Server 1 checked its lease before the pause and has no reason to check again: to the code, no time passed between one line and the next. Customer 42 is charged twice.

Checking the lease once more just before charging doesn't fix it. The pause can land between that check and the charge. A longer TTL makes it rarer, but some pause can always be longer.

This is not a lab curiosity. Database servers that lose their coordination session in a long garbage-collection pause are a known failure, and some systems shut such a server down on purpose, so it stops serving data that may already belong to another server. Independent tests of lock services have shown two clients holding the same lock at once.

So holding a lease is a belief, not a fact. No lease, however carefully built, guarantees on its own that only one holder acts. Something outside the holder has to check.

4 · The fix

Only the resource can enforce the lock

The fix is a fencing token, explained in full in Failure detection & split brain. In short: the lock store hands out a number that rises with every grant. The holder sends it with every write, and the resource remembers the highest number it has accepted and refuses anything lower.

In the renewal run, server 1 was granted token 33 before its pause and server 2 was granted 34. The billing database records each charge with a conditional write:

Server 1's late write, after server 2's

sql

UPDATE renewals
   SET status = 'charged', fence = 33
 WHERE customer = 42 AND period = '2026-10'
   AND fence < 33;
-- 0 rows: server 2 already wrote with token 34

Server 1 wakes, tries to record its charge, changes nothing, and learns it has lost the lock. The resource, not the holder, decided. For that the token must only ever go up, so where it comes from matters:

Lock store	Where the token comes from
ZooKeeper	The transaction ID (zxid) or version of the lock's node
etcd	The revision at which the lock key was created
Chubby	A "sequencer" carrying the lock's generation number
A database row	A counter you increase in the same transaction that takes the lock
Redis, one node or Redlock	None built in: the random value never increases, so it can't be compared

What if the resource can't check a token? The payment provider will never compare your fencing numbers. Then make the operation itself safe to repeat. An idempotency key such as renewal-42-2026-10 makes the provider charge once, however many servers send it (Idempotency).

Notice what happened. With a token check or an idempotency key in place, a double run is harmless, and the lock is back to being an efficiency measure. That is the healthy place for it.

5 · Where it lives

A lock is only as safe as the store that holds it

A single Redis node is fast and simple. It is also one machine: if it restarts without having saved the key, the lock is gone. Adding a replica for failover seems to fix that, but Redis copies to its replicas asynchronously:

Server 1

Redis primary

Redis replica

Server 2

SET lock NX PX 30000

crashes before copying

promoted to primary

SET lock NX PX 30000

The lock existed only on the machine that died. The new primary never heard of it and grants it again. Both servers now hold the lock.

The main options, from lightest to heaviest:

Store	How it holds the lock	Good for
One Redis node	A key with a TTL	Efficiency locks; the lock can vanish on a restart or failover
Redlock: five independent Redis nodes	The same key on a majority of them	Survives a minority of failures, but assumes bounded pauses and clock drift, and gives no token
ZooKeeper or etcd	A node or key tied to a session lease, stored by consensus	Correctness locks, with fencing tokens
The database you write to	A row lock, an advisory lock or a lease row	When the protected data already lives there

ZooKeeper's lock recipe also shows how to queue waiters fairly. Each server that wants the lock creates a numbered ephemeral node, one that disappears when its owner's session ends. The lowest number holds the lock, and every other server watches only the node just before its own:

When server 1's node goes, because it released the lock or its session expired, only server 2 wakes up. If every waiter watched the holder, each release would wake them all at once to find one winner: the herd effect.

Even here the session is a lease kept alive by heartbeats, so the pause from earlier still applies. What a consensus store adds is a store that doesn't lose the lock on failover, and a token to fence with (Consensus & leader election explains why).

The last row is the most underrated. If the data you are protecting sits in one database, that database's own locks are the lock. SELECT ... FOR UPDATE on the renewal row, or PostgreSQL's advisory locks (named locks whose meaning your application decides), released automatically when the transaction or session ends. Transactions & isolation levels covers the row locks.

6 · In practice

The best distributed lock is the one you don't need

Before reaching for a lock, ask whether the conflict can be designed away. Each of these removes the need for one:

Instead of a lock	How it works	At the store
Let the database decide	A unique constraint or a conditional write lets exactly one writer win	One renewal row per customer per month
Make repeats harmless	An idempotency key turns a duplicate into a no-op	`renewal-42-2026-10` at the payment provider
Give each key one owner	Split the work so one worker owns each key	Customers split across job servers by hash (Sharding & partitioning)
Elect a leader once	For a long-lived role, elect a leader instead of locking each job	One scheduler, chosen by consensus

When you do need a lock:

Decide efficiency or correctness first. It picks everything else.
For efficiency, one Redis node with a TTL and release-by-value is enough. Accept a rare double run.
For correctness, use a consensus-backed store, and make the resource check a fencing token, or make the operation idempotent.

In both cases:

Set a TTL longer than the work, and renew it only while the value is still yours.
When the lock is taken, retry after a random delay, so waiting servers don't all retry at the same instant (Timeouts, retries, backoff & jitter).
Release in a finally block, so an error doesn't leave the lock held until it expires.
Measure durations with the monotonic clock, never wall-clock time (Time, clocks & ordering).

Check yourself

5 questions

1. The hourly search-index rebuild runs on three servers and, very rarely, runs twice. What kind of lock does it need?

2. Why should a server release a Redis lock only if the key still holds its own random value?

3. A server holds a 30-second lease, confirms it is still valid, then pauses for 40 seconds before writing. What prevents damage?

4. Why can't the random value in a Redis lock serve as a fencing token?

5. Renewals are charged through a payment provider that can't check fencing tokens. What stops a customer being charged twice?

Takeaways

Remember this

A distributed lock is a lease in a shared store, and it must expire, because holders crash.
Decide first whether a lock is for efficiency or correctness; only correctness needs the heavy machinery.
A paused holder can outlive its lease and act anyway, so no lease alone guarantees a single holder.
For correctness, the resource must check a rising fencing token from a consensus-backed store, or the operation must be idempotent.
Often the best design needs no lock: a database constraint, an idempotency key, or one owner per key.