System designCore12 min

Idempotency

A request that times out may still have worked. Idempotency makes it safe to send it again: twice has the same effect as once, so a retry can never charge a card twice.

1 · The problem

A timeout can't tell you whether it worked

You order a coffee in an app on the train and tap Pay $4.50. The request reaches the server, the server charges your card, and the reply is on its way back when the train enters a tunnel. Ten seconds later the app has heard nothing. What should it do?

Phone

Coffee app API

Payment provider

Pay $4.50

Charge $4.50

Approved

Paid (lost in the tunnel)

No answer after 10 s

Pay $4.50 (retry)

Charge $4.50

Approved

Paid

The retry looks like a brand new payment to the server, so the coffee costs $9.00.

From the phone, three different situations look exactly the same:

The request never arrived.
It arrived and was done, but the reply was lost.
It is still being processed, slowly.

No amount of waiting tells them apart (Why distributed systems are hard explains why). If the app never retries, it loses sales that never went through. If it retries blindly, it charges people twice.

The way out is to make the request safe to repeat. Then the app can retry as often as it needs, and the worst case is a wasted message, not a second charge.

2 · The idea

Idempotent means twice has the same effect as once

An operation is idempotent if doing it several times has the same effect as doing it once. Pressing an elevator's call button is idempotent: press it five times and one elevator still comes. Adding a coffee to a cart is not: tap twice and you have two coffees.

Operation	Idempotent?	Why
Set the delivery address to 12 Elm St	Yes	The second time, it is already 12 Elm St
Add one coffee to the cart	No	Each repeat adds another coffee
Mark order 81 as paid	Yes	Paid stays paid
Charge the card $4.50	No	Each repeat is a new charge
Delete order 81	Yes	The second delete finds nothing to delete; the order is gone either way

HTTP builds this into its methods. GET, HEAD, PUT, DELETE and OPTIONS are idempotent by definition, so clients and proxies may retry them after a broken connection. POST is not, and neither is PATCH in general. That is why paying, which is usually a POST, needs extra care (HTTP covers the methods).

Two details people often get wrong:

Same effect, not same response. The first DELETE answers 200 and the repeat answers 404. It is still idempotent, because the state of the server ends up the same.
Only the requested effect counts. The server may still write a log line or a metric for every attempt; those are not what the client asked for.

Often the cheapest fix is to reshape the operation so it is idempotent by nature:

Set instead of add. "Set the quantity to 2" survives a retry; "add one" does not.
Let the client choose the ID. If the app names the new order order-7f3a and sends PUT /orders/order-7f3a, a repeat finds the order already there and changes nothing.
Move between states. "Mark as paid" can be repeated; "charge" cannot.

Charging a card can't be reshaped like that, which brings us to the general fix.

3 · The fix

An idempotency key makes any request safe to retry

An idempotency key is a unique name for one user action, chosen by the client and sent with every attempt of that action. The server remembers each key it has seen and what it answered. A repeat with the same key gets the saved answer back, and the work is not done again.

Make the key once
When you tap Pay, the app creates a random key, such as a UUID, and saves it with the pending payment, so it survives the app restarting.
Send it every time
Every attempt carries the same key, often in an Idempotency-Key header. A new key belongs to a new action, never to a new attempt.
The server checks first
A key it has never seen: do the work, then save the answer under the key. A key it has finished: return the saved answer.
Forget slowly
Keys are kept for a stated window, such as 24 hours. The window must be longer than any client keeps retrying.

Phone

Coffee app API

Payment provider

Pay $4.50, key k-91

Charge $4.50

Approved

Save k-91 → Paid

Paid (lost in the tunnel)

Pay $4.50, key k-91 (retry)

k-91 seen: no charge

Paid

The retry carries the same key, so the server returns the saved answer instead of charging again. One coffee, one charge.

The interesting cases are the ones in between. The server's answer depends on the state of the key:

The server sees	It answers	Why
A new key	Does the work, saves the result	First attempt
A finished key, same request	The saved result, even a saved error	The client must see what the first attempt did
A key still being worked on	409 Conflict: try again shortly	Running the work twice at once is the very thing to avoid
A finished key, different request	422: this key was used for something else	A client bug, not a retry

Payment APIs work this way. Stripe, for example, saves the status code and body of the first request for each key, errors included, compares the parameters of every repeat with the original, and may prune keys once they are 24 hours old. An IETF draft standardizes the Idempotency-Key header with the 409 and 422 answers above.

Scope keys to the caller. If the server stores keys in one shared space, two customers whose apps happen to pick the same key collide, and one could receive the other's saved answer. Store each key together with the account it came from.

4 · The catch

Claim the key and do the work together, or not at all

The obvious server code is: look the key up, and if it isn't there, charge the card and save the key. It has two holes, and both charge twice.

Two copies at once. You double-tap Pay, or a retry overtakes a slow first attempt. Both copies look up the key in the same millisecond, both find nothing, and both charge. Checking first and acting second is a race. The fix is to claim the key before doing anything: insert it into a table with a unique constraint. The database lets exactly one insert win; the other copy fails to claim it and answers 409.

A crash in the middle. The server charges the card, then crashes before saving the result. The retry finds the key claimed but unfinished. If the server treats that as "start again", it charges again. How to close this hole depends on where the work happens:

The work is…	Close the hole by…
Writes to your own database	Saving the key, the result and the order rows in one transaction: all of it commits, or none of it
A call to another system (a card network, an email service)	Passing a key on to that system, derived from yours, so it ignores the repeat
Several steps with outside calls between them	Recording each finished step with the key, so a retry resumes after the last one instead of starting over

Every hop protects itself. The API claims the key in its own database, and passes a derived key to the payment provider, so even a crash between the two can't produce a second charge.

The second row matters most. A database transaction can roll back your own rows, but it can't take back a charge or an email. Once a request has touched the outside world, the only protection is that the outside world deduplicates too. That is why payment providers accept an idempotency key on every request that creates or changes something.

5 · Messages

Queues deliver twice, so consumers must not care

Retries don't only come from phones. Many message queues promise at-least-once delivery: every message arrives, and once in a while it arrives twice. A queue that keeps copies on several servers may hand out a copy that was meant to be deleted, and a consumer that crashes before acknowledging a message gets it again.

The coffee app publishes an order paid event, and a loyalty service adds a stamp to the customer's card for each one. Adding a stamp is "add one": a duplicate event gives a free stamp. The fix is an idempotent consumer:

Each message carries a unique ID.
The consumer records the IDs it has processed, in the same transaction as the stamp itself.
A unique constraint on the ID makes a duplicate's insert fail, and the consumer acknowledges it and moves on.

Event arrives	Processed IDs before	Stamps	What happens
order-81 paid	(none)	4 → 5	New ID: stamp and ID saved together
order-82 paid	order-81	5 → 6	New ID: stamp and ID saved together
order-81 paid (duplicate)	order-81, order-82	6	Seen before: acknowledged, nothing added

Sometimes the data already holds the ID. If each stamp is a row keyed by the order it came from, a second stamp for order 81 can't be inserted, and no separate table is needed.

This is also the truth behind the phrase exactly once. A network can't guarantee a message is delivered exactly once, but a system can process it exactly once: at-least-once delivery plus an idempotent consumer. Delivery semantics covers the three guarantees in full.

6 · The edge cases

Idempotent is not the same as order-proof

Idempotency says a repeat changes nothing. It says nothing about when the repeat arrives. You set your delivery address to the office, but that request is lost on the way. You change your mind and set it to home, which goes through at once. Then the app's automatic retry of "office" gets through:

Phone

Coffee app API

Office: request lost

Set address: home

OK, home

Set address: office (retry)

Address is office again

Each request on its own is idempotent, yet the old one undid the newer one. Your coffee goes to the office.

Two tools stop it:

Conditional writes. Each address carries a version. The app sends the version it last saw (in HTTP, an If-Match header with the ETag), and the server refuses a write based on an old version with 412 Precondition Failed.
Idempotency keys help only partly. If the first "office" request was processed, the late retry carries its key and gets the saved answer. If it never arrived, the retry is the first real attempt, and it still wins. Versions are the complete fix.

A few more mistakes come up again and again:

Mistake	What happens	Do this instead
A new key for every attempt	Each retry looks new, so the key protects nothing	Create the key once per user action, before the first attempt
Keeping keys for less time than clients retry	A phone that comes back online two days later pays again	Keep keys longer than the longest retry, and say how long
Retrying a POST with no key	The classic double charge	Retry writes only when they are idempotent or carry a key
Saving the key after an outside call	A crash in between repeats the call	Claim the key first, and pass a key on to the outside system

Check yourself

5 questions

1. A DELETE of order 81 returns 200, and its retry returns 404. Is DELETE still idempotent?

2. The app creates a new UUID as the idempotency key inside its retry loop, once per attempt. The reply to the first attempt is lost. What happens?

3. Two copies of a payment with the same key arrive within a millisecond. Both look up the key, find nothing, and charge. What fixes it?

4. A loyalty service adds a stamp for every "order paid" event from an at-least-once queue. How do you stop duplicate stamps?

5. With no keys and no versions, you set the address to "office" (lost on the way), then to "home"; then the retry of "office" arrives. Each request is an idempotent PUT. What is the address?

Takeaways

Remember this

A timeout can't tell you whether a request worked, so any write that may be retried has to be safe to repeat.
Idempotent means the same effect, not the same response; set values instead of adding, and let clients choose IDs.
An idempotency key is made once per user action, sent with every attempt and scoped to the caller; a repeat gets the saved answer.
Claim the key and commit the work atomically, and pass a key on to every outside system you call.
Consumers of at-least-once queues must ignore duplicates; exactly-once processing is at-least-once delivery plus idempotence.