System designCore10 min
Delivery semantics
"At most once", "at least once", "exactly once": every message system promises one of them. Behind each promise is one decision, when to say "got it", and one fact: no network can deliver exactly once. Exactly-once effects are still possible; this page shows how they are built.
1 · The problem
Every hand-off can lose a message or deliver it twice
A fitness app counts steps. The watch syncs with the phone, the phone sends a batch to the app's ingest service, and the ingest service publishes it to a message broker. A step-counter service reads the batch and adds it to today's total in the database. User 77 has walked 6,000 steps so far, and a batch of 1,200 more is on its way:
Phone to ingest service, then three more hand-offs. Each ends with the receiver saying "got it", and each of those replies can be lost.
At every arrow the sender waits for an acknowledgment. When it doesn't come, the sender can't tell whether the message was lost on the way or only the acknowledgment was (Why distributed systems are hard). It has two choices, and each has a cost:
- Send again. Nothing is lost, but if the first copy did arrive, the receiver now has it twice.
- Don't send again. Nothing is duplicated, but if the first copy was lost, it is gone.
A delivery guarantee is simply which of those two a system chooses:
Guarantee | Promise | For the step batch |
|---|---|---|
At most once | Never twice, but may be lost | Today's total may stay at 6,000 |
At least once | Never lost, but may arrive twice | Today's total may become 8,400 |
Exactly once | Once, no more, no less | Always 7,200, which is what everyone wants |
The rest of this page follows the batch through the consumer, then the producer, and ends where every guarantee stops.
2 · The choice
The guarantee is decided by when the consumer says "done"
The step counter has two jobs for each batch: add it to the total, and tell the broker it is done, by acknowledging the message or saving its position in the log (Message queues & pub/sub). A crash can land between the two. Which job comes first decides what the crash costs. Acknowledge first, and a crash loses the batch:
At most once. The broker was told the batch was handled, so nobody will ever deliver it again. 1,200 steps are gone.
Process first, and a crash repeats the batch:
At least once. Nothing is lost, but the user has 1,200 steps they never walked.
Most systems choose at least once, because a duplicate can be detected and a loss can't. Lost data leaves no trace to repair. A duplicate carries the same message ID as the original, so the consumer can recognize it. At most once still has its place: when the next message replaces this one anyway, such as a live heart-rate reading sent every second, retrying an old reading is pointless.
A broker that says "exactly once" on the label has not solved the lost-acknowledgment problem. It has added deduplication somewhere. The useful question is always: where, and what does it cover?
3 · The fix
Exactly-once delivery doesn't exist; exactly-once effect does
No network can promise that a message arrives exactly once. A system can promise that its effect happens exactly once:
exactly-once processing = at-least-once delivery + a consumer that ignores repeats
There are three standard ways to build the second half.
1. Remember what you have processed. Each batch carries a unique ID. The consumer saves the ID in the same database transaction as the new total, and a unique constraint turns a repeat into a no-op. Idempotency walks through this pattern, the idempotent consumer, row by row.
2. Store your position with your result. On a log-based broker the consumer reads by offset. Instead of telling the broker where it got to, it keeps the offset in its own database, and updates it in the same transaction as the total:
Total and position commit together
BEGIN;
UPDATE daily_steps SET total = total + 1200
WHERE user_id = 77 AND day = '2026-10-02';
UPDATE consumer_position SET next_offset = 5522
WHERE topic = 'steps' AND partition = 3;
COMMIT;
-- after a restart: read partition 3 from next_offsetEither both rows change or neither does. After a crash, the consumer resumes exactly after the last batch it applied: batch 5521 is never applied twice and never skipped. The catch is that this works only when the result and the position can live in one database.
3. Send values, not changes. If the phone sends "today: 7,200 steps" instead of "+1,200", a repeat writes 7,200 again, which changes nothing. Order still matters: an old message arriving late must not overwrite a newer one. Daily steps only go up, so the consumer can keep the larger value, total = GREATEST(total, 7200), and both repeats and late arrivals become harmless.
Technique | Works when | Costs |
|---|---|---|
Remember processed IDs | Each message has a unique ID | A table of IDs to keep and expire |
Position with the result | Result and position fit in one database | Takes over offset tracking from the broker |
Values, not changes | The message can carry the whole new state | Needs a rule for late, older values |
4 · The other end
Producers make duplicates too, and the broker can drop them
Go back one arrow. The ingest service publishes the batch, the broker writes it, and the broker's acknowledgment is lost. The ingest service retries, and now the log holds the batch twice. Consumers that remember IDs would catch it, but the broker can stop it earlier.
Kafka's idempotent producer does this. The broker gives each producer an ID, and the producer numbers every batch it sends. The broker remembers the last number it wrote for each producer and partition, and quietly skips a repeat:
The retry is acknowledged but not written again. The log holds one copy, and the producer can retry freely.
Since Kafka 3.0 this is on by default, together with the strongest acknowledgment setting. That setting, acks, decides when the broker says "written", and so what a crash can cost:
acks | The broker replies after | If the leader dies right after |
|---|---|---|
0 | Nothing: the producer doesn't wait | The batch may never have been written |
1 | The partition's leader wrote it | Lost if no follower had copied it yet |
all | Every in-sync replica wrote it | Safe while any in-sync replica survives |
Other brokers have the same idea under other names. An Amazon SQS FIFO queue takes a deduplication ID with each message, or derives one from a hash of the body, and ignores a second send with the same ID within a 5-minute window. That covers quick retries; a retry 10 minutes later is a new message.
Retries can also break order. If a producer has several batches in flight and the first one fails, its retry can land after the second. Kafka's idempotent producer prevents that, keeping order with up to five batches in flight; without it, keeping order means sending one batch at a time.
5 · The limit
A broker's exactly-once stops at the broker's edge
A weekly-challenge job reads step batches from one Kafka topic and writes "badge earned" messages to another. Kafka transactions can make that read-process-write loop exactly once: the job's output messages and its new read position are committed in one transaction. Consumers set to read only committed messages never see the output of a run that crashed halfway.
Then an email service reads the badge messages and emails the user: "You earned the 70,000-step badge". That email is not inside any Kafka transaction:
Inside Kafka, from the steps topic to the badges topic, the job's effect is exactly once. The email leaves Kafka. If the email service crashes after sending and before saving its position, the badge email goes out twice.
That is where the guarantee ends: Kafka transactions cover data inside Kafka and nothing else. A call to another service, a write to another database, an email or a payment is outside, and repeats there are your problem. The fix is the same one as before, at the far end: the email service records each badge it has emailed, keyed by user, badge and week, and skips one it has already sent.
This is a classic design principle, the end-to-end argument. A guarantee like "no duplicates" can be fully provided only by the endpoints, which know what a duplicate means. The layers in between, the broker included, can make duplicates rarer and cheaper. They can't make them impossible for an effect they never see.
6 · In practice
Choose per message what a loss and a repeat would cost
Different messages in the same app deserve different guarantees. Ask two questions for each: what does losing one cost, and what does processing one twice cost?
Message | Loss costs | Repeat costs | Use |
|---|---|---|---|
Live heart rate, every second | Nothing; the next reading replaces it | Nothing | At most once |
Step batch | Missing steps | Invented steps | At least once, plus a consumer that ignores repeats |
Steps to badges, Kafka to Kafka | A missing badge | A double badge | Kafka transactions |
Badge email | No email | Two emails | At least once, deduplicated at the email service |
And a short checklist for any consumer:
- Acknowledge after the work is committed, never before, unless losing messages is fine.
- Assume every message can arrive twice, and make the second time a no-op.
- Give every message a stable ID at the source, so repeats can be recognized all the way down.
- Find every effect outside your database (emails, payments, calls to other services) and deduplicate it there, or pass an idempotency key along (Idempotency).
- Treat "exactly once" on a product page as a scope, and check what is inside it.
Check yourself
5 questions
Takeaways
Remember this
- Every hand-off ends with an acknowledgment that can be lost, so each sender chooses between resending (duplicates) and not (losses).
- Acknowledging before processing gives at most once; acknowledging after gives at least once, which most systems choose.
- Exactly once is an effect, not a delivery: at-least-once delivery plus a consumer that ignores repeats or commits its position with its result.
- Producers duplicate when they retry; idempotent producers and deduplication IDs let the broker drop the repeat.
- A broker's exactly-once stops at its edge; emails, payments and other outside effects need deduplication at the far end.