Dead-letter queues & poison messages

A parcel company scans each parcel at every step: picked up, sorted, out for delivery, delivered. About two million scan events a day flow through a broker to two consumers: one updates the public tracking page, the other texts customers. One morning a scanner firmware update starts sending ZIP codes as numbers, so "02139" arrives as 2139, and the tracking consumer rejects every event from those scanners.

Such an event is a poison message: retrying it changes nothing, because the problem is in the message itself. What it costs depends on the broker:

• In a queue, the message reappears after each failure and is retried forever, taking a worker's time on every attempt (Message queues & pub/sub).
• In a log, such as a Kafka partition, the consumer can't move past it without skipping it. Every event behind it, for thousands of other parcels, waits.

A dead-letter queue (DLQ) is the way out: after a set number of failed attempts, the message is moved to a separate queue or topic, the consumer moves on, and someone is alerted to look at it. The basic mechanism is on Message queues & pub/sub; this page is about using it well.

Not every failure belongs in a DLQ. The consumer should ask why a message failed, because each cause has a different cure:

The last row is the trap. A DLQ is for the few messages that are wrong. If a deploy breaks the consumer, every message fails, and a DLQ with a low attempt limit quietly fills with two million perfectly good events while the dashboards look calm.

Two defenses:

• Alert on the rate of dead-lettering, not just on whether the DLQ is empty. One event a day is noise; ten thousand an hour is an outage.
• Fail fast on systemic errors. Kafka Connect's default is to stop a connector at the first bad record, on the reasoning that bad data may be a symptom of an upstream problem that must be fixed first.

For known permanent errors, the consumer can dead-letter a message on the first attempt instead of waiting out the retries. Azure Service Bus, for example, lets an application dead-letter a message explicitly with its own reason code.

The idea is the same everywhere; the details differ in ways that bite:

Three details cause real losses:

Record why a message was dead-lettered, not just that it was. In Service Bus, the exception type goes in the dead-letter reason and the stack trace in the description; RabbitMQ keeps the queue and reason of each dead-lettering in the message. Without that, whoever opens the DLQ starts by reproducing the error.

A classic Kafka consumer group has no per-message acknowledgment, so it has no built-in DLQ. (Share groups, production-ready since Kafka 4.2, add per-record acknowledgment and delivery counts.) In a classic group the consumer has to move a failing event aside itself: publish it to another topic, then commit its offset so the partition keeps flowing. A common design adds retry topics with growing delays between the main topic and the DLQ:

Kafka Connect has this built in for connectors: errors.tolerance = all with errors.deadletterqueue.topic.name routes bad records to a DLQ topic, and errors.deadletterqueue.context.headers.enable = true attaches the reason to each one. Without those settings, the default is to stop at the first error.

Parcel 5513's "out for delivery" event fails on a transient error and goes to a retry topic. Ten seconds later its "delivered" event succeeds. Then the retry runs:

So don't use a DLQ with a FIFO queue in SQS if you can't afford to break the exact order of messages. Retry topics accept the same condition: the application must tolerate events handled out of order.

There are two ways to tolerate it:

• Make updates order-proof. Each scan carries a sequence number or the time it happened, and the consumer ignores any event older than what it already applied. The late "out for delivery" is then dropped, and so is a stale one replayed from the DLQ next week.
• Park the key. Once an event for parcel 5513 is set aside, set aside every later event for 5513 too, and release them together in order. This keeps order but needs a record of parked keys.

The first is usually simpler and also makes replays safe, which the next section needs anyway (Idempotency).

Messages in a DLQ aren't handled; they are waiting for a decision. Service Bus never cleans its DLQ automatically, and SQS expires messages by their original age, so an ignored DLQ either grows forever or quietly loses data. For the ZIP code events, the response goes like this:

Replayed messages arrive late and possibly twice, since some may have been partly processed before failing. Order-proof, idempotent consumers make that harmless. Also note that a replay can give messages a new identity: SQS treats redriven messages as new, with a new message ID and enqueue time, so deduplication must rely on the event's own ID, such as the scan ID, not the broker's.

A DLQ without an owner is a slower way of losing data. For the parcel company, the checklist:

• Classify failures in the consumer: retry transient ones, dead-letter permanent ones at once.
• Set the attempt limit high enough for ordinary hiccups, and know what your broker counts.
• Alert on the dead-letter rate, and treat a flood as a consumer bug, not as bad data.
• Record the reason, source and attempt count with every dead-lettered message.

And for keeping and replaying them:

• Keep DLQ retention longer than the source's, and clean it deliberately.
• Make consumers order-proof and idempotent, using the event's own ID and version.
• Replay at a controlled rate, only after the fix is deployed.
• On a log, publish failures to retry topics and commit, so one event never blocks a partition.

Dead-letter queues are the last stop after retries (Timeouts, retries, backoff & jitter); background-job systems keep the same idea under other names, such as Sidekiq's dead set (Background jobs & scheduling).