Message queues & pub/sub

A video platform accepts uploads. Every video must be transcoded into four resolutions before anyone can watch it, which takes one worker about 3 minutes. If the upload request did that work itself, the uploader's browser would sit on an open connection for 3 minutes. And when a hundred people uploaded at once, the transcoders would run out of room and uploads would start failing.

Instead the upload service saves the file, writes a small note, "transcode video 7731", to a queue, and answers the uploader at once: "Processing". Transcoding workers take notes from the queue whenever they are free:

The vocabulary: the note is a message, the service that sends it is the producer, the workers are consumers, and the system that holds messages between them is the broker. What the queue buys you:

• Decoupling in time. The producer doesn't need the consumers to be up. If every worker is being redeployed, jobs simply wait.
• Load leveling. A burst of uploads becomes a longer line, not a pile of failures.
• Independent scaling. Add workers when the line grows, without touching the upload service.
• Retries without the user. A job that fails can be tried again; the uploader never sees it.

Load leveling is easiest to see with numbers. Sixty workers at 3 minutes a job finish 20 jobs a minute. A viral moment brings 40 uploads a minute for 10 minutes, then the usual 10:

Without the queue, half the uploads during the spike would have been refused or timed out. The queue turned an overload into a delay. That trade only works while the delay is acceptable and the spike ends; the last section comes back to that.

When transcoding finishes, the video is published. Now three different services need to hear about it: notifications tell the channel's followers, search indexes the title, and recommendations start suggesting it. Putting "video 7731 published" on one shared queue would be wrong: each message would reach only one of the three.

This needs publish/subscribe. The producer publishes to a topic. Each service that cares creates its own subscription to the topic, and every subscription receives every message:

Real brokers combine the two. A subscription is itself a queue: if the search service runs five servers, they share search's subscription, and each message goes to one of the five. That is how pub/sub scales inside each subscriber. One detail surprises people: a subscription receives only messages published after it was created, so a brand-new service doesn't see last week's videos unless the broker keeps a history, as the log design below does.

Worker 1 takes the job for video 7731 and, 2 minutes in, its machine dies. If the broker had deleted the message when it handed it over, that video would never be transcoded. So brokers don't delete on delivery. They hide the message and wait for the worker to acknowledge it:

Brokers name this differently, but the idea is the same. Amazon SQS hides a received message for its visibility timeout, 30 seconds by default, until the consumer deletes it; a consumer can extend the timeout while it works, up to 12 hours. RabbitMQ waits for an ack, and requeues every unacknowledged message when the consumer's connection closes. Automatic acknowledgment, where a message counts as handled the moment it is sent, is fire-and-forget, and unsafe.

Two consequences follow:

• Set the timeout longer than the work. If transcoding takes 12 minutes and the timeout is 10, the job reappears while worker 1 is still busy, and a second worker starts it too. Either set a longer timeout or keep extending it while you work.
• The same message can arrive twice. A worker can finish and crash before acknowledging. So the usual promise is at least once, not exactly once, and consumers must cope with duplicates. Delivery semantics takes that apart.

And one failure the hand-off makes worse. A video with a corrupt file crashes the transcoder every time. Hidden, reappearing, crashing, forever: a poison message that keeps one worker busy doing nothing useful. The answer is a dead-letter queue:

Set the limit high enough that ordinary hiccups get their retries. A limit of 1 sends a message to the dead-letter queue after a single failure. What to do with the messages that land there is Dead-letter queues & poison messages.

Everything so far describes a queue that forgets: once a message is acknowledged, it is gone. Classic RabbitMQ queues and Amazon SQS work this way. Kafka works differently. It appends every message to a log and doesn't delete it after it is read; it keeps the log for a configured time, 7 days by default. Each consumer group only remembers its position in the log, its offset:

Replay is the log's biggest gift. Suppose a bug in the search service mangled the index for three days. With a delete-on-ack queue, those messages are gone. With a log, you fix the bug, move search's offset back three days, and let it process them again.

The price is that a log can't easily set aside one bad message while the others carry on, because each partition is read in order; with a classic consumer group, per-message retries and dead-lettering are something the consumer has to build.

Kafka 4.2 made share groups production-ready: several consumers read one partition and accept, release or reject each record, with a count of delivery attempts, a queue's habits on top of the log.

Video 7731 produces three events in a row: uploaded, transcoded, published. A consumer that sees "published" before "transcoded" might show a video that can't play yet. Does the broker keep them in order?

Only within limits. To scale, Kafka splits each topic into partitions spread over several brokers. Messages with the same key always go to the same partition, and a partition is read in exactly the order it was written. Across partitions there is no order at all. So the key decides what stays in order: with the video ID as the key, every event for video 7731 lands in one partition, in sequence.

Within a classic consumer group, each partition is read by exactly one consumer at a time. That keeps the order, and it also caps the parallelism (a share group lifts the cap by giving up that order):

Other brokers make similar promises. An SQS standard queue makes a best effort at order but may deliver out of order, and occasionally twice. An SQS FIFO queue keeps strict order within a message group, and won't hand out the next message of a group while the previous one is still being processed. Google Pub/Sub orders messages only when they share an ordering key.

Two warnings come with keys. A key that carries a lot of traffic, such as one wildly popular channel, makes one partition much busier than the rest (a hot key, Hot partitions & skew). And strict order means one slow or failing message holds up everything behind it with the same key.

A queue hides trouble well. Producers keep succeeding while the backlog quietly grows. The question to ask is not "is the queue up?" but "how far behind are we?":

The expected wait is Little's law again (Throughput, utilization & queueing). If it keeps rising, consumers are slower than producers, and no queue fixes that; it only postpones the moment you notice. Three ways out:

• Add consumers, automatically from the backlog per worker (Autoscaling).
• Make each message cheaper to process.
• Push back on producers, or refuse work you can't do in time (Load shedding & backpressure).

And going async has costs that never show up on the happy path:

• The caller doesn't get the result. The uploader sees "Processing" and must be told later, by a notification or by polling.
• Duplicates are normal. Consumers must be idempotent (Idempotency).
• Debugging spans time. A failure can surface minutes after the request that caused it.
• The broker is one more critical system to run, monitor and keep replicated.

Use a queue when the work doesn't need to finish before you reply. When the caller needs the answer to continue, a direct request is simpler and more honest.