Background jobs & scheduling

A booking platform serves 4,000 hair salons. When a client books a haircut, the platform saves the booking, confirms it by text message, emails the salon, and adds it to the stylist's calendar. Then, 24 hours before the appointment, it sends a reminder. Every month it renders an invoice for each salon, and every morning at 7:00 it emails each owner a summary of the day.

Only the first step has to happen before the client sees "Booked". The rest is slow, depends on outside services that time out, or is due at a time nobody is making a request. Putting it in the request would make the client wait for the text-message provider, and fail the booking when the provider is down.

A background job is a small message naming a piece of work, put on a queue and run later by a worker process (Message queues & pub/sub). A scheduler creates jobs at a set time, once or repeatedly. When the client needs to follow the job's progress, the job also needs a status the client can poll (Long-running & bulk APIs).

Three rules prevent most job bugs, and each comes from a real failure:

• Pass IDs, not objects. Job arguments are serialized, so a whole booking object either doesn't survive the trip or arrives stale. Pass booking_id: 9917 and load the booking when the job runs, so it sees the current data.
• Enqueue after the commit. A worker can pick up a job within milliseconds, before the transaction that created the booking has committed. It then finds nothing.
• Make the job idempotent. Job systems run each job at least once, not exactly once.

At-least-once is not a corner case. In Sidekiq, for example, a job that completed can run again if Redis goes down after the job finishes but before Sidekiq records it as done, and a job that fails halfway is retried from the start. A confirmation job should therefore check whether the text was already sent for booking 9917, or send it with an idempotency key the provider honors (Idempotency).

Celery makes the trade explicit. By default it acknowledges a task just before running it, so a worker that dies mid-task loses it rather than repeating it. With acks_late, it acknowledges after the task returns, so a crash means a repeat instead of a loss, which is only safe for idempotent tasks (Delivery semantics).

The text-message provider is down for twenty minutes. Retrying every confirmation immediately would hammer it and fail again. Job systems retry with growing delays. Sidekiq's formula is retry_count⁴ + 15 seconds, plus a random amount that grows with the count:

A twenty-minute outage is absorbed within the first handful of retries; a bug in the job gives the team about three weeks to ship a fix before the job stops retrying. After the last retry, Sidekiq moves the job to its dead set, where it stays for six months for someone to retry it by hand. Celery's automatic backoff adds random jitter and caps each delay at ten minutes by default.

Not every failure deserves a retry. A malformed phone number fails the same way on attempt 25 as on attempt 1, so the job should recognize a permanent error and stop at once, keeping retries for transient ones such as timeouts. Where failed jobs end up, and how to replay them safely, is the subject of Dead-letter queues & poison messages; the backoff arithmetic is on Timeouts, retries, backoff & jitter.

On the first of the month, the invoice job enqueues 4,000 PDF renders at once. If everything shares one queue, a client who books at 00:05 gets the confirmation text after the last invoice is rendered. The fix is several queues, with workers that know which matters more:

Workers can poll queues in two ways, and Sidekiq shows both:

• Strict order: check critical, then default, then bulk. A job in default runs only when critical is empty, so a flood of critical jobs holds everything else back.
• Weighted: a queue with weight 2 is checked twice as often as one with weight 1, so every queue keeps moving, at different speeds.

Sidekiq can't mix the two in one process, so a queue that must always go first gets its own processes, as in the figure. That is a bulkhead: a separate pool so one kind of work can't exhaust capacity for another (Bulkheads). Keep the number of queues small, and watch how long the oldest job in each queue has been waiting: a growing wait is the earliest sign the workers can't keep up (Throughput, utilization & queueing).

Some jobs run at a moment: the reminder for a 15:00 haircut is a job scheduled for 15:00 the day before. Others repeat: the invoice run is 0 2 1 * *, 02:00 on the first of each month, in cron's minute, hour, day, month, weekday notation. That one line hides several decisions:

The daylight-saving row is a real trap: two widely used schedulers make opposite choices for the same missing half hour, and in the autumn both run the job once. Schedule business jobs in an explicit time zone, and avoid the hour from 01:00 to 03:00 local time unless the job is harmless to skip or delay.

Then there is midnight. Many teams schedule their daily jobs at 0 0 * * *, so the launches pile up in the same moment, and a job that fans out to thousands of workers makes a spike for everyone.

Some cron systems add a ? field that lets the system pick a value by hashing the job's configuration, spreading launches over the hour; EventBridge Scheduler offers a flexible time window, for example any time in the 15 minutes after the scheduled time. Run the 7:00 summaries any time in a window ending at 7:00, not at the stroke of it.

The 7:00 summary is due in each salon's own time zone, so it is really 4,000 small deadlines spread around the clock. One giant job looping over all salons would run for hours, and a crash halfway would leave nobody knowing which owners got their email. Instead, a short scheduler job runs every 15 minutes, finds the salons that are due, and enqueues one job each:

Only one scheduler may make that decision, or three job servers will each enqueue the same 118 jobs. Elect a single leader, or let a unique key in the database reject the duplicates (Distributed locks). A distributed cron runs its replicas under a consensus protocol so they agree on what has been launched, and records each launch before and after starting it.

Even then, a scheduler can die between starting a launch and recording it. The safer choice is to prefer skipping a launch over risking a double launch, since a skipped run is easier to recover from, unless the operation is idempotent and can simply be repeated. The salon platform makes every job idempotent, which lets it choose repeating instead: a duplicate summary job finds the sent mark and does nothing.

A job that grows into many steps, such as "charge the card, then book the stylist, then refund if the stylist is unavailable", needs its progress recorded between steps so a crash resumes instead of restarting. That is a workflow, not a job (Workflow orchestration & durable execution).

For the salon platform, the checklist:

• Move everything the user doesn't wait for into jobs, and keep job arguments to IDs.
• Enqueue after the commit, or through an outbox.
• Make every job idempotent, keyed by what it acts on and when.
• Retry transient errors with backoff and jitter; stop at once on permanent ones.
• Separate urgent and bulk work into queues, with dedicated workers for the urgent one.
• Schedule in an explicit time zone, away from the daylight-saving hour, and spread launches out instead of all at midnight.
• Have one scheduler decide, and many small jobs act.
• Watch queue wait time and the dead set, and alert on both.