System designDeep dive12 min
Workflow orchestration & durable execution
Some processes take days and many steps: wait for a customer, call a flaky service, ask a person to decide, pay out exactly once. Built from jobs, status columns and cron sweeps, they scatter their state everywhere. A durable execution engine lets you write the whole process as ordinary code, records every step's result, and resumes the code after any crash as if nothing happened, at the price of a few strict rules.
1 · The problem
A process that lasts days doesn't fit in a request or a job
A car insurer handles 30,000 open claims at any time. Each claim follows the same process: register it, ask the driver for photos and wait up to a week, sending a reminder after two days, get a fraud score from an outside service, wait for an assessor's decision if the score is high, pay out, and send a letter. A claim takes days, sometimes weeks.
Built from background jobs (Background jobs & scheduling), the process ends up spread over many pieces, each of which must be right:
Need | Hand-built answer | What tends to go wrong |
|---|---|---|
Know where each claim is | A status column, updated by every job | A job crashes between its work and the update |
Wait two days, then remind | A cron sweep for claims waiting too long | Sweeps overlap, miss claims or remind twice |
Retry the fraud service | Retry logic in each job | Different rules in every job |
Wait for the assessor | Another status, polled | Nobody notices a claim stuck for a month |
Pay exactly once | Careful checks before paying | A retry after a timeout pays twice |
The process itself, the thing a new engineer needs to read, exists nowhere in one place. Workflow orchestration puts it in one place, and durable execution is one way to do that: the process is written as ordinary code, and the engine guarantees that code runs to completion despite crashes, deploys and waits of any length. Temporal, AWS Step Functions and Azure Durable Functions are common engines.
2 · The idea
The engine records every step and replays the code to resume
The claim process becomes one function. It reads like code that runs once from top to bottom, waits included:
The claim workflow (Temporal-style Python, simplified)
@workflow.defn
class ClaimWorkflow:
@workflow.run
async def run(self, claim_id: str) -> str:
await workflow.execute_activity(register_claim, claim_id, **opts)
await workflow.execute_activity(request_photos, claim_id, **opts)
# Wait for the driver's photos: remind after 2 days, give up after 7
if not await self.photos_within(days=2):
await workflow.execute_activity(send_reminder, claim_id, **opts)
if not await self.photos_within(days=5):
return "closed: no photos"
score = await workflow.execute_activity(fraud_score, claim_id, **opts)
if score > 0.8:
await self.assessor_decision() # a person decides; may take days
await workflow.execute_activity(pay_out, claim_id, **opts)
await workflow.execute_activity(send_letter, claim_id, **opts)
return "paid"Two kinds of code appear. The workflow function decides what happens next. Each activity, such as fraud_score or pay_out, does the actual work against the outside world. The engine sits between them. Every time the workflow starts an activity, starts a timer or receives a message, and every time an activity finishes, the engine appends an event to the claim's event history, an append-only log it stores durably.
When the worker running claim 4471 crashes after the fraud score, nothing is lost. Another worker picks the claim up and runs the workflow function again from the top. Each time the code asks for an activity that the history says already completed, the engine hands back the recorded result instead of running it again. The code fast-forwards to where it was and carries on:
Worker B re-runs the workflow code, but the fraud service isn't called again: the score comes from the history. Execution continues at the first step not yet recorded.
This is event sourcing applied to a running function (Event sourcing & CQRS): the history is the state, and the function's local variables are rebuilt from it. Azure Durable Functions works the same way, re-executing the orchestrator function from the start and consulting the history; Step Functions instead runs a declared state machine and records the history of its states.
3 · Determinism
Workflow code must make the same decisions every time it is replayed
Replay only works if the re-run takes the same path. If the workflow code read the clock to decide whether two days had passed, a replay a week later would read a different time and take a different branch, and the recorded history would no longer match. The engine detects the mismatch and stops the execution with a non-determinism error.
So workflow code must be deterministic: given the same history, it makes the same calls in the same order. In practice that means anything that varies between runs goes through the engine:
Don't, in workflow code | Do instead |
|---|---|
Read the system clock | Use the engine's time, which replays the recorded value |
Sleep, or use language timers | Use the engine's durable timers |
Generate random numbers or UUIDs | Use the engine's replay-safe version, or get them from an activity |
Call APIs, databases, files | Put every I/O call in an activity |
Read mutable global variables | Pass everything in as input or through messages |
Activities have no such rule, because their results are recorded, not re-executed on replay. The split is the whole design: workflow code is pure decision-making that can be replayed any number of times; activities are the side effects, run once and remembered. One harmless oddity follows from replay: a log line written in workflow code may appear several times.
4 · Activities
Activities are retried until they succeed, so they must be idempotent
The fraud service times out now and then. The engine retries the activity on its own, following a retry policy. Temporal's defaults: wait 1 second, double the wait each time, cap it at 100 times the first interval, and keep trying without limit. The workflow code just sees the activity eventually return, or fail with a non-retryable error it can handle.
Temporal's default policy: 1 s × 2 for each earlier retry, capped at 100 × 1 s = 100 s. The waits add up to 127 s by retry 7, then 100 s each, so a twenty-minute outage of the fraud service costs about 18 retries before the workflow carries on.
Retries make the payment dangerous. If pay_out sends the payment and the worker dies before the result is recorded, the engine can't know the payment happened, and runs the activity again. So in Temporal an activity can run more than once, and activities must be idempotent. For the claim, pay_out sends the payment with an idempotency key, the claim ID, so the payment provider turns a second attempt into a no-op (Idempotency).
Two more activity settings matter:
- A timeout. Without one, the engine can't tell a slow activity from a dead worker. Temporal requires at least one, typically start-to-close.
- Heartbeats for long activities. An activity that runs more than a few minutes should report progress periodically, so a stalled one is detected and retried quickly.
Step Functions makes the guarantee a choice of workflow type. Standard workflows run each task and state exactly once unless you configure retries, last up to a year and keep a full history, which suits non-idempotent steps such as payments. Express workflows run at least once, last at most five minutes, and are priced for high-volume event processing, so their steps should be idempotent.
5 · Waiting
Waiting days costs nothing, because a waiting workflow isn't running
Most of a claim's life is waiting: for photos, for the assessor. A durable timer is an event in the history, not a sleeping thread. Temporal's timers are persisted, so if workers are down when a timer fires, the workflow resumes as soon as they return, and workers spend nothing while waiting: one worker can wait on millions of timers.
Waiting for a person works the same way. The workflow pauses until a signal, a message sent to that one execution, arrives: the upload service signals "photos uploaded", the assessor's screen signals "approved". Step Functions offers the same with a task token: the state machine hands a token to an outside system and waits until the token comes back with a result.
Between events the claim is only a history in storage. 30,000 waiting claims cost storage, not threads or servers.
This replaces the cron sweep entirely. There is no query for "claims waiting too long" to get wrong; each claim carries its own timer, and its reminder is written next to the step it belongs to. Clients can also ask a running workflow for its current state with a query, so the status page reads the process itself instead of a status column that might be stale.
6 · Changing code
Code changes meet executions that started on the old code
The insurer decides to check the driver's policy before asking for photos, a new activity at the top of the workflow. Thousands of claims are mid-flight, and their histories were written by the old code. Replaying one with the new code, the first thing the code does is ask for the policy check, but the history says the first step was register_claim. Mismatch: non-determinism error.
So workflow code needs versioning, the same compatibility problem as data (Schema evolution & encoding), solved in one of three ways:
Strategy | How it works | Cost |
|---|---|---|
Patch in place | The code branches on a version marker recorded in the history: old executions take the old path, new ones the new | Old branches stay in the code until old executions finish |
Pin executions to code versions | Old executions keep running on workers with the old code; new ones start on new workers | Two versions of workers to run for a while |
Let old executions finish first | Drain, then deploy | Only workable for short processes |
Temporal pins workflows with worker versioning; Azure Durable Functions offers built-in orchestration versioning and side-by-side deployments. A long-running workflow also accumulates history: Temporal caps one execution at 51,200 events or 50 MB, and Step Functions Standard at 25,000 events. A workflow that loops forever, such as a monthly policy renewal, periodically continues as new: it starts a fresh execution carrying only its current state, which also lets it pick up new code at a clean boundary.
7 · In practice
Use an engine when the process has steps, waits and consequences
Shape of the work | Use |
|---|---|
One step, retried on failure | A background job |
A few steps, seconds long, all idempotent | A job chain, or a short express workflow |
Many steps, waits of days, people involved | Durable execution |
Steps that must be undone if a later one fails | Durable execution running a saga (Distributed transactions: 2PC & sagas) |
Data pipelines on a schedule | A batch orchestrator (Batch processing) |
The engine is itself a critical system: if it is down, no workflow advances, so it needs the same care as a database. And it moves complexity rather than removing it; the rules of determinism, idempotent activities and versioning have to be learned by every engineer who writes a workflow.
For the insurer, the checklist:
- Write the process as one workflow, so it can be read top to bottom.
- Keep workflow code deterministic: engine time, engine timers, no I/O.
- Put every side effect in an activity, with a timeout, a retry policy and an idempotency key.
- Use timers and signals for waiting, never sweeps.
- Version every change to running workflows, and test replays of real histories before deploying.
- Continue as new before histories grow large.
Check yourself
5 questions
Takeaways
Remember this
- Durable execution lets a multi-day, multi-step process be written as one function that survives crashes, deploys and long waits.
- The engine appends every step's result to an event history and resumes by replaying the code against it.
- Workflow code must be deterministic: engine time, durable timers and no I/O; side effects go in activities.
- Activities run at least once and are retried by policy, so give them timeouts and idempotency keys.
- Timers and signals replace sweeps and status polling, and every change to running workflows needs versioning.