System designCore9 min
Webhooks
A webhook is an HTTP request your system sends to a URL that someone else owns, to tell them something happened. It replaces endless polling with one message per event, and in exchange you inherit every way someone else's server can be slow, down, or not who it claims to be.
1 · The problem
Polling asks "anything new?"; a webhook is told when there is
An e-signature service lets companies send contracts for signing. 20,000 customer systems, such as CRMs, HR tools and billing platforms, want to know the moment a contract is signed. Without webhooks, each one polls: "has anything changed?" every minute. That is 20,000 ÷ 60 ≈ 333 requests a second, while contracts are actually signed about 3 times a second:
20,000 ÷ 60 ≈ 333 polls a second, nearly all answered "nothing new". A webhook sends one request per real event, and arrives seconds after it instead of up to a minute later.
With webhooks, the flow reverses. Each customer registers a URL and the event types it cares about, and the e-signature service sends an HTTP POST to that URL when an event happens. It is sometimes called a reverse API: instead of the customer calling you, you call the customer.
The rest of this page is about the price. The receiving server belongs to someone else: it can be slow, down, misconfigured or pointed at the wrong place, and it can be impersonated. Every section below handles one of those.
2 · The shape
A webhook is an HTTP request your system sends to a URL someone else owns
A delivery is an ordinary POST with a small JSON body and a few headers. Standard Webhooks, a format shared by dozens of providers, has three headers worth copying:
One delivery
POST /hooks/esign HTTP/1.1
Host: crm.customer.example
Content-Type: application/json
webhook-id: evt_2Lq9xT
webhook-timestamp: 1790956800
webhook-signature: v1,K5oZfzN95Z9UVu1EsfQmfVNQhnkZ2pj9o9NDN/H/pI4=
{"type": "envelope.completed", "data": {"envelope_id": "env_8812"}}Each part has a job:
- The event type (
envelope.completed) tells the receiver what happened, and receivers subscribe only to the types they need. - The event ID stays the same on every retry of this event, so the receiver can recognize repeats.
- The timestamp and signature prove the sender and the moment, covered below.
The event types and payloads are a public contract with readers you can't see, so they deserve the same versioning care as any API (Event-driven architecture).
3 · Delivery
The receiver will be slow, down or broken, so send from a queue and retry
Calling a customer's URL inside the request that completed the contract would tie your latency and your success to their server. Instead, record the event, put it on a queue, and let delivery workers send it, retrying with growing gaps:
The contract completes whether or not any customer is up. The event is written in the same transaction as the contract (the outbox, in Distributed transactions: 2PC & sagas) and delivered afterward.
Retry for days, not seconds, because receivers have real outages. Stripe retries for up to three days with exponential backoff. The Standard Webhooks spec suggests a schedule like this, with random jitter added (Timeouts, retries, backoff & jitter):
Attempt | Wait before it | Time since the event |
|---|---|---|
1 | None | 0 |
2–3 | 5 seconds, then 5 minutes | About 5 minutes |
4–6 | 30 minutes, 2 hours, 5 hours | About 7.5 hours |
7–10 | 10, 14, 20 and 24 hours | About 75 hours |
Delivery rules that keep one bad receiver from hurting everyone:
- Isolate endpoints. A queue or concurrency limit per endpoint means one customer's dead server can't hold up deliveries to the other 19,999 (Bulkheads).
- Time out. The spec suggests 15 to 30 seconds; GitHub asks receivers to answer within 10.
- Read the status. 2xx is success; 410 Gone means "stop sending"; 429 and 503 mean slow down, honoring
Retry-After; redirects count as failures. - Give up visibly. After days of failure, disable the endpoint and tell the customer by email.
4 · Security
Sign every payload, or anyone can post fake events
The customer's URL is public. If it accepts any POST that looks right, an attacker can send envelope.completed for a contract nobody signed. So the sender signs every delivery with a secret shared with that endpoint: an HMAC-SHA256 over the event ID, the timestamp and the exact body. The receiver recomputes it and rejects anything that doesn't match:
Verifying a delivery (receiver side)
const signed = `${id}.${timestamp}.${rawBody}`; // raw bytes, not re-serialized JSON
const expected = hmacSha256Base64(endpointSecret, signed);
if (!timingSafeEqual(expected, signatureFromHeader)) return reject(400);
if (Math.abs(nowSeconds() - timestamp) > 5 * 60) return reject(400); // replayed?
if (await seen(id)) return ok(200); // duplicate: already handled
await queue.push(event); return ok(200); // process later, answer nowThe details matter:
- Use the raw body. A framework that parses and re-serializes the JSON changes the bytes and breaks the signature.
- Check the timestamp. Because it is inside the signature, an attacker who captures a valid delivery can't change it, so rejecting old timestamps (Stripe's libraries default to 5 minutes) stops replays.
- Compare in constant time, so the comparison doesn't leak how much of a guessed signature was right.
- Rotate secrets with overlap. Stripe lets the old and new secret both sign for up to 24 hours, so receivers can switch without dropping deliveries.
Senders have a risk of their own. Customers choose the URL, and your workers will call it from inside your network. A URL like http://169.254.169.254/ or an internal hostname turns your webhook system into a tool for reading internal services: server-side request forgery (SSRF). Send every delivery through an egress proxy that refuses internal addresses, and run the workers in a network segment that can't reach anything internal anyway.
5 · Receiving
Receivers get duplicates and out-of-order events, so send an ID and a version
Retries guarantee duplicates: a receiver that processed an event but timed out before answering gets it again. And parallel delivery reorders events. Stripe states plainly that it doesn't guarantee delivery order. A contract might produce this:
A late retry of an older event must not move the contract back to "viewed", and a repeated event must not trigger the follow-up workflow twice.
Give receivers what they need to cope:
- An event ID that is stable across retries, used as an idempotency key (Idempotency). Keeping seen IDs for a few days covers the whole retry window.
- A version or sequence number on the object, so receivers can drop updates older than what they already have. Timestamps with one-second resolution aren't enough: distinct events can share one.
- Answer fast, work later. Receivers should verify, store and return 2xx in milliseconds, then process from their own queue, so slow processing never looks like a failed delivery.
6 · Payloads
A thin webhook plus a fetch is safer than a fat payload
What should the body contain? Two styles, and providers use both:
Aspect | Fat (full) payload | Thin payload |
|---|---|---|
Contains | The whole object as it was at the event | IDs and the event type; the receiver fetches the rest |
Receiver effort | No extra call | One API call per event |
Out-of-order events | An old payload can overwrite newer data | The fetch always returns the current state |
Data exposure | Sent to every listening endpoint | Each read goes through your API, with its access checks and audit log |
Changing it later | Hard: someone depends on every field | Easy: you can always add fields, not remove them |
For the e-signature service, thin is the safer default: a contract's signed PDF and signer details shouldn't be pushed to every URL a customer ever registered, and a fetch on arrival makes ordering problems disappear. Keep payloads small, usually under 20 KB, so receivers aren't forced to swallow data they don't want.
7 · In practice
Give receivers a way to see and replay what they missed
Even with three days of retries, a customer whose integration broke over a long weekend loses events. Good providers make that recoverable:
- A delivery log per endpoint: every attempt, status code and response time.
- Manual redelivery of one event or of everything that failed in a time range. Stripe allows resending from its dashboard for 15 days; GitHub doesn't retry automatically at all, but lets owners redeliver anything from the past 3 days.
- An events API to list recent events, so a receiver can reconcile after an outage instead of trusting that every push arrived.
- Sender: queue, retry, isolate
Deliver from a queue, retry with backoff and jitter for days, and isolate endpoints from each other.
- Sender: sign and protect yourself
Sign ID, timestamp and body; rotate secrets with overlap; send through an egress proxy that blocks internal addresses.
- Receiver: verify, deduplicate, answer
Check the signature and timestamp, skip seen IDs, return 2xx at once and process later.
- Both: plan for the gap
Thin events with a fetch, versions to reject old updates, and a replay or list API for the outage nobody noticed.
Check yourself
4 questions
Takeaways
Remember this
- A webhook pushes one HTTP request per event to a URL the receiver registered, replacing constant polling.
- Send from a queue with retries over days, exponential backoff and jitter, and isolate endpoints so one dead receiver can't block others.
- Sign the event ID, timestamp and raw body with HMAC; receivers verify in constant time and reject old timestamps.
- Delivery is at-least-once and unordered: include a stable event ID and an object version, and have receivers answer 2xx fast.
- Prefer thin events plus a fetch, protect the sender from SSRF, and give receivers a log and replay to recover missed events.