WebSockets & long polling

A charity runs a live online auction. Lot 42, a vintage guitar, has 20,000 people watching. A new bid arrives about every 30 seconds, and everyone watching should see it within about a second.

In plain HTTP the server can only answer requests, so the simplest plan is short polling: every watcher's page asks "anything new?" every few seconds. The load is the number of watchers divided by the interval, and the average wait to see a bid is half the interval:

Worse, almost all of that work is wasted. With a bid every 30 seconds on average, only about 1 poll in 15 finds anything new; the other 14 carry full HTTP headers, cookies and authentication checks to say nothing. Short polling is still the right choice when updates are rare and a delay of minutes is fine, such as checking whether a report is ready.

Long polling keeps the same request-and-response shape but changes when the server answers. The browser asks; the server doesn't reply until it has something to say, or until a timeout passes; the browser immediately asks again:

For the auction this is a large win. Each request now waits until the next bid or the 30-second timeout, about 19 seconds on average, so 20,000 watchers make about 20,000 ÷ 19 ≈ 1,050 requests a second instead of 10,000, and a bid reaches them almost at once.

The costs that remain:

• A header per message. Every bid still costs a full HTTP request and response.
• A gap after each answer. If two bids arrive a moment apart, the second waits while the first answer travels back and the next request travels out: three trips across the network.
• Waiting requests hold resources. 20,000 open requests are 20,000 connections and request slots on the servers.
• Several servers need care. The follow-up request may land on a different server, so the "after bid 18" position must be meaningful on any of them.

Most watchers never bid; they only listen. For one-way updates, browsers have server-sent events (SSE): the page opens one ordinary HTTP request, and the server never finishes the response. It keeps writing events into it, in a simple text format of type text/event-stream:

Each event ends with a blank line. The browser's EventSource API reconnects by itself if the connection drops, and sends back the last id it received in a Last-Event-ID header, so the server can replay what was missed. Because it is plain HTTP, SSE passes through most proxies and load balancers without special handling.

Its limits:

• One direction only. A bid goes up as a normal HTTP request.
• Text only, which is fine for JSON.
• A connection cap on HTTP/1.1. Browsers allow about 6 connections per site across all tabs, so a seventh tab's stream waits. Over HTTP/2, each stream shares one connection, and the default limit is about 100 (HTTP).

Bidders need both directions, quickly: send a bid, see everyone else's. A WebSocket starts as an HTTP request that asks to switch protocols. If the server agrees, the same TCP connection stops speaking HTTP and becomes a channel where either side can send a message at any time:

After the 101, messages travel as small frames of text or binary, with a few bytes of framing instead of HTTP headers. ws:// uses port 80 and wss://, the encrypted form you should always use, uses 443, the same ports as ordinary web traffic. Frames from the browser are masked, a safeguard for proxies that don't understand the protocol. Either side can send a ping, which the other must answer with a pong, to check that the connection is alive. A WebSocket can also run inside one stream of an HTTP/2 connection.

Each WebSocket or SSE stream lives on exactly one server for as long as it stays open. That is the very thing Stateless services & sessions warns about: a server now holds something that matters, its connected users. The auction runs 4 connection servers with about 5,000 watchers each. A bid placed by a bidder on server 1 must reach watchers on all four.

The usual design splits the work in two:

• Connection servers (often called gateways) only hold connections, track which user wants which topic, and push messages down. They contain no business logic, so they can be deployed rarely.
• A pub/sub layer carries each message to every connection server with an interested user (Message queues & pub/sub). Ordinary stateless services publish into it.

Slack's real-time messaging works this way: gateway servers in several regions hold the WebSocket connections, and channel servers, chosen by consistent hashing, send each message to every gateway with a subscriber; messages reach clients worldwide in about 500 ms.

Big audiences multiply the work. A bid on lot 42 becomes 20,000 frames; a busy chat room or a famous account's post becomes millions. The same trade-offs as feeds apply (Fan-out on write vs read).

Infrastructure built for short requests treats a quiet connection as a dead one:

• Idle timeouts. Amazon's Application Load Balancer and nginx both close a connection that carries no data for 60 seconds by default. A quiet lot would silently lose all its watchers. Send a ping every 20 or 30 seconds, or raise the timeout.
• Proxies. Upgrade and Connection are hop-by-hop headers: each proxy must be configured to pass them on, or the handshake fails.
• Balancing. A balancer places each connection once. Servers added later get only new connections, so existing load stays where it is, the same problem sticky sessions have (Load balancing).

Deploys are the hardest part. Restarting a connection server with 50,000 open connections drops all of them, and every client reconnects. If they all retry at once, the remaining servers and the authentication service take a spike:

So restart connection servers a few at a time, close connections gradually, and make clients wait a random time before reconnecting. Then make the reconnect lossless: the client says which message it saw last (bid 18), and the server replays anything newer, from a short history it keeps per topic. Without that, every bid placed during the reconnect is simply missed.

For the auction, the answer is mixed: watchers get server-sent events, bidders a WebSocket, and a long-polling fallback covers networks that block both. In general:

A checklist:

• Use wss:// and authenticate during the handshake, then authorize every message as you would every request.
• Ping every 20 to 30 seconds and set every proxy's idle timeout above that.
• Keep connection servers thin, and route messages to them through pub/sub.
• Number messages per topic so clients can resume from the last one they saw.
• Reconnect with random delay, and restart connection servers gradually.
• Watch open connections per server as a capacity metric, alongside requests.