Load balancing

A food delivery app gets 3,000 requests a second at lunchtime. One server cannot keep up, so you run six. But the app on every phone knows only one address, api.food.example. Something has to stand at that address and hand each request to one of the six.

That something is a load balancer. It has four jobs:

• Spread the work so no server is swamped while another idles.
• Route around failure: stop sending requests to a server that has died.
• Hide change: add a seventh server, or take one out for an upgrade, without any client knowing.
• Be one stable address for the outside world, however the servers behind it change.

It only works if any server can answer any request. A server that keeps a user's cart in its own memory forces the balancer to send that user back to the same server every time. Keep that state in a shared store instead, and the servers become interchangeable (see Scalability).

Balancers come in two kinds, named after the network layer they read (see How data moves across a network). The difference decides what each can do.

A layer 4 balancer picks a server when a connection opens, by hashing its addresses and ports, and every byte on that connection then goes to that server until it closes. That is a problem for modern clients. HTTP/2 and gRPC send hundreds of requests down one long-lived connection, so a layer 4 balancer pins all of them to whichever server it picked first. A layer 7 balancer reads each request and spreads them, even when they share a connection.

The obvious rule is round robin: server 1, then 2, then 3, and round again. Each server gets the same number of requests. With weighted round robin, a server twice as big gets twice as many.

It would be perfect if every request cost the same. They don't. Fetching the menu takes 5 ms; building a big order history can take 500 ms. In large real systems the most expensive requests cost a thousand times the cheapest. Round robin hands the next request to server 3 even while server 3 is still grinding through three slow ones, and server 4 sits idle.

Better rules look at how busy each server is right now:

• Least outstanding requests: send to the server with the fewest requests in progress. It needs an up-to-date count for every server.
• Power of two choices: pick two servers at random and send to the less busy of the two. Comparing just two gets most of the benefit of checking them all, and because each balancer picks a different random pair, several balancers don't all pile onto the same "least busy" server at once.
• Hashing: send all requests with the same key (a user, a cart) to the same server, useful when servers cache per key (see Consistent hashing).

Round robin is not wrong. When requests cost about the same and servers are identical, it is simple and even, which is why it is the default in most balancers. Reach for a load-aware rule when request costs vary widely.

If server 4 crashes, a balancer that keeps sending it a sixth of the traffic fails a sixth of your users. It finds out in two ways:

• Active checks: every few seconds to half a minute it asks each server for a small page, often /health. After a couple of failures in a row the server is taken out of rotation; after several successes it goes back in.
• Passive checks: it watches real traffic. A server whose requests keep failing or timing out is set aside for a while, for longer each time it happens again.

Three traps catch most teams:

Servers leave for upgrades and arrive when you scale out. Both moments can hurt users if the balancer handles them carelessly.

Sticky sessions (sending one user to the same server every time, by cookie) make this harder. Load stays uneven after you add servers, because existing users are stuck to the old ones, and a user whose server dies loses whatever was kept there. They are a crutch for keeping state on servers; moving that state into a shared store removes the need.

Every request passes through the balancer, so a single balancer is a single point of failure (see Redundancy & failure domains).

Three usual answers, from small to large:

• A pair with a floating address. Two balancers share one virtual IP; the standby takes it over within seconds if the active one stops answering.
• Many nodes behind DNS. Managed cloud balancers run a node in every zone, and DNS hands out all of their addresses.
• Two tiers. At very large scale, a fast layer 4 tier spreads connections across a fleet of layer 7 balancers, which then read each request. The layer 4 tier keeps no shared state: every node computes the same choice from a consistent hash, so any of them can take any packet.

Inside a company's own network, services often skip the middle box altogether: the calling service learns the list of servers and balances its own requests (client-side load balancing). It saves a network hop and a box to run, at the price of a smarter client in every language you use.