Autoscaling

An online tax-filing site serves 2,000 requests a second overnight and 10,000 at its afternoon peak. Each web server handles 200 requests a second comfortably, so the site needs 10 servers at 4 a.m. and 50 at 3 p.m. A typical weekday, modeled as a smooth curve:

A fleet of 50 all day costs 50 × 24 = 1,200 server-hours and averages about 50% busy. Following the curve with whole servers, in 5-minute steps, needs about 603 server-hours: roughly half. Size the fleet for the average instead, and the site falls over every afternoon.

Autoscaling resizes the fleet automatically from measurements. Adding servers is scaling out; removing them is scaling in. (Moving to bigger machines is scaling up, a different tool covered on Scalability.) It assumes servers are interchangeable, so any of them can take any request (Stateless services & sessions).

Most autoscalers work like a thermostat. You pick a metric and a target, say average CPU at 60%, and the autoscaler repeats four steps:

The computation is a proportion. If the load is spread evenly, doubling the servers halves the load on each:

servers wanted = ceil(servers now × metric now ÷ target)

At 3 p.m. the site has 20 servers running at 90% CPU against a 60% target: ceil(20 × 90 ÷ 60) = 30 servers. Kubernetes uses exactly this formula, and ignores changes within 10% of the target so that small wobbles don't resize the fleet.

Why aim for 60% and not 95%? The gap is headroom. It absorbs a rise in traffic during the minutes before new servers arrive. A high target is cheaper and a low one is safer; the right value depends on how fast your traffic can rise compared with how fast servers start (see the next sections).

The formula works only if the metric drops in proportion as servers are added. CPU per server does, when requests are spread evenly. Many tempting metrics don't:

Response time deserves a warning. If the database is the slow part, every web server waits on it, response time climbs, and the autoscaler adds servers. More servers send more queries to the struggling database and make things worse.

Queue workers need a different target. The site turns each filed return into a PDF, which takes a worker 3 seconds, and users accept up to a 60-second wait. That sets the target backlog:

acceptable backlog per worker = acceptable wait ÷ time per job = 60 ÷ 3 = 20 jobs

At 9 p.m. 1,200 PDFs are waiting. 1,200 ÷ 20 = 60 workers clears them within the minute; with 30 workers, each faces 40 jobs, a 2-minute wait, so the autoscaler doubles them (Message queues & pub/sub).

On deadline day, at 11 p.m., traffic climbs from 6,000 to 18,000 requests a second in 10 minutes. The autoscaler reacts, but each decision takes time to become a working server: the metric is collected and averaged, the decision is made, a machine boots, the application starts, caches warm up, and health checks pass. Suppose all of that takes 5 minutes. The fleet is always sized for the demand of 5 minutes ago, at 60% utilization (capacity = demand ÷ 0.6):

Two things decide the gap: how fast traffic can rise, and how long a new server takes to be useful. Headroom buys time against the first. Faster starts shrink the second: a container on an already-running host usually starts faster than a new virtual machine, and some platforms keep a warm pool of instances that have already been through their slow setup and wait, stopped, to be switched on. During the gap itself, the servers that exist should refuse what they can't handle rather than slow down for everyone (Load shedding & backpressure).

When the surge is predictable, don't wait to react to it:

• Scheduled scaling: the deadline is known, so raise the fleet's minimum to the deadline-night size from 10:30 p.m. on deadline day.
• Predictive scaling: a forecast built from past days and weeks adds servers before the usual afternoon rise. It suits daily and weekly cycles and applications that are slow to start.

Reactive scaling stays on underneath, for the traffic the forecast missed.

Scaling in is where autoscalers cause their own trouble. Traffic is noisy: if the fleet shrinks the moment load dips, it grows again a minute later, then shrinks, then grows. This flapping costs boot time on every cycle and leaves the fleet short whenever the dip ends.

So autoscalers add quickly and remove slowly. Kubernetes looks back over a stabilization window, 5 minutes by default, and scales in only to the highest size recommended during that window. AWS target tracking likewise scales in gradually and won't remove a server if doing so would push the metric back over the target.

A server being removed may still be in the middle of work. Removing it safely takes a few steps:

Queue workers have it harder: a worker halfway through a 3-second PDF is fine, but one halfway through a 20-minute job loses 20 minutes of work. Either let it finish (some platforms can protect a busy instance from scale-in), or rely on the queue: an unfinished message becomes visible again and another worker processes it. That second run is one reason consumers must handle repeats (Delivery semantics).

Every web server opens a pool of connections to the database. At 20 connections each, the afternoon fleet of 50 opens 1,000. PostgreSQL's default limit is typically 100 connections. The web tier scaled out perfectly, and the database refuses most of it. The things autoscaling quietly depends on:

The last row hides a trap. A retry storm or a flood of bot traffic looks exactly like popularity, and the autoscaler answers it with more servers. Limits and rate limiting decide what deserves capacity (Rate limiting algorithms); autoscaling only supplies it.

Autoscaling also needs servers that can appear and vanish without notice. A server that keeps users' sessions in memory, or files on its own disk, loses them when it is scaled in. Keep that state in a shared store, so any server can take any request (Stateless services & sessions).

Autoscalers offer a few kinds of policy. Most real fleets combine them:

When setting one up:

• Scale on a metric that falls as servers are added: CPU or requests per server, backlog per worker.
• Set a minimum that survives the loss of a server or a zone (Redundancy & failure domains), and a maximum your database, quotas and budget can take.
• Leave headroom that covers the traffic rise during one server start.
• Measure how long a new server takes to be useful, and work to shorten it.
• Scale in slowly, and drain servers so in-flight work finishes.
• Schedule the peaks you already know about, and keep load shedding for the minutes when capacity is behind.