Throughput, utilization & queueing

Latency told you how long one request takes. Throughput is how many requests a system finishes per unit of time: 1,000 orders per second, 50 MB per second. A highway makes the difference plain: the speed limit is latency, the number of lanes is throughput. Adding lanes gets no single car home sooner; raising the speed limit does not let more cars through a toll booth. They are different kinds of fast, and a design can have one without the other.

The slowest stage is the bottleneck, and a system is exactly as fast as its bottleneck. This is the theory of constraints: a system's throughput is limited by its constraint, and improving anything else is wasted effort until the constraint moves. So capacity work is always the same two questions: which stage is the bottleneck now, and what becomes the bottleneck once you fix it? What makes a stage hard to widen is the subject of Scalability.

A café gets 2 customers a minute and each stays 5 minutes. How many people are inside? 2 × 5 = 10. That is the whole of Little's law, and it works for requests exactly as it works for customers:

in flight = arrival rate × time inside

The orders database takes 800 orders a second and each spends 25 ms inside: 800 × 0.025 = 20 orders in flight at any moment. You did not have to look. The law is about averages, and it asks for one thing only: the system must be stable, meaning what comes in eventually goes out. Bursty or steady traffic, one server or fifty, any order of service: it still holds. Any two of the three numbers give you the third, which answers four everyday questions:

That is why it is the first sanity check in any capacity discussion. In textbooks you will see it written L = λ × W.

Utilization is the share of time a server is busy. Picture one worker that needs 10 ms per order. It can do 100 orders a second at most, so at 80 orders a second it is busy 80% of the time and idle the other 20%.

If it is idle a fifth of the time, why would any order wait? Because orders do not arrive evenly spaced. They come in clumps. An order that lands while the worker is busy has to wait, and the idle moment a second earlier cannot be saved up for it. The busier the worker, the fewer idle gaps are left to soak up the next clump. A supermarket cashier who is busy nine minutes in ten always has a line.

For one server and work that arrives at random, the average comes out as a simple formula: response time = work time ÷ (1 − utilization). With 10 ms of work per order:

Look at the right-hand side. Going from 80% to 90% busy is only one-eighth more traffic, and the response time doubles. From 90% to 95% it doubles again. That bend is the knee. The common mistake is to think a server is fine until it reaches 100%. It is not: at 90% every order already takes ten times longer than the work itself.

The formula is a simplified model (textbooks call it M/M/1: one server, random arrivals, random job sizes). Real systems differ in the details, but every load test shows the same bend. One thing does move it: several servers sharing one queue absorb clumps far better than a single server. The lower line is four servers, each as busy as the single one: at 90% they answer in about 30 ms, not 100. Bigger pools can safely run hotter.

The simple formula already assumes random arrivals. Real traffic is often worse: cron jobs fire on the minute, retries arrive in waves, and a 2-second report export sits in the same queue as 10 ms reads.

Kingman's approximation says what that costs. In words: wait ≈ (busy ÷ idle) × variability × work time.

• Busy ÷ idle is the knee from the last section: 80% busy gives 0.8 ÷ 0.2 = 4.
• Variability measures how uneven the arrivals and the job sizes are: 1 for fully random traffic, less for steady traffic, more for bursty.

The two multiply, so uneven traffic makes a busy server much worse:

Unevenness is easy to underestimate. One 2 s export in every hundred 10 ms reads pushes the variability to about 22, against 1 for ordinary random traffic: the reads behind the export wait for it, however idle the server is on average. So the fixes are about evenness as much as capacity:

• Separate slow work. Give it its own pool or queue so it cannot sit in front of fast work (Concurrency covers pools).
• Smooth arrivals. Add jitter to scheduled jobs and retries (see Timeouts, retries, backoff & jitter).
• Cap job size. Paginate, for instance, so no request is a hundred times bigger than its neighbors.

Precisely, variability is the average of two squared ratios, standard deviation ÷ mean: one for the gaps between arrivals, one for the job sizes.

What happens when more arrives than the system can finish? The formula has no answer, and neither does the queue: it grows without limit. Say 1,200 orders/s arrive at the 1,000/s database:

• The backlog grows by 200 every second.
• After one minute 12,000 orders are waiting.
• The newest one waits 12 s for its turn.

Nothing is broken, no error is logged, and every user is waiting. And working off the backlog takes far longer than building it did.

Then it gets worse:

• Memory runs out. Each waiting request holds memory, so the process eventually crashes.
• Work is wasted. A request that waited past its client's timeout is still served, and nobody is listening for the answer.
• Retries pile on. The client that gave up tries again, adding load to a system that already cannot keep up.

This is why you watch goodput, the requests answered correctly and soon enough to be useful, and not just throughput: under an unbounded queue goodput falls to zero while throughput looks fine.

Three moves replace the unbounded queue:

• Bound it, by length and by age. Throw out a request that has sat too long: a fresh one has a better chance of still being useful. A rule of thumb for steady traffic is a queue no longer than about half the thread pool.
• Shed what does not fit. Reject fast, with a 429 or 503, so the requests you do accept stay fast.
• Push back. Backpressure is the overloaded stage telling its producers to slow down, instead of failing or dropping work silently.

Backpressure is a choice, not a law. A load balancer can reject excess connections rather than queue them. An order pipeline may choose the opposite: accept every order even during a backlog, and sort out priorities behind the scenes.

Everything above points the same way: a resource that is nearly full has no room for a burst, and the queue that forms is the failure. Headroom is capacity you buy and deliberately do not use. Two calculations set how much. The first is the curve: at 60% busy a single server takes 2.5 times the work time, at 90% ten times. The second is what happens when a machine dies: its share of the traffic lands on the survivors.

Losing one of four servers multiplies each survivor's load by 4 ÷ 3:

• Plan the peak at 75% and one failure puts you at 100%, the top of the curve.
• Plan it at 60% and the same failure lands at 80%: on the knee, but alive.

Bigger pools can run hotter. They absorb bursts better (sixteen servers sharing a queue at 90% answer in about 14 ms, against 100 ms for one) and lose a smaller share when one dies. So there is no universal number: the right peak utilization comes from your pool size, how bursty your traffic is, and how fast you can add capacity, which is what Scalability is about.