System designCore10 min

Load testing & capacity planning

How much traffic a system can take is a guess until a test proves it, and a badly built test proves the wrong number. Load testing measures what each part can really serve within its targets; capacity planning turns forecasts and those measurements into machines, ordered before the traffic arrives.

1 · The problem

Capacity is a guess until a test measures it

A national pizza chain takes about 2,000 requests a second on a normal Friday evening. Next month brings the year's biggest football final, and last year's final night peaked at more than four times a normal Friday. Marketing has also planned a half-price offer that night. Nobody knows whether the ordering system can take it.

Capacity planning has three steps it can't skip:

Forecast organic demand, the natural growth of usage, far enough ahead to cover the time it takes to get capacity.
Add inorganic demand: launches, campaigns and events, such as the final and the offer.
Load test regularly, to translate raw resources, servers and disks, into service capacity, requests served within the targets.

The first two are estimates (Estimation, capacity & cost). The third is measurement, and it is where most of the mistakes hide: tests that send the wrong load, measure the wrong number, or run against a system unlike production. The chain's forecast for the final is 9,000 requests a second. The rest of this page is about finding out how many servers that takes.

A normal Friday evening: 2,000 requests/s
Final-night forecast: 9,000 requests/s

The forecast is 4.5 times a normal Friday: last year's final plus the planned offer. Both figures are the example's; the question is what 9,000 a second takes.

2 · Test types

Different tests answer different questions

A load test is not one test. Each shape of traffic answers a different question:

Test	Load	Duration	Answers, for the pizza chain
Smoke	Minimal	Minutes	Does the script work, and the system respond at all?
Average load	A normal Friday	5 to 60 minutes	Do we meet our targets on a normal night?
Stress	Above average	5 to 60 minutes	How do we behave at final-night levels?
Spike	A sudden surge	A few minutes	What happens when kick-off orders arrive in one minute?
Soak	Average	Hours	Do memory, connections or disks slowly run out?
Breakpoint	Rising until something breaks	As long as needed	Where exactly is the limit, and what fails first?

Run them in that order: a smoke test first, so a broken script doesn't waste an expensive large test, then higher loads and longer durations. A test also needs pass and fail criteria written down before it runs. Load-testing tools call these thresholds and typically use them to encode the service's objectives, such as "95% of order requests under 200 ms, and under 1% errors"; a test that breaks them fails, like a unit test.

3 · Open vs closed

A test that waits for answers stops pushing exactly when it should push harder

Load generators come in two kinds. In a closed model, a fixed number of virtual users each send a request, wait for the answer, and only then send the next. In an open model, requests arrive at a set rate, whatever the system does, the way real customers do: nobody waits for a stranger's pizza order to finish before placing their own.

The difference is invisible while the system is fast, and decisive when it isn't. Suppose one part of the chain's system can serve 1,000 requests a second, and the test aims for 1,200:

Closed, 120 users with 0.1-second requests: as the system saturates, responses slow down, each user sends less often, and the offered load falls to exactly what the system can handle. At 120 ÷ 0.12 s = 1,000 requests a second, responses settle at 120 ms. The report looks fine.
Open, 1,200 requests a second: 200 more arrive every second than leave. After one minute, 12,000 are waiting, and a new request waits 12 seconds.

Closed model reports: 0.12 seconds
Open model, after 1 minute: 12 seconds

The same overloaded system. The closed test throttles itself to the system's speed and reports 0.12 s; the open test keeps arriving like real users and shows the queue: (1,200 − 1,000) × 60 = 12,000 waiting, served at 1,000 a second.

This is coordinated omission again, the trap described on Latency: the tester and the system slow down together, so the test never sends the requests that would have suffered. Response times under open and closed models can differ by orders of magnitude at the same load. For capacity questions, generate load at a fixed arrival rate. Tools support both; k6, for example, uses arrival-rate executors for the open model and counts dropped iterations when the generator itself can't keep up.

4 · Finding the limit

Capacity is the highest rate that still meets the targets, not the crash point

To find one server's capacity, the chain runs a breakpoint test against a single ordering server: an open-model load that rises steadily, with thresholds that mark where the server stops meeting its targets:

Breakpoint test for one ordering server (k6)

javascript

export const options = {
  scenarios: {
    breakpoint: {
      executor: 'ramping-arrival-rate',   // open model
      startRate: 100,
      timeUnit: '1s',
      preAllocatedVUs: 200,
      maxVUs: 2000,
      stages: [{ target: 1500, duration: '30m' }],
    },
  },
  thresholds: {
    http_req_duration: [{ threshold: 'p(95)<200', abortOnFail: true }],
    http_req_failed: [{ threshold: 'rate<0.01', abortOnFail: true }],
  },
};

Response time stays nearly flat as the rate climbs, then bends sharply upward as the server's queues fill (Throughput, utilization & queueing). The useful number is where the bend crosses the target, not where the server crashes: if p95 passes 200 ms at 640 requests a second, the server's capacity is below that. The chain records 600 requests a second per server at its targets, leaving a little margin for test error.

Watch what fails first, because that is the real limit. It may not be the servers being tested at all: a database's connection pool, a cache's network card, a payment provider's rate limit. A breakpoint test is as much a search for the bottleneck as a measurement of it (Scalability).

5 · Realism

A test is only as good as its resemblance to production

The most common load-test results are confidently wrong, because the test differs from production in a way that matters:

Trap	What goes wrong	Do this instead
One endpoint only	Menu reads are cheap; checkout is not	Replay the real mix: menu, cart, checkout, tracking
Tiny test database	Everything fits in memory; queries look fast	Production-sized data, or a copy of production
Warm caches, repeated data	Every request hits the cache	Spread requests across realistic users and items
Real third parties	The payment provider rate-limits or bills you	Stub them with realistic delays, or agree a test with them
Generator too small	The load generator saturates first	Watch its CPU and dropped requests; spread it out
Too short	Leaks and slow growth never show	Soak for hours before a big event

The chain's stress-test setup. Everything it owns is real and production-sized; the payment provider is replaced by a stub that answers with a realistic delay. The 300 ms is the example's.

Some teams go further and test in production, with real infrastructure and synthetic orders marked to be discarded. That removes most of these traps, at the price of care: test traffic must never charge a card or bake a pizza. Production tests, including stress tests that find how many queries a server can take before requests fail, are standard practice in large services.

6 · The plan

Turn the forecast and the measurement into machines, with room for failure

The chain now has a forecast, 9,000 requests a second, and a measurement, 600 per server within the targets. The plan adds what the test didn't include: failures and maintenance during the peak (Redundancy & failure domains).

Step	Calculation	Servers
Serve the forecast	9,000 ÷ 600	15
Survive losing one of three zones	15 must fit in two zones: ⌈15 ÷ 2⌉ = 8 per zone, × 3	24
Deploys and spares during the night	Freeze deploys; the zone margin also covers single failures	24

Then check the plan end to end. Run a stress test at 9,000 requests a second against the 24-server fleet, then fail one zone during the test and confirm the targets still hold. Everything the servers call must be checked the same way: 9,000 requests a second may be fine for the servers and fatal for the database or the payment provider's agreed rate.

Timing matters as much as numbers. The forecast must reach further ahead than the time it takes to get capacity, and autoscaling only helps for growth slower than it reacts (Autoscaling). For a known event, scale up ahead of it rather than waiting for the surge. And plan for the forecast being wrong: decide in advance what the system will shed if demand exceeds even the plan (Load shedding & backpressure).

7 · In practice

Test regularly, against targets, with open-model load

For the pizza chain, the checklist:

Forecast organic and inorganic demand further ahead than capacity takes to arrive.
Write thresholds from the objectives before running any test.
Generate load at a fixed arrival rate, never only with users who wait for answers.
Find capacity at the targets, not at the crash point, and note what failed first.
Make the test look like production: real request mix, production-sized data, stubbed third parties.
Plan for failure: enough capacity with a zone down, and a shedding plan above that.
Re-test after significant changes and before big events, and soak before the night.

Forecast
9,000 requests a second on final night: last year's peak plus the half-price offer.
Write the thresholds
95% of order requests under 200 ms and under 1% errors, before any test runs.
Measure one server
An open-model breakpoint test: 600 requests a second within the thresholds.
Plan for failure
Enough servers for 9,000 a second with a zone down, and a shedding plan above that.
Rehearse
A stress test at final-night load and a soak for hours, the week before.

Check yourself

4 questions

1. A closed-model test with 120 users reports 120 ms responses against a system that can serve only 1,000 requests a second. What is happening?

2. Which number should the chain record as one server's capacity?

3. Forecast 9,000 requests a second, 600 per server, three zones, and the plan must survive losing a zone. How many servers?

4. A test with a tiny database and one endpoint shows 2,000 requests a second per server. Why distrust it?

Takeaways

Remember this

Capacity planning combines organic and inorganic forecasts with regular load tests that turn servers into service capacity.
Smoke, average, stress, spike, soak and breakpoint tests answer different questions; set pass/fail thresholds from your objectives.
Closed-model tests slow down with the system and hide overload; generate load at a fixed arrival rate.
A server's capacity is the highest rate that meets the targets, and the first thing to break is the real limit.
Plan servers for the forecast with a failure domain lost, check dependencies too, and scale ahead of known events.