Latency

A shopper in London presses Buy. The shop's servers are in Virginia. About 80 milliseconds later the confirmation appears. That 80 ms is the request's latency as most engineers use the word: the time from sending a request until its answer arrives. Strictly, that end-to-end figure is the response time, and some authors keep latency for the part of it a request spends waiting to be handled. This article uses the everyday meaning, and says wait time when it means the strict one.

Inside the response time there are two very different things. Service time is time spent actually working on the request: running code, reading rows. Wait time is everything else: bytes in flight on the network, the request sitting in a queue behind others. The rule from queueing theory is response time = service time + wait time, and under load it is the wait that grows, not the work.

Latency is measured per request. A service that handles 1,000 requests a second produces 1,000 different numbers every second, and they are not alike: most requests take the short path, a few hit a slow disk, a lock or a pause. There is no such thing as the latency of a service. There is a distribution, and the next section is about how to read it.

Take a minute's worth of response times and sort them. The p50, or median, is the value in the middle: half the requests were faster. The p95 is the value 95% of requests beat; the p99, 99%. Everything to the right of the p99 is the tail. Now compare those with the mean:

The mean is dragged upward by a few very slow requests, so it overstates the typical case and understates the bad one. Percentiles describe real requests, which is why targets are written as percentiles: p99 under 200 ms, not average under 50 ms. And the p99 is not an outlier you can ignore: at 1,000 requests a second it is ten slow requests every second, and a busy user who makes hundreds of requests a day meets it daily.

Two traps catch almost everyone the first time they measure latency. Both come with numbers you can check.

Trap one: averaging percentiles. Two servers sit behind your load balancer. Server B has a failing disk, so the balancer has all but stopped sending it traffic:

A dashboard that averages the two shows (30 + 900) ÷ 2 = 465 ms, and someone gets paged. The real p99 of the 10,000 requests is about 30 ms. Why? The slowest 1% is 100 requests. B supplied only 10 of them; the other 90 are A's slowest, which sit right at A's own p99. The average gave ten requests the same vote as 9,990.

The same thing happens over time: the p99 of an hour is not the mean of sixty one-minute p99s. Keep the raw values, or histograms you can add together, and compute the percentile from the merged data.

Trap two: the load test that waits politely. Your test sends a request, waits for the answer, then sends the next: 100 a second, 10 ms each. Halfway through, the server freezes for 10 seconds. The tester is stuck waiting on one request, so it logs one slow request and carries on. Real users do not wait their turn. They keep arriving, 100 a second, all through the freeze.

This is coordinated omission: the tester and the server stall together, so the test skips exactly the requests that would have suffered. The report says p99 = 10 ms for a system that made a thousand people wait. The fix is to send on a fixed schedule whatever the server does, and to time each request from the moment it should have been sent.

Why does crossing the Atlantic cost about 70 ms? Because light is slow. In glass fiber it manages about two-thirds of its vacuum speed: 299,792 km/s ÷ 1.468, the refractive index of standard fiber, is roughly 204,000 km/s. One rule falls out of that, and it is worth remembering: every 100 km of distance costs about 1 ms of round trip.

New York to London is about 5,570 km in a straight line, so physics alone charges 55 ms there and back. Measured round trips on that route sat at about 70 ms throughout 2025. The other 15 ms is the cable, which does not run in a straight line, and the routers along it. No code change removes any of this. The only way to pay less is to be closer.

Distance and handshakes are the network's share. The rest of the time goes to things inside the machines, and each has its own scale:

Every engineer should carry a rough price list in their head: what a memory read, a disk read and a network hop each cost. The digits shift with every hardware generation, and the lists copied around the internet often still show figures from 2010. So learn the order of magnitude of each rung, not the digits:

Read it as a ladder. Memory to SSD is about 300×; SSD to spinning disk about 300× again; disk to an ocean crossing about 7×. So one disk read costs about 10 ms ÷ 100 ns = 100,000 memory reads, and one transatlantic round trip about 700,000. That is why a design that adds one more round trip per item, or one more disk read per request, is never a small change.

The checkout page does not make one backend call. It asks the cart service, the pricing service, the stock service, the recommendation service, and each of those asks others: fan-out. The page is ready when the slowest answer arrives. If each backend is slow on 1% of calls, and the slow calls are independent, the chance that a page with n calls hits at least one slow call is 1 − 0.99ⁿ.

Rarer tails do not escape either: a p99.9 event, 1 in 1,000, hits 1 − 0.999¹⁰⁰⁰ ≈ 63% of requests that fan out to 1,000 servers. At scale the tail is not caused by broken machines but by ordinary ones: a background daemon, a shared network switch, a garbage-collection pause, a queue that happened to be long. You cannot remove those, so wide systems are designed to be tail-tolerant instead: they finish quickly even when a few parts are slow.

Before optimizing code, find the biggest term in the sum. In the checkout example that is 70 ms of distance and 140 ms of handshakes, against 10 ms of work. Every move below removes a term:

Parallel calls make the page wait for the slowest answer, the tail from the chart above. The fix for that is the hedged request:

• Send the request to one replica.
• If no answer has arrived by the time the p95 latency has passed, send a copy to a second replica.
• Take whichever answers first and cancel the other.

Only the slowest 5% of requests ever get a second copy, so the extra load is about 5%, while the tail shrinks. Hedge only calls that are safe to run twice, such as reads, and only when the replicas exist.