Observability: metrics, logs & traces

A hotel-booking site, Friday evening. Support starts getting messages: the Book button spins for several seconds. The dashboard looks fine. CPU is at 45%, the error rate is 0.2% as always, and the average response time is about 250 ms, well under its 500 ms alert. Here is what was really happening to booking requests:

The payment provider had stopped answering from one of its endpoints. The payment service waited 2 seconds, retried, waited again, and succeeded on the third try. Nothing failed, so the error rate stayed flat; one booking in fifty took 4.3 seconds, which the average barely shows. Retries had turned an outage into slowness (Timeouts, retries, backoff & jitter).

Monitoring watches for problems you predicted: CPU too high, errors above 1%. Observability is the ability to answer questions you didn't predict, from the outside, using what the system records about itself. Both rest on three kinds of data, called signals:

A metric is a number recorded at regular intervals, such as every 15 seconds. It keeps no detail about individual requests, which is why it stays cheap at any traffic: a counter of a million bookings takes the same space as a counter of ten. Three types cover almost everything:

Histograms matter because percentiles can't be averaged. Averaging the p99s of 40 servers doesn't give the p99 of all 40, but bucket counts can be added up across servers and the p99 read off the total (Latency). Had the booking site kept a histogram, its p99 would have shown the problem at 21:40.

Which numbers to keep? Three short checklists:

• RED, for every service: rate of requests, errors, and duration (as a histogram).
• USE, for every resource such as a CPU, a disk or a connection pool: utilization, saturation (work waiting), and errors.
• The four golden signals combine them: latency, traffic, errors and saturation.

Each metric can carry labels, such as endpoint="/book" or status="500", and every distinct combination of label values is a separate time series stored on its own. That count is the metric's cardinality:

• 20 endpoints × 5 status classes × 100 servers = 10,000 series: fine.
• Add a user_id label with 2 million users, and even if each user touched a single endpoint on a single server, that is 2 million more series, every one stored every 15 seconds.

Keep labels to small, fixed sets. Detail about individual users and requests belongs in logs and traces.

A log is a timestamped record of one event: a booking accepted, a payment retried, an exception thrown. Written as free text, it is easy for a person to read and hard for a machine to search. A structured log writes the same event as named fields, so you can ask for every line where provider_endpoint is eu-2:

The last field, trace_id, is what lets you find this line from a slow request later (see below).

Logs grow with traffic, and the bill grows with them. At 2,000 requests a second, five lines per request and 400 bytes per line:

So teams choose what to keep:

• Levels. Mark each line DEBUG, INFO, WARN or ERROR, and switch DEBUG off in production unless you are chasing a bug.
• Sample the routine, keep the unusual. One success line in a hundred is plenty; every error and every retry is worth keeping.
• Never log secrets. Passwords, tokens and card numbers end up in a log store that many people can read and that lives for months. Personal data needs a short, stated retention.

A booking passes through the web tier, the booking service, the availability service, the payment service and the database. Each one only knows its own part. A trace records the whole path. It is made of spans: one span per unit of work, each with a name, a start time, a duration and the ID of the span that called it (its parent). The first span, with no parent, is the root span. Here is the trace of one slow booking:

No metric would have said this so directly. The payment service's average call time was a little high; the trace shows that the slow bookings are exactly the ones that hit two timeouts in a row.

Tracing every request costs as much as logging it, so most systems sample:

• Head sampling decides at the start of the request, say keep 1 in 100. It is simple and cheap, but it decides before anyone knows whether the request will be interesting.
• Tail sampling waits until the trace is complete, then keeps every error, every slow request, and a small share of the rest. It needs a collector that holds all spans for a few seconds, which costs memory and effort.

Head sampling's blind spot is rare failures. A bug that hits 1 in 10,000 requests at 2,000 requests a second fails 0.2 times a second; at 1% sampling you keep one example every 500 seconds, about every 8 minutes. Tail sampling keeps all of them.

How does the payment service know its spans belong to the same trace as the web tier's? The trace ID travels with the request. Each service reads it from the incoming call, records its own spans under it, and passes it on. This is context propagation. Over HTTP, the standard header is traceparent, for example 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01: a version, a 32-character trace ID, the 16-character ID of the calling span, and flags (the final 01 means this trace is being sampled).

If one service in the chain drops the header, say a queue consumer or an old library, the trace breaks into two unrelated halves. Propagation has to work at every hop, including through message queues. With the ID in place, the Friday incident took five steps:

A symptom is what users experience: bookings failing or slow. A cause is why: a full disk, a busy CPU, a slow provider. There are endless possible causes and only a few symptoms, so alerts that wake someone up should watch symptoms. Every page should be urgent, visible to users, and something a person can act on now.

The symptom alert is best written against the service level objective (Availability & reliability). With an objective of 99.9% good bookings over 30 days, the error budget is 0.1% of bookings. The burn rate is how fast you spend it: an error rate of 0.1% is a burn rate of 1, which uses up exactly the budget in 30 days. A burn rate of 14.4 for one hour spends 14.4 × 1 ÷ 720 hours = 2% of the month's budget. A common starting point:

The short second window makes the alert stop soon after the problem does, instead of an hour later. On Friday, 2% slow bookings against a 0.1% budget is a burn rate of 20: the first rule pages within minutes, while a threshold on the average would never have fired.

Most teams use OpenTelemetry, an open, vendor-neutral standard and set of libraries for producing traces, metrics and logs. Its libraries instrument common web frameworks, database drivers and HTTP clients automatically, propagate trace context, and send the data to whichever backend you choose. You add your own spans and attributes where the business logic needs them.

A checklist:

• Record RED for every service and USE for every resource, with latency as a histogram, never an average.
• Keep metric labels small and fixed; put user and request detail in logs and traces.
• Write structured logs with the trace ID in every line, sample the routine, keep every error, and never log secrets.
• Propagate trace context through every hop, including queues, and prefer tail sampling if you can afford it.
• Alert on symptoms, with burn-rate alerts on your objectives, and send cause-level warnings to dashboards and tickets.
• Instrument before the incident: the data you didn't record is the data you will need.