System designCore8 min
Availability & reliability
What 99.99% available really allows, why a checkout that is up 99.9% of the time can still fail most of your customers, and why recovering faster is worth exactly as much as failing less often.
1 · Counting nines
Each nine is ten times less downtime
Availability is the share of the time, or of the requests, in which a service does its job when asked. It is quoted in nines, and each extra nine divides the allowed downtime by ten.
Availability | Down per year | Down per 30 days |
|---|---|---|
99% (two nines) | 3.65 days | 7.2 hours |
99.9% (three nines) | 8.76 hours | 43.2 minutes |
99.95% | 4.38 hours | 21.6 minutes |
99.99% (four nines) | 52.6 minutes | 4.32 minutes |
99.999% (five nines) | 5.26 minutes | 25.9 seconds |
What those numbers mean in practice:
- Three nines leaves 43 minutes a month. One bad hour breaks the promise.
- Four nines leaves 4.3 minutes a month: less than it takes a person to be paged and log in. Detection and recovery have to be automatic.
- Five nines leaves 26 seconds a month. One slow restart spends it.
There are two ways to count:
- By time: uptime ÷ (uptime + downtime).
- By requests: successful requests ÷ total requests.
Large services count requests, because a big system is rarely fully down: it is a little broken for some users. Counting requests also charges you for what users actually felt. Ten minutes down at 3 a.m. costs a few hundred requests; the same ten minutes at the evening peak costs many times more.
2 · The vocabulary
SLI is measured, SLO is targeted, SLA is promised
Three terms that get blurred into one, shown for a checkout API:
Term | What it is | Checkout example |
|---|---|---|
SLI (indicator) | What you measure | Share of requests that succeed in under 500 ms |
SLO (objective) | The target you hold yourself to | 99.95% of requests over any 30 days |
SLA (agreement) | A promise to customers, with a penalty | 99.9% over a month, or a service credit |
Error budget | 100% minus the SLO | 0.05% of requests: spend it on releases and experiments |
- The SLA is looser than the SLO, on purpose. Promise customers 99.9%, hold yourself to 99.95%, and a chronic problem shows up while it is still your problem and not yet a refund.
- The error budget is the unreliability you are allowed. At 99.95% over 30 days that is 21.6 minutes of full outage, or 64,800 failed requests out of 129.6 million at 50 a second.
The budget turns reliability into a decision rule: while budget remains, ship. When it is spent, releases stop until the window rolls forward.
A 99.95% SLO over 30 days is 21.6 minutes. Background errors cost about 0.1 minutes a day, an incident on day 6 costs 9 and a bad deploy on day 19 costs 7.5: 2.1 minutes are left. Anything else this month means a release freeze.
3 · Four words
Available, reliable, durable and fault-tolerant are four different promises
"Reliable" gets used for all four. They are different promises, and a system can keep one while breaking another.
Word | The question it answers | You can have it without… |
|---|---|---|
Availability | What share of the time does it work when asked? | Reliability: down for one second every hour is 99.97% available, and fails constantly |
Reliability | How likely is it to run with no failure for a given period? | Availability: 360 flawless days, then 5 days down, is reliable and only 98.6% available |
Durability | Does stored data survive, whether or not you can read it right now? | Availability: a store in an outage keeps every byte and serves nothing |
Fault tolerance | Does it keep working through a failure, with no visible interruption? | High availability settles for less: a 30-second failover is a visible blip |
Numbers make the first two easy to tell apart. Two checkout services are both 99.9% available over 30 days:
- Service A drops out for 43 seconds every 12 hours.
- Service B drops out once a month, for 43 minutes.
A five-minute checkout has about a 0.8% chance of hitting a failure on A, and about 0.1% on B. Same availability, but B is the more reliable one.
Durable but unavailable: for these 30 seconds no order can be placed, and not one existing order is at risk. The two properties are measured separately and bought separately.
- Durable, not available. Cloud object storage is typically designed for 99.999999999% durability and 99.99% availability over a year: eleven nines that you keep the object, four that you can fetch it today. Both are design targets, not measured history.
- Available, not durable. A cache answers every request and loses everything on restart.
- Fault-tolerant or highly available. Fault tolerance masks the failure: a server with two power supplies loses one and nobody notices. High availability keeps the interruption short: a standby takes over in seconds. Fault tolerance costs more, so most systems settle for high availability.
4 · The numbers
Fail less often or recover faster: the formula treats them the same
Two averages drive availability:
- MTTF (mean time to failure): how long the system runs before it fails.
- MTTR (mean time to repair): how long from failure to working again.
Availability = MTTF ÷ (MTTF + MTTR)
You will also meet MTBF (mean time between failures), which is MTTF + MTTR. Many texts put MTBF in the formula instead; MTTR is so small by comparison that the result barely moves.
The checkout fails about once a month (MTTF 720 hours) and takes 4 hours to fix: 720 ÷ 724 = 99.45%, about 48 hours down a year. Now improve it two ways:
Downtime per year = 8,760 h × MTTR ÷ (MTTF + MTTR). Halving MTTR halves the downtime, and so does doubling MTTF. The two factors are equally important.
Dividing MTTR by ten buys exactly what multiplying MTTF by ten buys. The two do not cost the same, though:
- Failing less often means finding bugs you have not hit yet.
- Recovering faster means better alerts, a rollback button and a written playbook. A prepared playbook alone makes recovery roughly 3× faster than improvising.
Recovery is usually the cheaper lever.
One trap: the R in MTTR can mean repair, recovery, respond or resolve, and each stops a different clock. Ask which one a number means before you compare it with anyone else's.
5 · The catch
Dependencies in a row multiply availability down
A checkout request needs the load balancer, the API, the auth service and the database, all of them, in one request. It succeeds only when every one is up. If their failures are independent, multiply their availabilities.
0.9999 × 0.999 × 0.999 × 0.999 ≈ 99.69%: worse than every single part, about 2.2 hours down per 30 days.
Every hard dependency costs you, even an excellent one. Three four-nines services in a row: 99.99% × 99.99% × 99.99% = 99.97%.
0.999ⁿ: ten three-nines dependencies in a row give 0.999¹⁰ ≈ 99.0%, about 7 hours a month. Each link you add takes another 43 minutes.
Two ways out:
- Make the dependency soft. If the recommendations service is down, show the checkout without recommendations, and it drops out of the product.
- Give the link a copy, so it fails only when every copy fails. That is redundancy, and it wins nines back only when the copies fail independently: see Redundancy & failure domains.
Check yourself
5 questions
Takeaways
Remember this
- Each nine is ten times less downtime; four nines is 4.3 minutes a month, so recovery must be automatic.
- SLI is what you measure, SLO is the target, SLA is the looser promise with a penalty; 100% minus the SLO is your error budget.
- Availability, reliability, durability and fault tolerance are four different promises: a store can be durable and unavailable, a cache available and not durable.
- Availability = MTTF ÷ (MTTF + MTTR): dividing MTTR by ten equals multiplying MTTF by ten, and it is usually the cheaper one.
- Hard dependencies in a row multiply availability down; copies in parallel win it back only if they fail independently.