Error budgets & chaos experiments

A city bike-share app lets riders unlock bikes from their phones. Its objective: 99.9% of unlock requests succeed, over a rolling four-week window (Availability & reliability). In four weeks the app receives about 2 million unlock requests, so its error budget, 100% minus the objective, is 0.1% of them: 2,000 failed unlocks.

The budget exists to settle an argument every team has. Product engineers want to ship; the people paged at night want stability. 100% is the wrong target: impossible to reach, and more reliability than users notice. A budget agreed in advance gives both sides one number: while unlocks fail less often than allowed, ship; when the budget is gone, stability comes first. Changes are the main thing that spends it; Google attributes roughly 70% of its outages to changes in a live system.

Two details of the window matter. A rolling window matches how riders feel, because an outage on the last day of a month isn't forgotten on the first day of the next. And it should be a whole number of weeks, so each window holds the same number of weekends, when bike-share traffic is very different.

Without consequences agreed in advance, a spent budget is just a sad number on a dashboard. An error budget policy has this shape, here as the bike-share team writes it:

The example policy states its non-goal plainly: it is not a punishment for missing objectives. It gives the team permission to work only on reliability when the data says reliability matters more than features. It also works in the other direction: when the budget is large, developers can take more risks, and when it is nearly drained, they tend to ask for more testing and slower releases themselves, because they don't want to stall their own launches.

Four weeks into the new season, the bike-share team reviews where its 2,000 failed unlocks went:

The policy now applies twice. The release used 45% of the budget, so it gets a postmortem with a top-priority fix, here an automatic rollback on failing unlocks (Deployment strategies). And with 150 failures left for the rest of the window, the team holds risky changes until the window rolls past the incident.

Waiting for the end-of-window review is too late to act, so teams alert on how fast the budget is burning (Observability: metrics, logs & traces). Common starting points: page someone when 2% of the budget burns in one hour (a burn rate of about 13.4 over the 4-week window, 14.4 over 30 days) or 5% in six hours (about 5.6, or 6 over 30 days), and open a ticket when 10% burns in three days. At the bike-share's budget, 2% is 40 failed unlocks in an hour.

The team's design says the app survives losing one of its three zones (Redundancy & failure domains). Nobody has ever checked. Chaos engineering is the discipline of experimenting on a system to build confidence that it withstands turbulent conditions in production. Its experiments follow four steps:

The events should be the ones that actually happen: servers that die, disks that fail, malformed responses, traffic spikes, prioritized by how likely and how damaging they are. The weaknesses experiments find are rarely in single components; they are in interactions. The principles list examples: wrong fallback settings when a service is down, retry storms from badly tuned timeouts, a dependency overwhelmed by too much traffic, and cascades when a single point of failure crashes.

The bike-share's experiment finds one. With a zone gone, unlocks keep working, but the remaining zones' connection pools to the dock-status cache run out within four minutes, because each pool was sized for a third of the traffic, not half. That is a fix for a quiet Tuesday, not for a real zone failure on a sunny Saturday.

Production is where experiments are most authentic, because real traffic exercises paths no test environment does; the principles of chaos engineering strongly prefer it. They also make it the experimenter's duty to keep the blast radius small. Plan each experiment first, and run it in a pre-production environment before production. In practice the ladder looks like this:

Every experiment needs a way to stop itself. In AWS FIS, a stop condition is an alarm on the steady state; if the alarm fires, the experiment stops, and a stopped experiment can't be resumed:

And experiments spend the budget, by design. The policy should say so: run planned experiments while budget remains, scale them down when it is low, and stop when it is spent. A team that refuses ever to spend budget on experiments will spend it later, unplanned, on the failures the experiments would have found.

Software isn't the only thing that fails during an outage. Pages don't reach anyone, runbooks are out of date, nobody has the permission to fail over the database. A game day simulates an event in a production-like environment and has the teams who would really respond do exactly what they would do. Run them regularly, until responding is a habit.

Game days go wrong in familiar ways, and each has a bike-share version:

• Procedures documented but never exercised: the failover runbook names a dashboard that no longer exists.
• Business decision makers left out: nobody decides whether to waive ride fees during the outage.
• Stakeholders not told: support learns of the game day from confused riders.
• Lessons not fed back: the same gap appears at the next game day.
• Blame: people who find problems stop reporting them.

Game days and chaos experiments overlap: an experiment tests whether the system survives an event, a game day whether the organization does. A zone-loss experiment run as a game day tests both at once, at the cost of a few minutes of budget.

For the bike-share team, the checklist:

• Set the objective from what riders notice, over a rolling window of whole weeks.
• Write the policy before it is needed: what freezes, what triggers a postmortem, who decides.
• Charge every incident to the budget, and postmortem anything above 20%.
• Alert on burn rate, not only on the window total.
• Turn design assumptions into chaos hypotheses, starting with failures that happen most.
• Climb the ladder: pre-production, small production scope, full scope, automated.
• Give every experiment a stop condition tied to the steady state.
• Run game days with the real responders and the business, and fix what they find.