Deployment strategies

An airline's check-in API runs on 120 servers in three regions and handles about 30,000 check-ins an hour. Version 4.8 contains a bug: check-in fails for passengers traveling with an infant, about 2% of all check-ins. Nobody knows that yet. The question is how many passengers find out before the team does.

Roughly 70% of outages are due to changes in a live system, and three practices limit the damage: progressive rollouts, quick and accurate detection, and safe rollback. Deployment strategies are different mixes of those three:

Recreate is the only one with planned downtime, and the only one where old and new versions never run at the same time. Every other strategy has a period when both versions serve traffic side by side, which shapes everything else on this page.

A rolling update replaces old servers with new ones in batches, keeping enough of them serving at all times. Kubernetes Deployments do this by default with two dials: maxUnavailable, how many may be down, and maxSurge, how many extra may be started, both 25% unless set. For the 120 check-in servers, that means at least 90 serving and at most 150 running during the rollout.

A new server only takes traffic once it says it is ready:

Draining matters as much as readiness. A load balancer stops sending new requests to a server being removed but lets in-flight ones finish; AWS load balancers wait 300 seconds by default before completing the removal (Load balancing).

Rolling is cheap, needing only a few extra servers, and is the default for good reason. Its weaknesses: readiness checks catch crashes, not wrong answers, so the infant bug sails through; by the time anyone notices, much of the fleet may run 4.8; and rolling back is another rolling update, minutes long. Kubernetes keeps the last 10 old versions by default so kubectl rollout undo has something to return to.

Blue-green deployment keeps two production environments. Blue runs 4.7 and serves everyone; green gets 4.8 and is tested while idle. Then the router switches all traffic to green in one step. If anything goes wrong, it switches back:

Three catches:

• Double capacity during the switch, or the time to build it.
• The database is usually shared. Schema changes must work for both versions, so they are deployed separately from the application, ahead of it (Zero-downtime migrations).
• All or nothing. The switch exposes every passenger to 4.8 at once. Rollback is fast, but the infant bug still hits 2% of everyone until someone notices.

A canary is a partial, time-limited deployment of a change, plus its evaluation. A small part of production, the canary, runs 4.8; the rest, the control, keeps 4.7; and the two are compared over the same period. If the canary looks no worse, the rollout widens; if it looks worse, it stops.

The share is the point. With 5% of check-ins going to the canary, the infant bug in its first 30 minutes affects far fewer passengers than a full release would have:

The comparison must be fair to be useful:

• Compare canary and control at the same time. A before-and-after comparison mixes the change with whatever else changed that hour, such as traffic or a dependency.
• Pick metrics tied to what users feel, such as error rate and latency by version. A metric that swings wildly with no user impact makes a poor canary signal.
• Run one canary at a time. Overlapping canaries contaminate each other's signal and make the system's state hard to reason about.

Tools encode the steps. AWS CodeDeploy's Canary10Percent5Minutes shifts 10% of traffic, then the remaining 90% five minutes later; its linear options move 10% every one or three minutes. Argo Rollouts lists steps such as setWeight and pause, and can run an analysis in the background that aborts the rollout if it fails. Canary and blue-green also combine: keep the old fleet whole, move traffic in steps, and switch back at any point.

The airline runs three regions, so a bad release should never reach all three at once. Amazon's pipeline does this, and it generalizes the canary to the whole fleet:

During every step, alarms watch fault rates, latency at p50, p90 and p99, resource use, log errors, failing health checks and synthetic canary tests. If any fires, the deployment rolls back automatically, to the previously deployed package. Deployments also run only in time windows when on-call engineers respond quickly, which excludes nights, weekends and holidays.

Zones and cells are natural wave boundaries because they are already failure domains: a bad release confined to one zone is an outage the rest of the system is built to survive (Redundancy & failure domains, Bulkheads). The bake times look slow, but some bugs only appear at a certain hour, on a certain load, or after a cache expires.

Every strategy above ends with "and if it's bad, go back". That assumes the old version can cope with whatever the new version did while it ran. Suppose 4.8 also changed how boarding passes are stored, from XML to JSON. Servers on 4.8 write JSON; roll back to 4.7, and 4.7 servers meet JSON they can't read. The rollback causes a second outage.

The answer is a two-phase deployment. Split the change into two releases, each safe to roll back:

It is the same idea as expand and contract for databases (Zero-downtime migrations) and compatible schema changes for messages (Schema evolution & encoding): make every reader ready before any writer changes. Amazon tests this directly: its one-box stage runs new code beside old, so a mixed fleet is exercised before the wider rollout.

One more way to make going back cheap: decouple deploying from releasing: ship 4.8 with the infant change behind a flag that is off, then turn it on for a few passengers at a time; turning it off is faster than any rollback (Feature flags & configuration).

For the check-in API, the checklist:

• Never deploy everywhere at once: one box, then a small share, then zones and regions in turn.
• Gate every step on readiness and on user-facing metrics, compared with the old version at the same time.
• Roll back automatically on alarms, and keep the previous version ready to run.
• Make every release safe to roll back: prepare readers before activating writers, and change schemas separately.
• Drain servers before stopping them.
• Deploy when people are around to respond, and bake long enough for slow bugs to show.
• Put risky behavior behind flags, so it can be turned off without a deploy.