System designCore11 min
Feature flags & configuration
A feature flag is a decision the running code asks about at runtime: show the new transfer screen to this customer, or not. Flags separate deploying code from releasing it, let a feature reach 1% of users before everyone, and turn a misbehaving feature off in seconds. They are also configuration, and configuration changes can take a system down as fast as code, so they need the same care.
1 · The idea
A flag separates deploying code from releasing a feature
A bank with 3 million customers is building instant transfers. The code takes weeks to finish, touches the transfer screen, the payments API and a new partner bank's network, and must reach customers gradually. Without flags, every one of those decisions is a deploy: merge when finished, release when deployed, roll back by redeploying.
A feature flag moves the decision into the running system:
A flag check in the transfer service
def transfer_options(customer):
options = ["standard"]
# Off by default: if the flag service can't answer, nothing changes
if flags.get_boolean("instant-transfers", False, context={"customer_id": customer.id}):
options.append("instant")
return optionsAspect | Without flags | With a flag |
|---|---|---|
Unfinished code | Kept on a branch until done | Merged and deployed, switched off |
First customers | Everyone, at deploy time | Staff, then 1%, 10%, 50% |
Turning it off | A rollback deploy, minutes | A flag change, seconds |
Who decides when | Whoever deploys | Whoever owns the feature |
This is how large systems release. Facebook ships new code to production early but disabled, then uses a tool called Gatekeeper to enable it incrementally, for employees only or for 1% of the users of one phone model, and to disable it quickly if problems arise. Deployment strategies limit the damage of new code (Deployment strategies); flags limit the damage of new behavior.
2 · Four kinds
Flags differ in how long they live and how often they change
Not all flags are alike, and treating them alike causes trouble. A widely used taxonomy sorts them by two questions: how long the flag will live, and how dynamic its decision is:
Kind | Purpose | At the bank | Lives for |
|---|---|---|---|
Release | Ship unfinished or untested code switched off | instant-transfers until fully launched | Days to weeks, then removed |
Experiment | Send each user consistently down one of two paths, compare results | Two wordings of the transfer button | As long as the test needs |
Ops | Let operators degrade or disable a feature in production | A kill switch for the partner-bank network | Short, or permanently for kill switches |
Permission | Give a feature to certain users only | Instant transfers for premium accounts | Often permanent |
The kinds need different handling. A release flag is temporary inventory and should have a removal date. An experiment flag must not change mid-test, or the results mean nothing. A kill switch must work when everything else is failing, so it should be tested and simple, and the bank's should turn off instant transfers if the partner network slows down, falling back to standard transfers. A permission flag is really product logic, and may belong in the account model rather than a flag system.
3 · Evaluation
A percentage rollout hashes the user, so each user stays in or out
"Turn it on for 10%" can't mean flipping a coin per request, or a customer would see instant transfers on one screen and not the next. Flag systems hash a stable key, usually the user ID, into a fixed range and compare it with the percentage. Unleash hashes the user ID with MurmurHash into a number from 0 to 100; LaunchDarkly divides users into 100,000 buckets. The same user always gets the same number, so the same answer.
That also makes rollouts grow cleanly. Customers whose number is under 1 are in at 1%; at 10%, everyone under 10 is in, which still includes the first 1%. Nobody who had the feature loses it as the rollout widens:
Each step is a flag change, not a deploy: 3,000,000 × 1%, 10%, 50%. Each larger group contains the smaller ones, because a customer's hash never changes. The staff count is the example's.
Where the decision happens matters for speed and failure. Server-side SDKs typically keep all flag rules in memory and evaluate them locally, in nanoseconds per Unleash, receiving changes from the flag service within seconds. If the flag service goes down, the SDK keeps using its cached rules. And if a flag can't be evaluated at all, the code falls back to the default written in the call: in OpenFeature, evaluation never throws and always returns the default on errors.
Requests never wait on the flag service. If it goes down, servers keep their last rules; if a server has none, the default in the code applies, which is why defaults must be the safe, old behavior.
4 · Configuration
A flag change is a production change, and can break production as fast
Flags are one kind of dynamic configuration, values changed in a running system without a deploy: timeouts, rate limits, routing weights, allowed currencies. Facebook reports thousands of online configuration changes a day, and calls configuration errors a major source of site outages. A change that takes effect everywhere in seconds is also a mistake that spreads everywhere in seconds.
Google Cloud's outage of June 12, 2025 shows both halves. A new quota-checking feature had shipped region by region, but its failing code path ran only when a certain policy was present, so the rollout never exercised it. That code path had no error handling and was not protected by a feature flag. Then a policy change was written to a datastore that replicates globally almost instantly:
Had the new code path been flag-protected, enabled region by region starting with internal projects, the issue would have been caught in staging. In the largest regions, restarting tasks then overloaded the datastore, because they lacked randomized backoff.
The defenses are the same as for code, applied to configuration:
- Validate every change against rules before it is accepted, as Facebook's configuration compiler does.
- Review config changes like code changes, and keep them in version control.
- Roll out gradually, never as one global push, and roll back automatically when health checks fail. Together these make a configuration change safe.
- Guard new code paths with a flag that is off by default, so data alone can't wake untested code everywhere.
5 · Flag debt
Old flags are loaded weapons: remove them, and never reuse one
Every flag doubles the paths through its code, and flags that outlive their purpose pile up. The most expensive example on record is from 2012. Knight Capital, a US broker, deployed new order-routing code that repurposed a flag formerly used to activate "Power Peg", functionality it had stopped using years earlier but never deleted:
- The deploy
The new code was copied to seven of the eight routing servers. Nobody noticed that the eighth still had the old Power Peg code, and no second person reviewed the deployment.
- The flag
On August 1, orders arrived carrying the repurposed flag, meant to turn on the new feature. The seven updated servers handled them correctly; on the eighth, the flag triggered Power Peg.
- 45 minutes
Processing 212 small customer orders, the server sent millions of orders to the market: over 4 million executions in 154 stocks, more than 397 million shares.
- The cost
Knight lost over $460 million on the positions it had taken.
Two lessons for every flag system. Remove dead code and dead flags: a code path no one uses but anyone can switch on is a hazard. And never reuse a flag for a new meaning: give every flag a new name, so an old server can't misread it.
Teams keep flag inventory down deliberately. Some add a removal task the day a release flag is created; some give flags expiration dates; some add a "time bomb" test that fails once a flag outlives its date. Record an owner and a purpose for every flag, so nobody has to guess whether it is safe to delete.
6 · Testing
Test the combinations you will actually run, not all of them
Ten flags give 1,024 combinations; nobody tests them all, and nobody needs to. Most flags don't interact, and most releases change one flag at a time. The practical set to test:
- Production as it is now, plus the flags about to be turned on.
- The fallback: the same, with those flags off.
- Everything on, to catch surprises in a future release.
Run the tests with | Flags | It catches |
|---|---|---|
Production as it is now | Today's flags, plus the ones about to turn on | Breakage in the release you are about to make |
The fallback | The same, with the new flags off | A rollback that doesn't work |
Everything on | Every flag on | Surprises waiting for a future release |
That only works with one convention, and it is worth enforcing: off means the old behavior, on means the new. Then the safe default in the code, False, is always the behavior production already has, and a flag service outage, a missing rule or a typo in a flag name all land somewhere known.
Flags can also make problems harder to see. When the bank's support team hears that a transfer looks different, someone must be able to tell which flags that customer had at that moment. Log flag decisions with requests, or at least expose each service's current flag values, so incidents can be traced (Observability: metrics, logs & traces).
7 · In practice
Flags make release safe only if flag changes are safe too
For the bank, the checklist:
- Deploy dark, release by flag: staff, then 1%, 10%, 50%, 100%, watching transfer errors at each step.
- Hash a stable ID for percentages, so each customer stays in or out.
- Evaluate locally from cached rules; never make a request wait on the flag service.
- Default to off, and off means the old behavior.
- Keep a tested kill switch for anything that depends on an outside system.
- Treat every flag and config change as a production change: validated, reviewed, rolled out gradually, rolled back automatically.
- Give every flag an owner, a purpose and an end date, delete it when done, and never reuse its name.
- Deploy dark
The new transfer code ships with its flag off; customers see no change.
- Staff
Turn it on for the bank's own staff, and watch transfer errors.
- 1%, then 10%
A stable hash of the customer ID decides who is in, so each customer stays in or out.
- 50%, then 100%
Keep watching at each step; turning the flag off is the rollback.
- Delete the flag
Once it is fully on and stable, remove it and the old code path.
Check yourself
4 questions
Takeaways
Remember this
- A feature flag moves the release decision from deploy time to runtime, so code ships dark and features reach users gradually.
- Release, experiment, ops and permission flags differ in lifespan and dynamism, and need different handling.
- Percentage rollouts hash a stable user ID, so users stay in as the rollout grows; evaluation is local and falls back to a safe default.
- Flag and config changes are production changes: validate, review, roll out gradually and roll back automatically.
- Remove flags and dead code when done, never reuse a flag name, and keep off meaning the old behavior.