System designDeep dive12 min
Multi-region replication
Copying data to another building protects against a broken machine. Copying it to another continent protects against a broken region and brings data closer to users, but every copy is now tens to hundreds of milliseconds away. Multi-region design is deciding, for each kind of data, who may write where and what happens when two regions disagree.
1 · The problem
One region is far from most users and one outage away from none
A project-board app lets teams move task cards between columns. Everything runs in one cloud region on the US East Coast: app servers, database, cache. Teams in Berlin and Singapore have started complaining that the board feels sluggish. Distance explains it. These are measured median round trips from a US East region (Azure's published figures, 2026):
Median round-trip times between cloud regions. Most of each number is the speed of light in fiber over the route (Latency); no amount of server tuning removes it.
The obvious fix backfires. Move only the app servers to Singapore, and every database query crosses the Pacific instead. A board page that runs four queries one after another now waits 4 × 224 ms ≈ 900 ms before it can render. The data has to move too.
Latency is one of three reasons to run in several regions:
- Latency: users are served from a nearby copy.
- Region outages: a whole region can fail. On October 19 and 20, 2025, a DNS automation bug left one AWS region's DynamoDB endpoint unresolvable for about three hours, and some dependent services took more than half a day to recover fully. Customers with replicas in other regions could keep serving from them.
- Data residency: some laws and customer contracts restrict where personal data may be stored or sent, such as the EU's rules on transferring personal data outside the EU.
Copies across regions are kept in sync by replication (Database replication). What changes at this distance is the cost of waiting for a copy, and that cost decides everything that follows.
2 · Reads
Copies in every region make reads local, not writes
The simplest step keeps one primary region for writes and adds an asynchronous read replica in each other region:
Reads are served in a few milliseconds from the local copy. Every card move still crosses the ocean to the primary.
Managed databases offer this shape directly. Aurora Global Database, for example, has one primary region and up to 10 read-only secondary regions, replicating with a lag that is typically under a second. For a board that is read far more often than it is changed, this alone makes the app feel local.
The catch is the lag. A user in Singapore drags a card to Done; the write goes to US East; the page reloads from the Singapore replica, which hasn't received the change yet, and the card jumps back. That is the read-your-own-writes problem across an ocean (Consistency models). Common fixes:
- Read your own recent changes from the primary, for example for 2 seconds after a write.
- Remember the version of the last write in the session and wait until the local replica has caught up to it.
- Update the screen from the write's response, not from a fresh read.
3 · Failover
An asynchronous copy is behind, so a region failover loses its latest writes
Replicas in other regions also make a standby: if US East fails, the Europe replica can be promoted to primary. Two numbers describe how well that goes:
- RPO (recovery point objective): how much recent data you can lose. With asynchronous replication it equals the replication lag at the moment of failure.
- RTO (recovery time objective): how long until writes work again.
The write was acknowledged, then lost: it existed only in the region that failed. When US East comes back, its unreplicated writes conflict with what Europe accepted since.
Three questions make failover between regions harder than failover inside one region:
Question | Why it is hard | What helps |
|---|---|---|
Is the region really down? | From Europe, a dead region and a broken link look the same; promoting too early creates two primaries (Failure detection & split brain) | A person makes the final call, or a third region breaks the tie |
What happens to stranded writes? | Writes acknowledged in the failed region reappear when it returns | Decide in advance: discard, replay or merge them |
Can failover run without the failed region? | Its control systems, DNS or deploy pipeline may be down too | Capacity already running and data already copied, so only traffic moves |
A failover you have never run is a hope, not a plan. Planned switchovers, which move the primary with no data loss while both regions are healthy, are the way to rehearse it; some teams move the primary on a schedule for exactly that reason. The wider plan, with backups and recovery targets, belongs to Disaster recovery & backups, and routing users to the surviving region to Health checks & failover.
4 · Writes everywhere
Writing in every region lets two people change the same card at once
To make writes local too, every region accepts writes and replicates them to the others: active-active, or multi-leader, replication. Card moves in Singapore now take milliseconds. The new problem arrives within a week:
Both writes succeeded locally, within the same fraction of a second, before either region heard of the other. Both users were told "saved".
The most common automatic answer is last writer wins: keep the version with the later timestamp and discard the other. It always converges, and it always loses something. In DynamoDB global tables, for example, the later write's version of the whole item wins, so Raj's new assignee can disappear because Ana changed the title, a different field. Those lost writes are not even logged. Timestamps from different machines also disagree, so "later" may be wrong (Time, clocks & ordering).
Strategy | How it resolves | Fits |
|---|---|---|
Last writer wins | Keep the later timestamp, drop the other write | Values where any recent answer will do: a status flag, a last-seen time |
Merge by data type (CRDTs) | Types designed so concurrent changes always combine: counters add, sets merge | Counters, a card's label set, collaborative text |
Merge in the application | Keep both versions and merge per field, or ask a person | Documents and records where every edit matters |
Avoid the conflict | Route all writes for one record to one region | Anything where losing a write is not acceptable |
CRDTs (conflict-free replicated data types, CRDTs: data types that always merge) let each region update its copy without coordination, with a guarantee that all copies converge. Redis's active-active databases, for example, are built on them. They work when the data has a natural merge: a "watchers" set where adds from two regions both survive. They can't decide which of two titles is right.
5 · The usual fix
Give every record a home region
Most conflicts disappear if each record is written in only one region. Every board gets a home region, usually where its team works, and all writes for that board go there. Every region still holds a read copy of everything:
Each region is the primary for its own boards and a read replica for everyone else's. Writes are local for the people who make most of them, and no record has two writers.
Each region acts as the primary for its own share of the data, which brings back the simple single-writer rules record by record. Databases support this directly: CockroachDB's regional-by-row tables store a home region in every row, and DynamoDB global tables support the same pattern, "write to your region". It also helps with residency: a European customer's boards can live, and be written, in Europe.
The edges need rules of their own:
- A team spread across continents: one side always pays the round trip to the board's home.
- Moving a record's home (a team relocates) is a small migration: stop writes, let replication catch up, then switch.
- Data everyone reads and few change, such as plans and prices, can trade slower writes for fast reads in every region; CockroachDB calls these global tables.
- When a home region fails, its records need a new home, which is the failover problem again, one region's share at a time.
6 · Strong consistency
Agreement across regions costs a round trip on every write
Some data can't accept any lost or conflicting writes: billing, permissions, the last free seat on a paid plan. For those, a write is committed only when a majority of replicas in different regions have it, using consensus (Consensus & leader election). Spanner, CockroachDB and DynamoDB's multi-region strong consistency mode all work this way. A committed write survives the loss of a whole region: the RPO is zero.
The price is a cross-region round trip on every write. With replicas in US East, West Europe and Southeast Asia, a write needs 2 of 3, so the leader waits for its nearest other region:
Minimum commit wait for a majority of three regions: the round trip from the leader to its nearest other replica, using measured median round trips for each pair; the two directions of one pair differ slightly. A client far from the leader adds its own trip to the leader on top.
Placement matters as much as the protocol. Put the leader where most writes come from, put two of the three regions close together if most users are there, and use a third region as the tie-breaker. Two regions can never form a majority after losing one, which is why these systems ask for three, and why DynamoDB's strong mode lets the third be a lightweight witness that only votes. Reads can stay local when slightly old data is acceptable.
7 · In practice
Choose per kind of data, not per system
The board app ends up with several patterns at once, each matched to what the data can tolerate:
Pattern | Writes | If a region fails | Board app uses it for |
|---|---|---|---|
One primary + read replicas | Cross-ocean for remote users | Promote a replica; lose the lag (RPO > 0) | Plans and prices, rarely changed |
Home region per record | Local for each record's owners | Its records move home; recent writes may be lost | Boards and cards |
Active-active with merges | Local everywhere | Other regions carry on; merge on return | Watchers, reactions, last-seen times |
Consensus across 3 regions | One cross-region round trip | No data lost; writes continue in the majority | Billing and permissions |
Before calling it multi-region:
- Copy everything the app needs, not just the database: caches, queues, object storage, secrets and configuration.
- Measure replication lag per pair of regions and alert on it, since it is your RPO.
- Make writes idempotent (Idempotency), so writes replayed after a failover or a merge apply only once.
- Rehearse region failover on a schedule, without depending on the region being evacuated.
- Keep each request in one region. A request that hops between regions pays the ocean several times.
Check yourself
5 questions
Takeaways
Remember this
- Cross-region round trips are tens to hundreds of milliseconds, so move the data with the users, not just the servers.
- Read replicas in every region make reads local; writes still travel to the primary, and replication lag breaks read-your-writes.
- With asynchronous replication, a region failover loses whatever had not been copied: the RPO is the lag.
- Writing in every region creates conflicts; last writer wins loses data silently, so prefer a home region per record or data types that merge.
- Strong consistency across regions costs a round trip to the nearest other region on every write and needs at least three regions.