System designDeep dive10 min
Anycast & global traffic routing
A service in three regions still needs one address that users everywhere can type. Anycast announces that address from every site at once and lets the internet's routing pick the nearest; global load balancers build on it to choose a region by health and spare capacity, not just by distance.
1 · The choice
Something must pick a region before any of your servers sees the request
A weather data company sells an API that phone apps in 150 countries call every few minutes for forecasts. It runs in three regions, Frankfurt, Virginia and Singapore, each with its own load balancer. A user in Lisbon should land in Frankfurt; if Frankfurt is down or full, somewhere else. Inside a region a load balancer picks the server (Load balancing), but something has to pick the region first.
There are three ways to make that choice, and they decide at different layers:
Method | Who decides | When it changes its mind |
|---|---|---|
DNS steering | The authoritative DNS server, returning a different address per asker | As caches expire, after up to one TTL (DNS resolution) |
Anycast | The internet's routers, sending each packet toward the nearest site announcing the address | When a site stops announcing it and routes update |
Global load balancer | Edge proxies reached by anycast, which then pick a region by health and capacity | On the next new connection |
DNS resolution covers the first. This page covers the other two, which share one trick: the same address in many places.
2 · Anycast
Anycast announces one address from many places and lets routing pick
Normally an address belongs to one place: every router on the internet has a route that leads to one network. With anycast, two or more sites announce a route to the same address block. Routers between a user and the service each see several paths and pick one by their own routing policies, so each packet goes to whichever site is closest in network terms. Across the internet the announcements travel through BGP, the protocol networks use to tell each other which addresses they can reach.
Every site announces the same block. A packet from Lisbon follows its routers' best path, which ends in Frankfurt; a packet from Jakarta ends in Singapore. The app never knows there is more than one site.
The set of networks whose traffic ends up at one site is that site's catchment. Catchments follow the shape of the internet, not the map: being close in routing terms doesn't always mean a short round trip, and some users may be sent to a site farther away than another. The address never changes, though, so nothing has to expire from a cache when a site is added or removed.
The DNS root is the best-known user. Its 13 server identities, run by 12 independent organizations, are served by over 2,000 instances around the world, all reached through anycast. Large public DNS resolvers, CDNs (Content delivery networks (CDNs)) and DDoS protection services work the same way.
3 · Failover
A site leaves by withdrawing its route, and its users flow to the next one
If the Frankfurt site fails, anycast only helps once Frankfurt stops announcing the address; otherwise routers keep sending Lisbon's packets to a broken site. So the announcement is tied to health: software on each site announces the route while the service passes its checks and withdraws it when it fails. Routers then pick their next best path, and Frankfurt's catchment drains to Virginia or Singapore with no DNS change at all.
Rule | Why |
|---|---|
Withdraw only on real failure | A route that flaps on and off disrupts users and the routers carrying it |
Wait before re-announcing | A minimum delay after a withdrawal stops rapid oscillation |
Size each site well above average load | A failed site's whole catchment lands on its neighbors at once |
Don't withdraw under a DDoS attack | The attack simply moves to the next site, which can fail in turn |
The last rule is also anycast's biggest advantage against attacks. Traffic from a botnet spread around the world is split across every site's catchment, so each site absorbs only its share, and an attack from one region stays near that region. Keeping the overloaded site announced confines the damage; withdrawing it would hand the whole attack to the next site (DDoS protection & WAFs).
4 · Connections
Routes can change in the middle of a connection
Anycast routes packets, not connections. If routing changes while a connection is open, the next packet may arrive at a site that has never heard of it:
UDP exchanges of one packet each way, such as most DNS queries, don't care. A TCP connection that is split between two sites is reset and has to start again.
How much this matters depends on how long connections last compared with how long routes stay stable. Short exchanges rarely notice, which is why CDNs serve web traffic over TCP on anycast addresses despite the risk. Long downloads, video streams and long-lived WebSockets are more exposed. Two ways to limit the damage:
- Clients that retry. A reset connection reconnects and lands on the new site (Timeouts, retries, backoff & jitter).
- Anycast only for the first step. Use the anycast address to find a nearby site, then hand the client a site-specific address for the long session.
5 · Global load balancers
A global load balancer uses anycast to reach the edge, then chooses the region itself
Plain anycast has a weak spot: load follows the map, not the servers. Each site gets whatever its catchment sends, so anycast spreads load only coarsely, and there's no routing setting for "send 10% of Europe to Virginia". Cloud global load balancers (Google Cloud's external Application Load Balancer, AWS Global Accelerator) split the job in two:
Anycast only has to get the user to a nearby edge. The edge proxy ends the user's connection there and opens its own to the region it chooses, using health checks and capacity that routing knows nothing about.
Ending the connection at the edge has three effects:
- Faster handshakes. TCP and TLS setup round trips happen over the short hop to the edge, not across an ocean.
- A better road. The long leg runs on the provider's own network instead of the public internet.
- Real choices. The proxy can pick a region by health, weights and spare capacity, and change its pick on the next connection.
6 · Capacity
The nearest region fills first, and the rest overflows to the next nearest
Google's default algorithm, Waterfall by Region, sends users to the closest region until its backends reach the capacity you configured, then sends the excess to the next closest region, and so on. At 7 a.m. in Europe the weather API sees 4,600 requests a second from European apps, and the Frankfurt backends are set to serve 4,000:
The 600 extra requests a second pay for a trip across the Atlantic instead of queueing in Frankfurt. As autoscaling adds capacity, the overflow shrinks back to zero.
Health works the same way. If more than half of a region's backends fail their health checks, Google's edge proxies start moving traffic to the next closest region before the rest give out. AWS Global Accelerator sends each new connection to a healthy endpoint as soon as one is marked unhealthy, and offers two dials for planned moves:
- Traffic dial, per region: accept only a percentage of the traffic that would go there, and send the rest elsewhere. Dialing Frankfurt to 0 drains it for maintenance.
- Weights, per endpoint within a region: split a region's traffic, for example to give a new version 10%.
Existing connections are the catch. Global Accelerator keeps sending an established connection to its endpoint until it has been idle for the timeout (340 seconds for TCP, 30 for UDP), even if that endpoint is now unhealthy. Long-lived connections must be closed and reopened to move (Health checks & failover).
7 · In practice
Rent anycast, and match the method to how fast traffic must move
Running anycast yourself means owning an address block, peering with networks in every location and operating routing; most teams rent it, from a DNS provider, a CDN or a cloud global load balancer. The weather API ends up with all three layers, each doing what it does best:
Layer | Job | Speed of change |
|---|---|---|
Anycast DNS provider | Answers the API's name from nearby, survives DNS attacks | Never needs to change |
Global load balancer | One static address, edge proxies, region choice by capacity and health | Next connection |
Regional load balancers | Pick a healthy server within the region | Next request |
- Give clients one stable address
Static anycast addresses are also easy for customers to put in firewall allow-lists.
- Set real capacities
Overflow is only as good as the per-region capacity numbers behind it; measure them with load tests (Load testing & capacity planning).
- Plan for the neighbors
When a region drains, its traffic lands on the next closest; each region needs headroom for that (Multi-region replication).
- Make clients reconnect
Routes and edges change under open connections; clients should retry and reconnect without user-visible errors.
- Measure from outside
Probe from many networks and countries; a routing problem in one region's internet is invisible from inside your own.
Check yourself
4 questions
Takeaways
Remember this
- Anycast announces one address from many sites, and routing sends each user to the nearest one in network terms, with no cache to expire.
- Anycast failover is a route withdrawal tied to health checks; don't withdraw under attack, or the attack moves to the next site.
- Routes can change under an open connection, so anycast suits short exchanges best and long sessions need clients that reconnect.
- Global load balancers use anycast only to reach a nearby edge proxy, which ends the connection and picks a region by health and capacity.
- The nearest region fills to its configured capacity and the rest overflows to the next nearest, so capacity numbers and headroom matter.