Service discovery & registration

An online chess platform runs a game service, a matchmaking service and a ratings service on Kubernetes. Matchmaking calls ratings for every pairing. At 18:00 the evening peak arrives and the autoscaler grows ratings from 4 instances to 20; at 01:00 it shrinks back. In between, a deploy replaces every instance once, and a failing node moves three of them elsewhere. Every one of those events gives instances new IP addresses:

A config file listing ratings' addresses would be wrong within the hour. Service discovery replaces it with two mechanisms:

• Registration: keeping a live record of which instances of each service exist, where, and whether they are healthy. The record is the service registry.
• Discovery: getting that record, or a request routed by it, to every caller.

The caller asks for "ratings", a stable name, and never stores an address for long.

There are two ways for instances to get into the registry:

• Self-registration: the instance registers itself at startup, renews its entry with periodic heartbeats, and deregisters at shutdown. Netflix's Eureka works this way.
• Third-party registration: the platform registers instances for them. Kubernetes watches which pods match a service and are ready, and keeps that list in EndpointSlices; the application does nothing.

Registration alone isn't enough, because a registered process can be alive and useless. Good registries also run health checks and list only instances that pass them: Consul returns only healthy instances, and Kubernetes includes only pods whose readiness check passes (Health checks & failover). Third-party registration has one more advantage: a frozen process can't deregister itself, but the platform can remove it.

The registry is now something every call depends on, so it is replicated. Consul and etcd, which backs Kubernetes, keep their data consistent across several servers using consensus (Consensus & leader election). Eureka leans the other way, toward staying available: its clients cache the registry, and its servers would rather keep stale entries than wipe them (more below).

Once the registry knows the instances, a request must reach one of them. In client-side discovery, the caller fetches the list and load-balances itself:

In server-side discovery, the caller sends the request to a stable address and something in between picks the instance. That can be a load balancer, a virtual IP handled on every machine, or a proxy running next to each service:

A service mesh puts a proxy such as Envoy next to every service. A control plane pushes each proxy the current endpoints of every service it calls (Envoy's Endpoint Discovery Service), so the application only ever talks to localhost. It gets client-side discovery's direct calls and per-request balancing without a library in each language, at the cost of running the mesh.

Every language can resolve a name, so most systems expose discovery through DNS. In Kubernetes, the ratings service is reachable as ratings.chess.svc.cluster.local; Consul answers names like ratings.service.consul with only healthy instances. SRV records add the port to the address, for services that don't run on a fixed one.

Two kinds of answer behave very differently:

• A normal Kubernetes Service resolves to one stable virtual IP. The name never changes its answer; the list behind it changes, and kube-proxy on each machine follows it.
• A headless Service (no virtual IP) and Consul DNS resolve to the instance addresses themselves, so the answer changes as instances come and go.

The second kind inherits every DNS caching problem. A client that caches an answer keeps calling the old addresses until the cache expires, and some runtimes cache until the process restarts (DNS resolution). DNS answers are also limited in size and say nothing about load, zones or which instance is new. That is why meshes push endpoint lists over their own protocol instead of relying on DNS.

Matchmaking calls ratings over gRPC, which sends all requests over one long-lived HTTP/2 connection per destination. Through a normal Kubernetes Service, kube-proxy chooses a pod once per connection. At 18:00 ratings scales from 4 to 8 pods. Matchmaking's connections were opened at 17:00, so this is how the load looks after the scale-out:

Discovery worked: the new pods are registered and ready. Balancing per connection just never consults the new list, and in the worst case, a single caller with a single connection, one pod gets everything. The fixes move balancing to the request:

• Client-side balancing across instance addresses from a headless Service, with the client opening connections to each.
• A layer 7 proxy or service mesh that balances each request across the current endpoints.
• Limiting connection age, such as gRPC's MAX_CONNECTION_AGE, so clients reconnect periodically and land on the new pods. It is a partial fix: balance improves only as connections are renewed.

Every discovery system lags reality. In Eureka, changes can take up to about 2 minutes to reach all clients; DNS answers live until their TTL expires; health checks need several failures before acting. A caller will sometimes pick an instance that has just died, and a new instance will sometimes wait to receive traffic. Make that harmless:

• Short timeouts and a retry on a different instance for idempotent calls (Timeouts, retries, backoff & jitter).
• Passive ejection of instances that keep failing, on the caller's side, without waiting for the registry (Health checks & failover).
• Keep the last known list when the registry is unreachable, rather than failing every call.

Eureka takes the last point furthest. When renewals suddenly drop below about 85% of what it expects, it assumes a network problem rather than a mass die-off, enters self-preservation and stops evicting anyone. A stale list is a smaller problem than an empty one.

Instances leaving is the easiest case to get right, because it is planned. A graceful shutdown in the right order means no caller ever picks a dying instance: