Presence & push notifications

A turn-based word game has 3 million daily players. The friends list shows a green dot for friends who are online, so players can start a quick match, and when someone plays a word, the opponent should hear about it: instantly in the app if it is open, by notification if it isn't. That is two different problems with two different channels:

The live connection itself is a WebSocket or similar long-lived channel (WebSockets & long polling). This page is about what sits on top: deciding who counts as online, telling the right people when that changes, and handing everything else to push services that you don't control.

The obvious rule, "online while the socket is open", fails constantly. A phone that enters a tunnel or loses Wi-Fi doesn't close its socket; it simply stops talking, and the server may not notice for minutes. So presence works like any failure detector (Failure detection & split brain): the app sends a small heartbeat every so often, and the server marks it offline after a timeout without one:

The numbers are a trade. A short timeout notices departures quickly but flickers a friend's dot every time a train passes under a bridge; a long one shows ghosts who left minutes ago. A short grace period helps: an offline that is followed by a reconnect within a few seconds is never broadcast. The app also says goodbye explicitly when it can, such as when the user closes it, so the timeout only covers the silent cases.

Players use several devices, so presence is per connection, rolled up per person:

• Each connection (phone, tablet, browser tab) has its own entry.
• A person is online while any of their connections is.
• "Last seen" is written when the last connection goes, with the server's clock, not the phone's.

Firebase and Phoenix Presence both model it this way: a list of connections under one user.

Storing presence is cheap; broadcasting changes isn't. At the evening peak 600,000 players are online, and each changes state (opens, backgrounds or closes the app) about once every 10 minutes: 600,000 ÷ 600 s = 1,000 changes a second. Each player has about 50 friends:

Groups make it quadratic. The game runs weekend tournaments with a lobby that shows everyone who is online. With N players in a lobby, each changing once every 10 minutes, and every change sent to the other N − 1, the lobby generates N × (N − 1) ÷ 600 messages a second:

The fixes all reduce who is told, or how often:

• Counts instead of lists for large groups ("2,341 online"), with the full list fetched on demand.
• Batch and debounce: send one update a few seconds later covering every change in between.
• Subscribe by screen: a client gets presence only for the people it is currently showing.
• Typing indicators and other short-lived signals are never stored and may be dropped when late; they are worth something for a few seconds only.

600,000 open connections don't fit on one server, so they are spread over a fleet of gateway servers, say 50,000 sockets each (WebSockets & long polling). Ana's gateway must reach Ben's, wherever Ben connected. The usual shape:

• Presence store: a shared, in-memory store of connection entries that expire unless refreshed.
• Pub/sub between gateways: each gateway subscribes to the channels its connected users care about, such as a player's friends or a lobby, and publishes the changes it sees (Message queues & pub/sub).
• Clients reconcile on reconnect: after any gap, the app fetches current presence for what it shows, instead of trusting the updates it might have missed.

The dangerous moment is a gateway crash. Its 50,000 connections vanish at once, all their entries expire together, and the system announces 50,000 departures, then 50,000 arrivals a few seconds later when the apps reconnect to other gateways. Grace periods absorb most of that. So does accepting that presence is eventually consistent by design: it is allowed to be a few seconds wrong, never allowed to be expensive to keep right.

Some frameworks avoid a central store altogether. Phoenix Presence replicates presence between the servers of a cluster without a single source of truth, and sends clients only diffs of joins and leaves. Either way, presence is a cache of a guess, so keep it out of anything that must be exact: matchmaking still asks the game service whether a player accepted, not whether their dot is green.

When Ben's app isn't running, the game can't reach Ben's phone directly. Only the platform can: Apple Push Notification service (APNs) for iOS, Firebase Cloud Messaging (FCM) for Android, and a browser's push service for the web (the Web Push protocol). Each phone app registers with its platform and receives a device token, which it sends to the game's backend. The backend sends notifications to that token through the platform:

The platforms share the same knobs under different names:

Tokens need care too. A token stops working when the app is uninstalled or the platform rotates it, so store each token with the time it was last refreshed, delete it the moment the platform says it is invalid (FCM returns UNREGISTERED with HTTP 404), and treat tokens that haven't connected for a month as stale, as FCM itself does.

Push services make no promise that every notification arrives, and they say so in their rules:

• Offline phones keep only a little. APNs stores only the most recent notification for an app while the device is offline; FCM keeps the latest per collapse key, and on Android discards all stored non-collapsible messages if more than 100 pile up.
• Expiry. A notification whose lifetime ends before the phone is reachable is dropped.
• Power saving. Android's Doze suspends network access between maintenance windows; high-priority messages can wake the app, but high-priority messages that don't show the user anything may be deprioritized.

So a push is a hint, never the data. "Ana played QUIZ" is a notification; the move itself lives in the game service, and the app fetches the current game state whenever it opens, whatever notifications did or didn't arrive. Use the collapse ID per game, so ten moves while Ben is offline become one notification about game 77, not ten.