Fan-out on write vs read

A recipe-sharing app has a home feed. Mia follows 200 cooks, and when she opens the app she expects the 50 newest recipes from all of them, newest first, in well under a second. Every post exists once, in the posts table, stored under its author. The feed has to gather posts from 200 authors and merge them.

Copying one item out to many places is called fan-out. The question is when it happens:

• Fan-out on read (pull): when Mia opens her feed, fetch the recent posts of all 200 cooks she follows and merge them.
• Fan-out on write (push): when a cook posts, copy the post's ID into a ready-made feed for each of the cook's followers. Mia's feed is already built when she opens it.

Either way each post must reach each follower; the difference is copying it once when it's posted, or gathering it again on every read. Who pays, and how often, changes the total work. Feeds are opened far more often than recipes are posted. The app assumes these numbers, and the rest of the page uses them:

The last two averages are always equal. Every follow is one "following" for one user and one "follower" for another, so both averages are total follows ÷ users. The spread is what differs: nobody follows 8 million accounts, but some accounts have 8 million followers. That asymmetry decides the rest of the page.

With fan-out on read, posting is a single insert. Opening a feed does the gathering:

It has real strengths:

• Posting is cheap and instant: one row, visible to everyone the moment it is saved.
• Nothing extra is stored: no per-user feeds to keep, repair or delete.
• The feed is always current: an unfollow, a deleted post or a block takes effect on the next read.

The cost lands on the busiest path. At 200,000 feed opens a second with 200 accounts each, the posts store answers 200,000 × 200 = 40 million lookups a second. And each feed waits for the slowest of its many lookups, so its latency is set by the tail of the posts store, not its average (Latency). Large systems do run feeds this way: Facebook's news feed has been served by aggregators that query, at request time, many leaf servers holding recent activity, then rank and filter the results.

With fan-out on write, every user has a precomputed feed: a short list of post IDs, newest first, usually kept in an in-memory store. When a cook posts, the post is saved once, and a background job adds its ID to the feed of every follower:

Now compare the two with the app's numbers. Fan-out on write turns each read into one lookup and moves the copying to the much rarer writes:

More than 60 times less work, and a feed read that touches one list instead of 200 authors. The costs are new ones:

• Delay. Followers see a post only after the workers reach their feed, usually seconds later. The author's own feed is updated directly, so they never wonder where their post went.
• Storage. A list for every user, copied for availability (see the details section).
• Work for nobody. A post is copied into the feeds of followers who may never open the app again.

The app's most famous chef has 8 million followers. Each of her recipes is not 200 feed inserts but 8,000,000. If the whole fan-out fleet can insert 1 million IDs a second, one post occupies it for 8 seconds, while every other cook's posts wait behind it. Post three times in a minute and the fleet spends 24 of those 60 seconds on her.

Twitter hit exactly this. Its home timelines were built by fan-out on write, and for accounts with tens of millions of followers, delivering one tweet to every timeline could take up to 5 minutes. That produced a visible bug: followers saw replies to a tweet before the tweet itself, because the replies came from ordinary accounts and were delivered first:

Fan-out on read has the mirror problem, but milder: nobody follows millions of accounts, so a read never has to gather from millions of authors. What hurts fan-out on read is the total volume of reads, not any one user.

The standard answer combines the two. Accounts below a follower threshold are fanned out on write as before. Accounts above it are not fanned out at all; their posts are pulled and merged in when a feed is read. Twitter moved to this hybrid: celebrity tweets are fetched at read time and merged with the precomputed timeline.

Mia follows 200 cooks, and 2 of them are above the app's threshold of 100,000 followers. Opening her feed now takes three lookups instead of one, run at the same time:

The threshold is a dial between the two costs:

The pull side stays cheap because popular accounts are few and their recent posts are read constantly, so they sit in cache. The same account is pulled by millions of feeds, and each pull is one cached read.

Precomputed feeds are kept small and cheap with three habits, all used by Twitter:

• Store IDs, not posts. Each entry is a post ID and its author's ID. The recipe text, photos and like counts are fetched when the feed is shown, so an edit appears everywhere at once.
• Cap each list. Twitter kept 800 entries per home timeline. Nobody scrolls further in a feed; older posts can be pulled if someone ever does.
• Only active users. Twitter kept timelines in memory only for users who had logged in within 30 days. When an inactive user returns, their feed is rebuilt once by fan-out on read, and pushing to them resumes.

With these habits the memory is easy to estimate. Say 50 million active users, 800 entries each, 16 bytes per entry (two 8-byte IDs):

Copies of a post ID in 200 feeds also make removal harder. When a recipe is deleted, an account goes private, or Mia unfollows a cook, the pushed IDs are already in place. Chasing every copy is slow and easy to get wrong, so feed services usually filter at read time: when the feed is shown, posts that are deleted, hidden or from someone Mia no longer follows are dropped, and a background job cleans the lists later.

Feeds are the classic case, but the same choice appears wherever one thing reaches many people: notifications, group chats, activity streams, dashboards. Two numbers settle most of it:

And whichever you pick:

• Run fan-out on write in the background, through a queue, and answer the author as soon as the post is saved (Message queues & pub/sub).
• Measure delivery time, from post to the last follower's feed, and watch it for the biggest accounts.
• Keep lists of IDs, capped, and fetch the content when the feed is shown.
• Push only to active users, and rebuild a returning user's feed on read.
• Filter at read time for deletions, blocks and unfollows.
• Watch hot keys: a famous account's posts are read by millions of feeds and need caching and replication of their own (Hot partitions & skew).