Recommendation & feed ranking

An outdoor app has 3 million hikers who post trail reports, photos and closure notices, about 400,000 posts a day. Ravi opens the app at a trailhead near Denver. The home feed should show posts from people Ravi follows, but also a closure notice for a nearby trail from someone Ravi has never heard of, and nothing from the account that posts 40 sunset photos a day.

A purely chronological feed of followed accounts (Fan-out on write vs read) misses the closure notice and lets the busiest posters crowd everyone else out. Ranking everything is the opposite extreme. With the last two weeks of posts eligible, and a full ranking model that costs, say, 0.2 ms of CPU per post, the arithmetic rules it out:

The design question is how to build that shortlist so it still contains the best posts, and then how to order it. The answer in every published large-scale design is a funnel of stages, each one looking at fewer items with a more expensive model.

Published designs share the same shape. YouTube's recommender narrows millions of videos to hundreds with candidate generation, then ranks those. Instagram's Explore runs retrieval, a lightweight first-stage ranker over thousands of candidates, a heavy second-stage ranker on the best 100, and a final re-ranking. Facebook's feed runs a light model to pick about 500 posts, a main scoring pass, and a final pass for rules such as content-type diversity. For Ravi's feed:

Each stage has a different job, and judging them by the same measure is a mistake. Retrieval is judged by recall: did the posts Ravi would have liked make it into the 1,500? A post it drops can never be shown, however good the later stages are. The rankers are judged by how well their order matches what Ravi actually does. The light ranker exists only to keep the heavy one affordable.

The candidate sources are deliberately different from each other. X's For You pipeline takes about half its posts from accounts the user follows and the rest from out-of-network sources such as a graph of who engaged with what. The usual sources: similar items, personal history, local versus distant items, popular or trending items (Count-Min Sketch & top-K), and the social graph.

The broadest source is usually an embedding model. A two-tower model learns two networks, one turning a user's features into a vector and one turning a post's features into a vector, trained so that a user's vector lands close to the vectors of posts that user engaged with. Instagram computes post vectors offline, loads them into an approximate nearest-neighbor index (Vector search & ANN indexes), and at request time runs the user tower on the freshest user features to search that index. The request path for Ravi:

Each source returns a score of its own, such as an embedding similarity, but those scores aren't used to rank. Scores from different sources aren't comparable, and a shortlist is small enough for one model with far richer features to score every candidate the same way. Where the query is known ahead of time, as in "posts similar to this one", the nearest neighbors can even be computed offline and stored.

The heavy ranker doesn't predict "how good is this post". It predicts several concrete actions, each as a probability: will Ravi like it, save it, comment, or hide it. X's ranker predicts ten such actions, from a like to a report, using about 6,000 features per post. The predictions are combined into one value score as a weighted sum, the same formula Instagram and Facebook use to combine their predictions.

The weights are product decisions, and negative actions get large negative weights. In X's April 2023 configuration a predicted reply weighed 13.5, a predicted like 0.5, and a predicted report −369. Suppose the trail app weighs a like 1, a save 4, a comment 6 and a hide −40. Three posts for Ravi:

The objective shapes the feed. Ranking by click probability promotes click-bait that people don't finish, which is why YouTube's ranker predicted expected watch time instead; but pure watch time favors very long videos. Mixing several predicted actions, including negative ones, is how feeds balance these. Facebook's weights are calibrated against surveys asking people which posts were worth their time, because engagement alone doesn't say that.

Sorting by value score alone gives a poor page. If the sunset photographer's 40 posts all score well, they fill the screen. Some rules depend on the whole list, not one post, so they run after scoring. X's pipeline applies author diversity, a balance of followed and recommended posts, fatigue from earlier negative feedback, removal of posts already seen, and filtering of blocked and muted accounts. Instagram's final stage avoids showing items from the same author in sequence.

Freshness also enters earlier. Users prefer fresh content, but not at the expense of relevance; YouTube feeds its model each training example's age so it can learn how popularity changes after upload, and retraining often and using a post's age as a feature help too. Finally, keep a fallback: X's For You timeline has a reverse-chronological pipeline to fall back on if the scored pipeline fails, so a ranking outage degrades the feed instead of emptying it.

A ranker is only as good as its features, and most are counts that change by the minute: how many saves a post has, how many trail reports Ravi opened today. Facebook notes its feed needs to know that an article got many likes even if it was posted minutes ago. Those counts come from streaming jobs over the event log (Stream processing), while slower features such as a user's long-term interests come from batch jobs; X's repository has one framework for aggregate features computed in either mode.

The classic failure is training-serving skew: the model performs worse in production than in training because the features differ. Code computing a feature one way in the training pipeline and another way at serving time causes it, and so does data changing between training and serving. The standard defenses:

The point-in-time row deserves an example. If a model learns whether Ravi saved a post, the save count it sees must be the count at the moment the post was shown, not today's count, which already includes Ravi's own save. A feature store such as Feast stores features as time series and reproduces their value at each training example's timestamp.

Every click the model learns from happened on a page the model arranged. Posts at the top get clicked more because they are at the top, and a model trained naively learns to rank them high again. That is position bias. Common fixes are to give the model the position as a feature during training and a fixed default when serving, keeping that feature separate from the rest; YouTube's ranker adds a small separate network that models the bias so the main model doesn't absorb it.

A second loop starves new posts. A model that ranks only by each post's own history can never learn that a new post deserves to be shown, because it never shows it. A new trail report needs exposure before there is any engagement to learn from, so reserve a few slots per page for new posts and new authors, and measure how they do.

Offline metrics guide development, but they aren't the verdict. Live A/B results are not always correlated with offline experiments, so every change ships as an experiment and is judged by live metrics, including negative ones such as hides and reports. The checklist for the trail app: