Design a recommendation feed

Open a short-video app at 10:45 pm, just after an IPL final, and the first screen is not a list of people you follow. It is a “For you” feed: a clip you have never seen, picked from about a billion candidates in under a fifth of a second, with the next one already loading. Skip two cricket clips in a row and the app should notice before you reach the end of the page. Behind that swipe sit three systems interviewers love to probe: retrieval over a billion vectors in tens of milliseconds, ranking that spends GPU time only where it pays, and a data loop that turns 60 billion events a day into features and models. You will design it for 300 million daily users and 5 million uploads a day — as a systems problem, with just enough machine learning to justify each box.

Understand the problem

A recommendation feed is a search engine whose query is the user’s history. Weak answers draw one box labelled “ML model”. Strong answers split the work in two — a fast, approximate, degradable request path that picks 10 items, and a large, mostly asynchronous learning loop that turns what happened into features, vectors and models — and say what joins them: the feature store and the impression log.

Functional requirements

1. Users should be able to open the app and scroll a personalised feed endlessly. Each page holds about 10 items, nothing repeats within 30 days, and the first page appears as soon as the app opens.
2. Users should see the feed react to what they just did. A like, a long watch, a run of skips or a “not interested” should change the next page, not tomorrow’s.
3. New users and new items should get a fair start. A first-time user gets a sensible feed from their language, region and a few picked interests; a clip uploaded five minutes ago can be shown before anyone has engaged with it.
4. The feed should follow safety and business rules. Removed or unsafe items and blocked or muted creators never appear, and no single creator or topic floods the screen.
5. ML engineers should be able to train on what was shown and test new models on a slice of traffic, without changing serving code.

Non-functional requirements

• Latency. Server-side p50 under 100 ms, p99 under 200 ms per page. The app prefetches, so only the first page after opening is ever waited for.
• Scale. 300M daily active users, about 100k feed requests/s at peak, ~1M impressions/s and ~2M events/s. A corpus of 1B items, with ~5M new ones a day.
• Freshness. A user’s actions reach the features the ranker reads within about 5 s. New items are recommendable within 15 minutes of passing moderation. Models retrain daily and fine-tune hourly.
• Availability. 99.95% for the feed endpoint, and never an empty screen: every stage has a fallback, down to a precomputed popular-in-your-language feed.
• Correctness where it matters. Zero tolerance for removed content and blocked creators; repeats on under 1% of impressions.
• Bounded cost. GPU spend per request is set by design — candidates × model cost.
• Logging. Every shown item is logged with its features, model version and experiment arm, kept about 90 days for training.

Out of scope

• Model research. We use standard model shapes (two-tower retrieval, multi-task ranking) and focus on serving and data; tuning architectures is an ML-team problem.
• Upload, transcoding and video delivery. The path for the video bytes is covered in design TikTok and design YouTube.
• The Following tab. A chronological feed of followed accounts is a fan-out problem; see design a news feed.
• Ads. The feed leaves slots; a separate ads system with its own auction fills them and counts clicks, as in ad click aggregation.
• Moderation models and search. We enforce moderation verdicts but do not build the classifiers; query-driven search is a different problem.

The set-up

Plan the approach

Name the shape first: “a funnel on the request path — retrieve, rank, re-rank — and a data loop around it — log, join, train, index.” Then meet each functional requirement with the simplest thing that works, and spend most of your time in the deep dives. Keep the machine learning to what justifies a box: why two towers, why a cheap ranker before an expensive one, why features must be logged.

Core entities

• User. ID, region, languages, onboarding interests, blocked and muted creators. Long-term features (favourite topics and creators, a daily embedding) and short-term ones (the last 100 interactions, skips per topic in the last 10 minutes).
• Item. ID, creator, duration, language, topics, upload time, moderation status, a content embedding, and counters such as views, completion rate and likes in the last hour.
• Embedding. A 128-number vector for a user or an item, tagged with the model version that produced it. Vectors from different versions must never be compared.
• Candidate. An item ID plus its source, seed item and source score — provenance the ranker uses as features.
• Impression. One item shown to one user at one position in one request, with the model version, experiment arm and a snapshot of the features the ranker used.
• Engagement event. What happened to an impression: started, watched 12.4 of 15 s, liked, shared, skipped, “not interested”.
• Feed session. The slate behind a cursor: items served and scored candidates not yet shown.

The impression record is the most important row in the system, because training data is built from it:

impression_id: imp_7c1e:3          # request id : position
user_id: u_42
item_id: v_81a
served_at: 2026-10-27T21:04:05.120Z
position: 3
sources: [two_tower, related:v_77f]
model: heavy-ranker-v42
experiment: { feed_ranker: arm_b }
scores: { p_finish: 0.61, p_like: 0.08, p_skip: 0.22, value: 1.37 }
features: serve-log/2026-10-27/21/imp_7c1e   # snapshot the ranker actually used

API

The app talks to two public endpoints. Retrieval, ranking and the feature store are internal gRPC services.

Get a page. The cursor is opaque: it names a server-side session slate and a page number, not an offset.

GET /v1/feed?surface=for_you&limit=10&cursor=Y3M6czlmMjpwMg
Authorization: Bearer <token>

{
  "items": [
    {
      "item_id": "v_81a",
      "impression_id": "imp_7c1e:3",
      "creator": { "id": "c_19", "handle": "@chef.meera" },
      "media": { "hls": "https://cdn.example.com/v_81a/master.m3u8", "duration_ms": 15000 },
      "reason": "related_to:v_77f"
    }
  ],
  "next_cursor": "Y3M6czlmMjpwMw",
  "prefetch_after": 8
}

prefetch_after: 8 tells the app to request the next page once the user reaches the eighth item, so the response is already there when they swipe past the tenth.

Report what happened. The app batches events every few seconds, and when it goes to the background. Every event carries the impression ID, which is what lets you join an outcome to the exact features that produced it.

POST /v1/events
Content-Type: application/json

{
  "events": [
    { "type": "view_end", "impression_id": "imp_7c1e:3", "watched_ms": 12400, "ts": "2026-10-27T21:04:19.900Z" },
    { "type": "like", "impression_id": "imp_7c1e:3", "ts": "2026-10-27T21:04:18.200Z" },
    { "type": "not_interested", "impression_id": "imp_7c1e:4", "reason": "topic", "ts": "2026-10-27T21:04:31.050Z" }
  ]
}

The response is 202 Accepted: events are queued, not processed inline. Onboarding is one more small call, PUT /v1/users/me/interests with a body such as {"topics": ["cricket", "cooking"], "languages": ["hi", "en"]}.

Back-of-envelope estimates

Assume 300M daily active users who each watch about 100 items a day — 25 minutes at roughly 15 seconds an item, skips included — with 10 items per page and an evening peak of 3× the daily average. (For the method, see back-of-envelope estimation.)

Feed requests. 100 items ÷ 10 per page = 10 pages per user per day. 300M × 10 = 3B requests/day; 3B ÷ 86,400 s ≈ 35k/s on average, and 35k × 3 ≈ 100k requests/s at peak.

Impressions and events. 300M × 100 = 30B impressions/day, about 350k/s on average and ~1M/s at peak. Each impression yields about two events (start, and end with watch time; likes and shares add a few per cent): 60B events × 150 bytes ≈ 9 TB/day, and 2M events/s × 150 B ≈ 300 MB/s into Kafka at peak.

Serve-time feature logs. About 1 KB per shown item: 30B × 1 KB = 30 TB/day. Logging all 200 scored items per request would be 3B × 200 × 1 KB = 600 TB/day, so log shown items plus a 1% sample of the rest.

Retrieval. Six sources return about 1,500 candidates per request. Exact search of one 128-dimension user vector against 1B item vectors is 1B × 128 ≈ 1.3 × 10¹¹ multiply-adds ≈ 256 GFLOP per query; at 100k queries/s that is 2.6 × 10¹⁶ FLOP/s, or about 26,000 servers at a generous 1 TFLOP/s each. Retrieval has to be approximate.

Index memory. 1B × 128 dimensions × 4 bytes = 512 GB as raw float32 vectors. As 64-byte product-quantised codes plus an 8-byte ID, 1B × 72 B = 72 GB, which fits in one server’s RAM. If one replica answers 5k queries/s, 100k ÷ 5k = 20 replicas, about 30 with headroom.

Ranking compute. The pre-ranker scores 100k × 1,500 = 150M candidates/s, each a 128-dimension dot product (256 FLOP): about 38 GFLOP/s, which is nothing. The heavy ranker scores 100k × 200 = 20M candidates/s. Budget 50 MFLOP per candidate: 20M × 50 MFLOP = 10¹⁵ FLOP/s = 1 PFLOP/s. Ranking models are bound by memory and embedding lookups, so assume a GPU server delivers only ~5 TFLOP/s of useful work, or 100k candidates/s: 20M ÷ 100k = 200 servers, about 300 with headroom. Send all 1,500 candidates to the heavy model and you need 1,500 servers before headroom. Sanity check: 20M candidates/s × 6 predicted actions = 120M predictions/s, the same order of magnitude as the 90M model predictions per second Instagram reported for Explore in 2019.

Feature store. A user row — profile, last 100 interactions, counters, embedding — is about 4 KB and is read once per request: 100k × 4 KB = 400 MB/s. Item features (about 2 KB) for 200 candidates mean 20M item lookups/s; popularity is very skewed, so ranker hosts cache hot items locally, and at a 90% hit rate 2M lookups/s reach the online store as batched multi-gets. Storage: 500M monthly users × 4 KB + 1B items × 2 KB = 4 TB, 12 TB with three replicas. Streaming adds about one user write per finished impression: ~1M writes/s at peak.

Seen-items filter. 30 days × 100 items = 3,000 items per user. An exact list of 8-byte IDs is 24 KB per user, 7.2 TB for 300M users. Bloom filters (deep dive below) need about 5.7 KB per user, ~1.7 TB.

Latency. Split the 200 ms p99: 10 ms to load context, 40 ms for candidate sources in parallel, 10 ms to filter and pre-rank, 20 ms to fetch features, 50 ms for the heavy model and 15 ms to re-rank and respond — 145 ms of work, leaving 55 ms for the tail.

High-level design

Build the design one requirement at a time; by the end it works, but not yet at full scale.

Users get a personalised feed they can scroll

Start with the two-stage split from YouTube’s 2016 paper: a candidate generator narrows millions of videos to hundreds, and a ranker orders those hundreds using far richer features. Retrieval must consider everything, so each item can cost only nanoseconds; ranking sees a few hundred items, so each can afford a deep model.

For retrieval, use a two-tower model. A user tower turns the user’s features — recent watches, languages, region — into a 128-number vector; an item tower turns each item’s features into a vector in the same space, trained so that a high dot product means “likely to engage”. Because the towers meet only at the dot product, all billion item vectors can be computed offline and loaded into an approximate-nearest-neighbour (ANN) index, while the user vector is computed per request from fresh features. That cacheability is why Instagram uses two towers for retrieval. The price: the towers cannot use user–item interaction features such as “how often has this user finished this creator’s clips?”, which are usually the strongest signals. That is the ranker’s job.

The ranker is a multi-task network that predicts several outcomes for each candidate — P(finish), P(like), P(share), P(skip), P(“not interested”) — and a small value model folds them into one score, for example 2·P(finish) + 4·P(share) + P(like) − 8·P(not interested). Instagram describes this weighted sum, and X’s published weights are heavily negative for negative feedback and reports. Rank on watch outcomes, not clicks: YouTube found click-through rate promotes clickbait that users abandon.

1. The app calls GET /v1/feed.
2. The feed service reads the user’s features and runs the user tower to get a vector.
3. Retrieval asks the ANN index for the 1,500 nearest item vectors.
4. The ranker fetches features for those items, scores them and sorts them.
5. The feed service returns the top 10 with a cursor; the next page excludes what was served.

This works as a demo. It has one source, so everything looks alike; item vectors refresh daily, so nothing new appears; and the heavy model scores 1,500 items per request, 7.5× the GPUs it needs. The next steps fix the first two; the deep dives fix the third.

The feed reacts to what the user just did

Three skipped cricket clips in a row say something now; nightly features would keep serving cricket until tomorrow. Events must flow back in seconds.

The app posts batched events to a collector, which appends them to Kafka, partitioned by user ID so each user’s events stay in order. A Flink job keeps per-user state — the last 100 interactions with watch fractions, and counters such as skips per topic in the last 10 minutes — and upserts it into the online feature store within about two seconds. On the next request the user tower reads those fresh features, so the user vector itself moves; the ranker sees the counters directly.

Add a second source: related items, the nearest neighbours of the last few items the user finished, liked or shared. Instagram describes this source: it tracks current interests more closely than one user vector, and it filters low-quality seeds out first, so one accidental tap does not flood the feed.

1. The user skips; the app sends view_end with 1.2 s watched, and the collector appends it to user-events.
2. Flink updates the user’s recent actions and topic counters in the online store.
3. On the next prefetch, the feed service reads the updated row: cricket is demoted, and related items seed from what the user did finish.

One trade-off hides in the prefetch: the app asks for page n + 1 with about two items of page n left — some 30 seconds of runway — so actions on those two items cannot influence it. Prefetch later and slow responses become spinners; earlier, and the feed reacts a page late.

New users and new items get a fair start

The two cold starts look alike but need different fixes.

A new user has no history, so the user tower has only weak inputs: language, region, device, and whatever they chose in a few seconds of onboarding (“cricket, cooking, stand-up”). YouTube notes that such features give new users sensible priors. Add a starter source: what is trending in their language and region — the output of a top-K trending system — plus popular items in the topics they picked. After a dozen swipes, the streaming features from Step 2 take over.

A new item has no engagement, and a model trained on engagement will not pick it — so it is never shown, so it never gets engagement. Break the loop with content. The item tower takes content features (visual and audio embeddings, caption text, language, creator) as well as an ID, so a clip has a usable vector the moment moderation clears it, as in YouTube’s content-aware two-tower retrieval (Yi et al.). A stream indexer inserts the vector into a small fresh index within about a minute, and an exploration source reserves a few slots per page for new items. A deep dive turns that into a budget.

1. Onboarding writes topics and languages to the user’s features; the starter source returns trending-in-region and top-in-topic items.
2. A creator uploads, and moderation clears the clip within minutes.
3. The item tower embeds it from content alone, the stream indexer adds it to the fresh index, and the exploration source can nominate it.

The feed follows safety and business rules

Some rules are absolute and some are taste; treat them differently.

Hard filters remove candidates outright: removed or under-review items, items unsuitable for this user (age, region), blocked or muted creators, and items the user has already seen. Run them right after retrieval, before any GPU time is spent, and re-check removals just before responding, because moderation can act mid-request. Keep the seen check cheap with a per-user Bloom filter, sized in a deep dive below.

Soft rules reorder rather than remove. A ranker scores each item on its own, so its top 10 is often five clips from one cricket highlights channel. A re-ranker walks down the scored list and fills the page slot by slot, taking the best item that satisfies the constraints — here, the same creator at least three slots apart and at most two items of one topic in any four consecutive slots. Instagram describes penalising consecutive posts from the same author, and X’s home mixer lists author diversity, a balance of followed and recommended posts, and removal of previously seen posts among its final filters.

The rules cost a little engagement per page and usually win it back across a session: a varied page lets the user reveal a new interest.

Engineers learn from what was shown and test safely

Training data is only what you logged, so log two streams, both keyed by an impression ID (request ID plus position):

• The serve log, written by the ranker: for each shown item, the features it used, its predicted scores, the model version and the experiment arm.
• The event log, from the app: what happened to each impression — watch time, like, share, skip.

A joiner, another Flink job, pairs them into training rows: features as they were at serve time, with labels as they arrived. ByteDance’s Monolith paper describes this design: Kafka queues of actions and features, a Flink joiner keyed by request, and an on-disk store for features whose labels arrive late. Trainers retrain daily and fine-tune hourly, and a model registry versions each model and pushes it to the rankers.

Experiments sit on top. Hash each user ID into one of 1,000 buckets per experiment layer; an experiment maps bucket ranges to arms, and each arm names a model version and value-model weights. Engineers ship a model by creating an arm, not by changing code. As the YouTube paper stresses, offline metrics guide development but live A/B results decide, and the two do not always agree.

1. The ranker serves a page with the arm’s model and writes the serve log; the app’s events land in the event log.
2. The joiner matches them by impression ID within one hour; later labels join in a daily batch.
3. Trainers produce a candidate model; after offline checks it becomes a canary arm, and only a winning A/B test promotes it.

You now have a working feed: personal, reactive, fair to newcomers, safe, and able to learn. The deep dives are about what breaks at a billion items and 100k requests a second.

Deep dives

How do we find 1,500 good candidates among a billion items?

The estimates ruled out exact search — 256 GFLOP per query, about 26,000 servers at peak — so you trade a little accuracy for a lot of speed with an approximate nearest-neighbour index (embeddings and semantic search and vector databases cover the basics).

Bad: exact search, sharded. Split the 512 GB of vectors across eight machines and scan them all. Exact, but every query still costs 256 GFLOP and touches every shard, so the slowest machine sets your latency.

Good: an HNSW graph index. HNSW links each vector to its near neighbours in a layered graph and walks greedily towards the query, visiting a tiny fraction of the corpus with high recall. The Faiss guidelines give its memory as d × 4 + M × 2 × 4 bytes per vector; with 128 dimensions and M = 32 links that is 512 + 256 = 768 bytes, so 768 GB for a billion items — more than the raw vectors, because the links are extra. You must shard it (say eight shards of about 96 GB) and fan every query out. Faiss’s HNSW also cannot remove vectors, so takedowns become tombstones until the next rebuild.

Great: a compressed inverted-file index, replicated. An IVF index clusters the vectors — Faiss suggests about a million clusters for 100M to 1B vectors — and each query scans only the few clusters nearest to it. Product quantisation (PQ) stores each vector as a short code, 64 bytes instead of 512, and computes approximate dot products from small lookup tables. At 64 bytes plus an 8-byte ID, a billion items take 72 GB, so the whole index fits in one server. Replicate it about 30 times instead of sharding it: any replica answers any query, there is no fan-out, and capacity grows by adding replicas. You pay in recall, because quantisation blurs close calls, so ask for 2–3× the neighbours you need and re-score them with exact vectors. That matters less than you might fear: retrieval needs recall, not perfect order, and YouTube reported A/B results that were not very sensitive to the choice of nearest-neighbour algorithm.

Keeping the index fresh. A product-quantised index is built in batch — train codebooks, encode a billion vectors, ship 72 GB to 30 replicas — which takes hours; YouTube’s two-tower system rebuilt its quantised index of about 10M videos every few hours. New uploads need minutes, so run two indexes side by side.

• Main index. Rebuilt every 4 hours from the current item tower, versioned, and swapped atomically on each replica: load the new one beside the old, then flip a pointer.
• Fresh index. Uploads since the last build — 5M × 4/24 ≈ 830k items, about 0.6 GB as HNSW — added by a stream indexer within a minute. HNSW takes incremental inserts well, and at this size its overhead is irrelevant.
• Removed set. Takedowns cannot wait four hours for a rebuild, so the merge step drops any ID in a removed-items set before returning.

The detail interviewers probe is versioning. The user tower and the item vectors must come from the same model. A new two-tower model moves every vector, so a v9 user vector searched against v8 item vectors returns noise. Build the v9 index, load it beside v8, and switch tower and index together — per experiment arm, if you are A/B testing the new retrieval model.

How do we spend ranking compute where it matters?

The heavy ranker is the most expensive thing on the request path, and its cost is candidates × model cost. The funnel is how you control the first factor.

Bad: run the heavy model on everything retrieval returns. Simple, and as accurate as your model can be — but 100k × 1,500 = 150M heavy scores a second, about 1,500 GPU servers before headroom, and heavy-model latency that grows with the candidate count.

Good: truncate each source, then heavy-rank 200. Cheap, but each source scores in its own currency — an ANN distance, a graph visit count, a trending velocity — and the YouTube paper notes that combining such incomparable sources is one of ranking’s jobs. Truncating per source discards good candidates before anything compares them fairly.

Great: four stages, so the expensive model sees only the best 200.

1. Retrieval from six sources in parallel, each with a quota: two-tower ANN (500), related items (400), graph walk (300), follows (100), trending in region (100) and the new-item exploration pool (100). Pinterest’s Pixie is the reference for the graph walk: random walks over a pruned graph of 3B nodes and 17B edges held in about 120 GB of RAM on each machine, with one server answering about 1,200 requests a second at a p99 under 60 ms. At 100k requests/s that is 100,000 ÷ 1,200 ≈ 83 servers before headroom.
2. Pre-ranking of all ~1,500 with a light model. Instagram trains its first-stage ranker to predict the heavy model’s top picks — distillation — as a two-tower model whose item embeddings can be cached. Scoring becomes a dot product: 1,500 × 256 FLOP ≈ 0.4 MFLOP per request. Keep the top 200.
3. Heavy ranking of those 200, with a few thousand features: the user’s history with this creator, the item’s recent completion rate, which source nominated it and with what score. X’s home mixer hydrates about 6,000 features per candidate. Split the 200 into four batches of 50 on parallel GPU streams; Google’s Wide & Deep paper cut ranking latency from 31 ms to 14 ms by splitting one batch into smaller parallel ones.
4. Re-ranking for rules and diversity, as in Step 4.

Instagram’s Explore runs the same idea with finer steps: 500 candidates, then 150, then 50, then 25 for the first grid.

Give each stage a deadline and a fallback, so a slow dependency costs quality, not availability:

• A source that misses its 50 ms deadline is dropped for this request.
• If the heavy ranker is overloaded, serve the pre-ranker’s order to a share of requests — load shedding by quality instead of by refusal.
• If ranking fails outright, serve the user’s cached slate or a precomputed popular-in-your-language feed. X’s home mixer keeps a backup pipeline for when scoring fails, and Instagram precomputes some users’ recommendations off-peak.

Never retry inside the funnel. A retried 50 ms call blows the budget of every stage after it; drop the source and move on.

How do we serve features fast without training–serving skew?

Features are the ranker’s raw material, and there are two ways to get them wrong: slowly and inconsistently.

Bad: compute features on the request path from source databases, and recompute them separately for training. “Completion rate of this creator’s last 20 clips” becomes 20 queries on a busy database, times 200 candidates. Worse, training computes the same feature with different code — a Spark job over the warehouse — and the two drift: different windows, deleted likes handled differently, a time-zone bug. The model learns one definition and is served another: training–serving skew, which quietly erases model gains.

Good: a feature store with one definition and two stores. Define each feature once and materialise it to an offline store (the warehouse, full history, for training) and an online store (a key-value store of latest values, for millisecond serving). Batch features are computed daily; streaming ones come from Flink. For training, use point-in-time joins: for each labelled impression, take each feature’s value as of the impression’s timestamp, never the latest one. The Feast documentation describes exactly this — scanning back from each row’s timestamp up to a TTL — and adds a filter on when each value was created, so that backfilled corrections cannot leak into earlier rows.

Great: log features at serve time, and join labels to them. The ranker already has every feature value in hand when it scores, so write them to the serve log with the impression ID (Step 5). Training rows then contain exactly what the model saw: no recomputation, no skew, no leakage. Point-in-time joins are needed only to backfill a new feature that was never logged. The cost is volume — 30 TB a day — so log shown items plus a small sample of unshown ones, which the pre-ranker’s distillation needs. Add skew monitoring: daily, compare each logged feature’s distribution with an offline recomputation and alert on divergence.

The online store itself holds about 4 TB, takes about 2M reads/s after local caching and ~1M writes/s at peak: a sharded key-value store such as Cassandra or DynamoDB on SSD, with Redis for the small, hottest real-time slice. Store one compact row per entity, not one key per feature, so a user is one read. Give real-time features a TTL, so a counter whose updates have stopped expires instead of lying.

How does the feed react within seconds instead of tomorrow?

There are three layers of freshness, each costing more than the last.

Bad: nightly batch. Features and user embeddings are recomputed once a day: cheap, simple, and a day behind the user.

Good: streaming features into a model trained daily. Step 2’s Flink job updates the last 100 interactions and short-window counters within seconds. The user tower computes the user vector on each request from those features, and the heavy ranker reads them directly. The model’s weights are a day old, but its inputs are seconds old. The YouTube paper calls up-to-the-second impression and watch history vital for responsive recommendations, and credits “shown but not watched” counts with making successive pages differ.

The Flink job is ordinary stateful stream processing: key by user ID, keep a ring buffer of 100 events and a few windowed counters in state, emit an upsert per event, and checkpoint state so a restart loses nothing. Two details matter. Deduplicate events by impression ID and type, because the app retries uploads. And watch consumer lag: if Flink falls ten minutes behind during an IPL-final spike, “real-time” features silently go stale, so alert on lag, not only on errors.

Great: also update the model continuously. Item IDs and their learned embeddings are the fastest-moving part of a ranking model — a clip can go viral in an hour. Instagram fine-tunes its ranking models every hour on new data. Monolith goes further: it trains online from the Kafka stream and, every minute or so, pushes only the sparse embedding rows touched since the last sync to the serving parameter servers, syncing the slow-moving dense layers less often. Its production models are several terabytes, so shipping the whole model each time is impossible. You pay with a far more complex training system, a model a burst of bad data can steer (so it needs anomaly checks and fast rollback), and weaker durability: Monolith snapshots daily and accepts losing up to a day of updates if a parameter server fails.

For most teams, Good plus hourly fine-tuning captures most of the benefit; minute-level updates pay off when items are short-lived and plentiful, as in short video or ads.

How do new items and new users get exposure without wrecking the feed?

Step 3 gave new items a vector and a fresh index. That makes them findable, but a ranker trained on engagement still scores them low, because their engagement features are empty. Exposure needs a budget.

Bad: wait for organic engagement. New items reach only followers. Most creators have few, so most uploads get no views, the model never learns about them, and the feed fills with the same proven creators — a rich-get-richer loop.

Good: content-based vectors plus a freshness signal. The content-aware item tower places a new clip near similar ones, and the ranker sees item age. YouTube found that a model trained on weeks of history is biased towards the past; it feeds the age of each training example as a feature and sets it to zero at serving time, which lets the model learn how fresh uploads are actually consumed. Good new items now get some exposure — but how much is left to chance.

Great: an explicit exploration budget with graduation. Reserve 5% of impressions for new items: 30B × 5% = 1.5B impressions a day. With 5M uploads a day, that is an average of 300 impressions per item. Spend it in stages:

1. Every moderated upload enters a test pool and gets about 200 impressions, shown to users whose interests match its content vector: 5M × 200 = 1.0B impressions.
2. The top 10%, judged by completion and like rates relative to similar items, graduate to a wider pool of 1,000 more impressions: 500k × 1,000 = 0.5B.
3. Winners now have real engagement, so normal retrieval and ranking take over.

Pick which item gets the next exploration slot with a bandit rather than a fixed queue: Thompson sampling or an upper-confidence bound favours items whose early numbers are promising or still uncertain. You pay for it: 5% of impressions are, on average, worse than the ranker’s best pick, and the pools invite gaming, so cap exploration per creator and count only engagement from accounts that pass integrity checks. In return you get training data the current model did not choose.

New users need the same idea. Global popularity gives a student in Indore and a retiree in Chennai the same feed; onboarding plus regional trending is better; best is to treat the first session as exploration — a deliberately varied first page, then streaming features after every swipe — so by page three the feed reflects behaviour, not the onboarding form.

How do we serve infinite scroll without repeats or recomputing everything?

There are three ways to produce page n.

Bad: precompute every user’s feed offline. A nightly job stores 200 ranked items per user; serving is a lookup. Fast, but stale — nothing from today, no reaction to this session — and wasted on users who never open the app. It survives as the fallback feed.

Good: run the full funnel for every page. Freshest and simplest, but it pays the full 20M heavy scores a second, and consecutive runs can repeat items: page 1’s items may not be in the seen filter yet when page 2 is ranked, because their events are still in flight.

Great: a slate cache per session, refreshed on signal. Run the funnel, show 10, and keep the other scored candidates — IDs and scores, about 3 KB — in a session store under the cursor. For the next page, re-rank that slate with the pre-ranker and the newest session features, in about 15 ms, instead of re-running the funnel. Re-run the full funnel every second page, or at once on a strong signal: a like, a share, a “not interested”, three skips in a row. If 70% of second pages can use the slate, full runs drop to (1 + 0.3) ÷ 2 = 0.65 per page, cutting heavy-ranker load by about 35%, from 20M to 13M scores a second. The cost is a session store — 1M impressions/s × 15 s per item ≈ 15M people scrolling at peak, so with a 15-minute TTL plan for ~30M slates × 3 KB ≈ 90 GB of Redis — and slightly older scores on cached pages.

Pagination. Offset pagination (“page 3 is items 20–29”) assumes a stable list, and this list is rebuilt between pages. Use an opaque cursor that names the session slate and the page; see offset versus cursor pagination. If the cursor has expired, treat the request as a fresh page 1; the user will not notice.

No repeats. Use two structures, because events lag:

• The session keeps the exact list of items served in this session. It is checked first, and it catches the in-flight case above.
• A per-user Bloom filter holds items actually seen (from view_start events) over 30 days. It never says “not seen” for a seen item, and its rare false “seen” just skips a candidate. Bloom filters cannot delete, so keep five weekly filters and drop the oldest each week. Five filters at 0.2% false positives each give 1 − 0.998⁵ ≈ 1% overall. Sized for 700 items a week, each needs −ln(0.002) ÷ (ln 2)² ≈ 12.9 bits per item, about 1.1 KB per filter: 5.7 KB per user and 1.7 TB for 300M users, against 7.2 TB for exact lists. Size filters by activity tier: someone who watches 500 items a day overfills a 700-item filter, and its false-positive rate climbs.

Items that were served but never seen — page 2 arrived, but the user closed the app — return to the pool, because only view_start events reach the Bloom filter.

The final design

Put the pieces together. The request path is the feed service, its session store and the funnel; everything below the funnel is the asynchronous learning loop.

Follow one page through the system:

1. Request. The app asks for the next page with its cursor, two items before it needs it.
2. Context. In parallel, the feed service loads the session slate, the seen filter, the user’s online features and the experiment arm, which names the model versions. If the slate is valid and no strong signal has arrived, it re-ranks the slate and jumps to step 5.
3. Retrieve. Six sources run in parallel under a 50 ms deadline; the two-tower source searches the main and fresh ANN indexes with the user vector. Hard filters drop removed, blocked and seen items, leaving about 1,500 candidates.
4. Rank. The pre-ranker keeps 200; with features from the online store and local caches, the heavy model scores them on GPUs, and the value model combines its predictions.
5. Re-rank and respond. Rules build a page of 10; the rest of the slate goes to the session store, and the serve log records features, scores, model and arm for every shown item.
6. Events. The app batches watch, skip, like and “not interested” events to the collector, which writes them to Kafka keyed by user ID.
7. Stream. Flink updates real-time features within seconds, adds seen items to Bloom filters, indexes new uploads, and joins events to the serve log into training rows.
8. Learn. Daily training with hourly fine-tuning ships models to rankers as canary arms; the item tower rebuilds the main ANN index every 4 hours, switched with its matching user tower.

Check it against the requirements. A full run budgets 145 ms of work under the 200 ms p99, a cached page about 15 ms, and prefetching hides both. Peak load needs about 30 ANN replicas and 300 GPU servers, fewer with the slate cache. Actions reach features in seconds; new items are findable in minutes. Per-stage deadlines and a fallback chain ending in a precomputed feed mean the screen is never empty. Hard filters, applied twice, enforce zero tolerance for removed content and blocked creators; the served list plus Bloom filters keep repeats near 1%. Serve-time logging keeps training free of skew, and every model change is a reversible A/B test.

What interviewers look for by level

Follow-up questions

What if the ranking cluster is overloaded right after an IPL final?

Traffic jumps within minutes of the last ball, and GPUs do not appear in seconds. Degrade quality before availability, in a pre-agreed order: first reuse slate caches for two pages instead of one; then halve the heavy ranker’s input from 200 to 100 candidates, which halves its load; then send a slice of users to pre-ranker-only ranking; and only then fall back to cached popular feeds. Each step is a flag the load shedder flips from queue depth and p99. Pre-warm capacity too: the match end time is known. See backpressure and load shedding and autoscaling done right.

What if a new ranking model quietly drops watch time by 2%?

It should never reach everyone. A new model is first replayed offline against logged impressions, then shadow-scored on live traffic (scored and logged, but not served), then given a 1% canary arm with guardrail metrics — watch time, “not interested” rate, creator diversity, latency — and an automatic stop if any guardrail crosses its threshold. Every impression carries its model version, so the drop is attributable within hours, and rollback means pointing the arm at the previous version. Run tests for at least a week; novelty inflates early numbers. It is a canary deployment, applied to models.

What if the feed recommends products during a Big Billion Days–style sale?

The funnel stays, but item state changes by the minute: price, stock, deal windows. Retrieval and ranking tolerate slightly stale vectors and features; hard filters cannot, so check stock and price against a fast inventory cache in the final filter rather than an hourly feature snapshot. The objective shifts to purchases and margin, and labels arrive late — a purchase may follow a view by days, the case Monolith’s on-disk joiner store exists for. Sale and sponsored boosts become capped re-ranker rules, so the feed does not turn into an advert. Expect a sharp spike when the sale opens, and precompute slates the night before for likely visitors.

What if creators game the system with fake engagement?

A ranker that rewards likes and completions invites bots that produce them. Defend at three layers. At ingestion, rate-limit and validate events: an impression ID must exist in the serve log and a watch cannot outlast the clip, so forged events never join. In features, count engagement only from accounts with good integrity scores, and prefer rates per impression served over raw counts. In training, drop rows from flagged accounts and weight negative feedback heavily. Then flag creators whose engagement comes from new or tightly linked accounts — a graph problem like people you may know, in reverse.

How to say it in the interview

“I’ll design a personalised For You feed for about 300 million daily users and a billion items, with a server-side p99 under 200 milliseconds and a feed that reacts to what you just did.

The shape is a funnel on the request path and a learning loop around it. For each page, a feed service fans out to six candidate sources in parallel: a two-tower model whose user vector searches an approximate nearest-neighbour index, items related to what you just liked, a graph walk, follows, regional trending, and a pool of new uploads. That gives about 1,500 candidates. A distilled two-tower pre-ranker cuts them to 200 with dot products, a multi-task model on GPUs predicts finish, like, share and skip for each, and a value model combines those. A re-ranker removes blocked and seen items and spaces out creators and topics before returning 10.

The number that shapes the design is candidates times model cost: 200 candidates at 100 thousand requests a second is 20 million heavy scores a second, about 300 GPU servers. Hence the funnel, and a per-session slate cache so I re-run it only every other page or on a strong signal.

For retrieval, a billion 128-dimension vectors are half a terabyte raw but 72 gigabytes with product quantisation, so I replicate the index instead of sharding it, rebuild it every four hours, and add a small fresh index for new uploads. The user tower and the index ship as one versioned pair.

Freshness comes from Kafka and Flink: events become features in about two seconds. Every shown item is logged with the exact features the ranker used and joined to its outcome by impression ID, so training has no skew. New items get a 5 percent exploration budget with graduation, every model change ships as an A/B arm with guardrails, and a fallback chain means the screen is never empty.”

Concepts used in this design

• Embeddings and semantic search — what the two towers produce.
• Vector databases — ANN indexes, quantisation and the recall–memory trade-off.
• Kafka — the event and serve logs, partitioned by user ID.
• Flink and stream processing — real-time features, the impression joiner and the stream indexer.
• Batch vs stream processing — daily training beside second-level features.
• Bloom filters — the seen-items filter and its false-positive arithmetic.
• Pagination: offset vs cursor — why a feed rebuilt between pages needs a cursor.
• Redis — session slates and the hottest real-time features.
• Cassandra and DynamoDB — the online feature store at terabyte scale.
• OLTP vs OLAP: warehouses and columnar storage — the offline store and the training data lake.
• Backpressure and load shedding — shedding ranking quality instead of failing requests.
• Deployment strategies: canary, blue-green and feature flags — rolling models out as canary A/B arms.
• Design top-K trending — the regional trending candidate source.
• Design TikTok — the upload and delivery half of a short-video app.
• Design a news feed — the follow-graph feed this one complements.

References and further reading

1. Deep Neural Networks for YouTube Recommendations (Covington, Adams and Sargin, RecSys 2016)research.google.comThe candidate-generation-then-ranking funnel, the example-age feature for fresh content, impression-count features for churn, and why live A/B results beat offline metrics.
2. Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations (Yi et al., RecSys 2019)research.googleTwo-tower retrieval at YouTube with a content-aware item tower, and a quantised index of about 10M videos rebuilt every few hours.
3. Pixie: A System for Recommending 3+ Billion Items to 200+ Million Users in Real-Time (Eksombatchai et al., WWW 2018)arxiv.orgGraph random walks over 3B nodes and 17B edges in about 120 GB of RAM, about 1,200 requests/s per server at under 60 ms p99, and a daily graph rebuild.
4. Powered by AI: Instagram’s Explore recommender system (Meta AI, 2019)ai.meta.comRanking stages of 500, 150, 50 and 25 candidates, a distilled first-pass model, the value model, author-diversity penalties and 90M model predictions per second.
5. Scaling the Instagram Explore recommendations system (Meta Engineering, 2023)engineering.fb.comReal-time and pre-generated retrieval sources, why two towers are cacheable, user vectors computed on request, hourly fine-tuning and off-peak precomputation.
6. X’s recommendation algorithm (open-source code and READMEs)github.comHome-mixer’s stages: about 6,000 features per candidate, author diversity and seen-post filters, a backup pipeline and cursoring; the heavy ranker’s weighted sum with large negative weights.
7. Faiss wiki: Guidelines to choose an indexgithub.comAll indexes live in RAM; HNSW costs d × 4 + M × 8 bytes per vector and cannot remove vectors; PQ code sizes; IVF cluster counts for 100M–1B vectors.
8. Feast documentation: Point-in-time joinsdocs.feast.devJoining feature values as of each event timestamp with a TTL, and filtering on created time so backfilled values do not leak into training.
9. Monolith: Real Time Recommendation System With Collisionless Embedding Table (Liu et al., 2022)arxiv.orgKafka action and feature queues joined by a Flink job on a request key, an on-disk store for late labels, minute-level sparse parameter sync and daily snapshots.
10. Wide & Deep Learning for Recommender Systems (Cheng et al., 2016)arxiv.orgRetrieval then ranking under a serving budget of about 10 ms, 10M+ items scored per second at peak, and 31 ms cut to 14 ms by splitting batches across threads.