20M
heavy-model scores a second at peak
100k requests a second × 200 candidates. Every extra 100 candidates costs about 100 more GPU servers; the funnel keeps that product small.
A billion items in, ten on screen
Six sources nominate 1,500 in parallel; a cheap pre-ranker keeps 200; a heavy GPU model scores those; a re-ranker applies rules and returns 10.
Search a billion items in tens of milliseconds, yet react to your last swipe?
The trick
Two towers meet only at a dot product. Item vectors are indexed offline; only your vector is computed, per request.
Skip three cricket clips. The next page knows.
The skip reaches your features in about 2 s; the next page is fetched at about 10 s. The model didn’t change, only its inputs.
How do you build training rows?
Bad: Recompute features later
Training code drifts from serving, and today’s like count leaks the future.
Good: Log what the ranker saw
Join outcomes, by impression ID, to the exact features served.
A new clip has no views. Give it a budget.
Reserve 5% of impressions: 200 test views for every upload, then 1,000 more for the top 10%. Winners graduate to normal ranking.
Serving the endless scroll
- Save unshown scored candidates under the cursor
- Next page: re-rank that slate in ≈15 ms
- Re-run fully every second page, or on strong signals
- Served list + seen filter stop repeats
Remember
A funnel on the request path, a learning loop around it. Spend GPUs only on the best 200, and log exactly what the model saw.