120k/s
requests on an election-results day
But only about 23 articles a second arrive. Precompute on the small write side; cache the huge, mostly identical read side.
How do you find new articles?
Bad: Poll everything every minute
3,300 fetches a second, nearly all unchanged.
Good: Push, then poll adaptively
WebSub pushes in seconds. Other feeds poll at their own rate: ≈150 fetches/s, mostly 304s.
One wire report, fifty outlets. Count it once.
MinHash each article’s 4-word shingles, in 20 bands of 6. Copies at 0.7 similarity collide 92% of the time; same-event rewrites at 0.3, 1.5%.
A batch pass finds the Pimpri blast and the Pune factory fire are one story. Which ID survives?
The trick
The one it overlaps most. The other becomes a 301, so links, pins and clicks survive.
A landing breaks at 18:00. It tops the Budget by 18:23.
Each distinct publisher adds its authority, halved every 4 hours. 120 fresh sources outweigh 300 from the morning. No clicks, no editor.
333/s
reach the origin on results day
300 CDN edges answer most of 72k shared-page requests a second. A shield tier of 10 coalesces their misses: up to 10k/s becomes 333/s.
“For you”: personalise only the last step
- ≈ 800 candidates: edition, topics, co-visitation, clusters, local
- A light model re-ranks them in 4 ms
- Each click updates counts within seconds
- Under load, degrade a step at a time
Remember
Push plus adaptive polls find articles fast. Group copies, cluster stories with stable IDs, rank every minute. A CDN serves snapshots; personalise only the last step.