Design a news aggregator (Google News)

A news aggregator reads the whole news web so its readers don’t have to. Within minutes of a publisher posting an article, the aggregator has found it, understood it, noticed that 300 other outlets cover the same event, and decided whether it belongs atop the India edition, in the Tamil edition’s Sports section, or only in one reader’s “For you” feed. It is really four systems joined together: ingestion that is fresh yet polite to 100,000 publishers, a text pipeline that turns articles into stories, a ranker that decides what matters now, and a read path for 100 million people — whose traffic doubles the moment election results or a cricket final begin.

Understand the problem

“News aggregator” can mean a headline reader — the latest items from each source, newest first — or a product that understands that 300 articles about one moon landing are a single story. Interviewers mean the second, and the story is the unit everything hangs on: ranking, personalisation, caching and the URL a reader shares.

Functional requirements

1. Users should see new articles within minutes of publication. The system collects them from about 100,000 publishers through RSS and Atom feeds, news sitemaps, push notifications and plain web pages for sites with no feed, in English, Hindi, Tamil, Bengali and dozens of other languages.
2. Users should be able to open a story and see all of its coverage. “Chandrayaan-4 lands”, reported by 300 outlets, is one story with one headline, a handful of diverse sources up front and the rest a tap away. Syndicated wire copies collapse into a single entry.
3. Users should be able to browse top stories by edition and section. An edition is a country and language, such as India–English or India–Hindi; sections are Top stories, India, World, Business, Sports, Technology and so on, plus local news for a city.
4. Signed-in users should be able to get a personalised “For you” feed that mixes the big headlines with their topics, sources and places, and tune it with “more like this” or “hide this source”.

Non-functional requirements

• Freshness. Breaking stories from about 2,000 priority publishers — wire services, national outlets — visible within 5 minutes of publication at p95. The long tail within an hour.
• Ingest scale. About 100,000 publishers with 200,000 feed and sitemap URLs between them; 1–3 million new articles a day. Plan for 2 million, with 5× bursts on big news days.
• Read scale. 100 million daily users; about 60k requests a second at a normal evening peak and 120k/s on an election-results day.
• Latency. Edition pages from the CDN in under 100 ms; the personalised feed API under 200 ms at p99 on the server.
• Availability. 99.95% for reads. If ingestion or ranking stalls, readers still get slightly stale but sensible pages.
• Quality. No two copies of one wire story in a top-stories list; no publisher holding more than 2 of the top 10 headlines; a story’s ID never changes while it grows.

Out of scope

• Crawler internals. URL frontiers, robots.txt and politeness at web scale are covered in design a web crawler; here we reuse a small focused crawler.
• Ads, comments and sharing.
• Full-text search over the archive, beyond a follow-up question at the end.
• The business side: paywalls, licensing deals and publisher payments.
• Producing fact checks. We consume labels from fact-checkers and trust signals; we do not write them.

The set-up

Plan the approach

Say the shape early: a write path that turns a stream of articles into stories, and a read path that serves precomputed lists, meeting at a ranking job that runs every minute. Writes are tiny — a few dozen articles a second — and reads are huge, so push work to the write side wherever you can. Build each requirement with the simplest boxes, then spend the deep dives on the six hard parts: freshness, duplicates, clustering, ranking, personalisation and serving under a spike.

Core entities

• Publisher. Name, home URL, country, languages, ownership group, authority score and politeness limits.
• Feed endpoint. A URL to watch — feed, news sitemap, WebSub topic or front page — with its ETag and Last-Modified validators, estimated update rate and next due time.
• Article. Canonical URL, publisher, headline, snippet, extracted text (internal only), language, UTC publish and first-seen times, entities, MinHash signature, embedding, syndication group and story ID.
• Story. A stable ID, representative headline and image, member articles, centroid, top entities, first and last update times, per-edition scores and a version.
• Edition list. A versioned snapshot mapping an edition and section to an ordered list of story IDs.
• User profile. Followed topics and sources, home city, recent clicks, MinHash cluster IDs and a topic-interest vector.

API

The read path. Edition and story pages are the same for everyone, so they are plain cacheable GETs.

GET /v1/editions/in-en/sections/top?limit=30

{
  "edition": "in-en",
  "section": "top",
  "version": 1842,
  "generated_at": "2026-10-27T12:41:00Z",
  "stories": [
    {
      "story_id": "st_8f31c2",
      "headline": "Chandrayaan-4 lander touches down near the lunar south pole",
      "source_count": 312,
      "labels": ["breaking"],
      "updated_at": "2026-10-27T12:40:12Z",
      "top_articles": [
        { "publisher": "The Example Times", "url": "https://example.in/science/chandrayaan-4-lands", "published_at": "2026-10-27T12:33:00Z" },
        { "publisher": "Dainik Udaharan", "url": "https://udaharan.in/vigyan/chandrayaan-4", "language": "hi", "published_at": "2026-10-27T12:36:00Z" }
      ]
    }
  ]
}

GET /v1/stories/st_8f31c2?edition=in-en&cursor=… returns the full coverage of one story, page by page. The personalised feed needs the user, so it carries a token and is never cached publicly; each item says why it is there.

GET /v1/feed/for-you?edition=in-en&cursor=eyJ2IjoxODQyfQ
Authorization: Bearer <access-token>

{
  "items": [
    { "story_id": "st_8f31c2", "reason": "top_story" },
    { "story_id": "st_2a7d90", "reason": "follows:topic/isro" },
    { "story_id": "st_c0b114", "reason": "local:pune" }
  ],
  "next_cursor": "eyJ2IjoxODQyLCJvIjoyMH0"
}

Clients batch impressions and clicks to POST /v1/events; those clicks feed personalisation and quality metrics.

The ingest side, where the aggregator is the client. A routine feed poll is a conditional GET, and most answers are a bodiless 304:

GET /rss/national.xml HTTP/1.1
Host: news.example.in
If-None-Match: "6f1c-5a2"
If-Modified-Since: Tue, 27 Oct 2026 06:58:10 GMT
User-Agent: HandbookNewsBot/1.0 (+https://example.org/bot)

HTTP/1.1 304 Not Modified
ETag: "6f1c-5a2"

Publishers that support push deliver their updated feed to a callback we registered, signed with a shared secret:

POST /websub/cb/feed-29411 HTTP/1.1
Content-Type: application/rss+xml
X-Hub-Signature: sha256=9b2f0c...

Back-of-envelope estimates

These numbers decide what is expensive (method: back-of-envelope estimation).

Feed polling. Polling all 200,000 feed and sitemap URLs every minute is 200,000 ÷ 60 ≈ 3,300 fetches/s; at about 50 KB per feed, 3,300 × 50 KB ≈ 167 MB/s or 1.3 Gbit/s, almost all unchanged. Polling by tier instead — 2,000 feeds every minute (33/s), 20,000 every 5 minutes (67/s), 78,000 every 30 minutes (43/s) and 100,000 every 3 hours (9/s) — totals about 150 fetches/s, 22 times fewer.

Articles. 2M a day ÷ 86,400 s ≈ 23 articles/s, about 120/s in a 5× burst. Each article page is fetched once at roughly 150 KB: 2M × 150 KB = 300 GB/day, about 28 Mbit/s. Compressed to 30 KB, raw HTML is 60 GB a day, 22 TB a year; extracted text and metadata at 7 KB is 14 GB a day, about 5 TB a year.

The live working set. Stories change for about two days, so the hot set is 48 hours of articles: 4M. At 6–7 articles per story on average (a few giants, many singletons), that is about 600k live stories. A MinHash signature (120 four-byte values) plus 20 eight-byte band keys is 640 B per article: 4M × 640 B ≈ 2.6 GB. A 384-dimension float centroid is 1.5 KB: 600k × 1.5 KB ≈ 0.9 GB. All of it fits in memory.

Reads. 100M daily users × 10 page or feed loads = 1B requests a day ÷ 86,400 ≈ 11.6k/s on average. An evening peak at 5× is 58k/s, and results day doubles that to about 120k/s. If 60% are shared edition, section and story pages, the CDN absorbs them; the personalised 40% reach our servers: 0.4 × 120k ≈ 48k/s.

Clicks. 100M users × 6 clicks = 600M clicks a day ≈ 7k/s, 35k/s at peak. Each click updates about 10 co-visitation pairs in both directions (20 increments) and about 15 user-cluster counters: 35k × 35 ≈ 1.2M counter increments/s at peak, which must be batched.

Profiles. 100M users × 2 KB = 200 GB in a key-value store.

Ranking. 70 editions × 12 sections = 840 lists. Rescoring up to 50k live stories per list every minute is 840 × 50k = 42M scores a minute, about 700k/s — a few cores of arithmetic over in-memory features.

High-level design

Build the design one requirement at a time. By the end it works for one region at moderate traffic; the deep dives then make it fast, accurate and robust.

Users should see new articles within minutes of publication

There are three ways to learn that a publisher has a new article, and you need all three because publishers differ:

• Pull. Most news sites publish RSS or Atom feeds per section, and many a news sitemap. A feed poller fetches them on a schedule and diffs the items; RSS 2.0 recommends a guid per item precisely so aggregators can tell new items from old.
• Push. Some feeds advertise a WebSub hub. We subscribe once, and the hub calls our WebSub receiver within seconds of each update.
• Crawl. For sites with neither, a page crawler fetches section front pages and looks for unseen links.

The article fetcher drops URLs it has already fetched, fetches the rest politely (one host per worker, with per-host limits, as in the web crawler) and stores the raw HTML. The extractor turns each page into a clean record:

• Canonical URL. Follow redirects, prefer rel=canonical, strip tracking parameters such as utm_source, and map AMP copies to the main URL, so an article listed in three section feeds becomes one record.
• Main text. Drop navigation, “also read” boxes and ads using text- and link-density heuristics, helped by per-publisher templates learned from blocks that repeat across a site’s pages.
• Language. Detect it from the body, not the feed’s label, or a Hindi site’s English section lands in the wrong edition.
• Publish time. RSS uses RFC 822 dates, news sitemaps W3C dates (with a zone whenever a time is given), pages anything at all. Normalise to UTC; a time with no zone, such as “27 अक्टूबर 2026, 6:05 PM”, takes the publisher’s zone, so 18:05 IST is 12:35 UTC. Reject future times and use the earlier of the stated time and our first-seen time, so re-stamping old articles as new gains nothing.
• Entities. Link people, places and organisations to knowledge-base IDs, so “ISRO”, “इसरो” and “Indian Space Research Organisation” become one entity.

The record goes onto an articles topic in Kafka, and a consumer writes it to the Article DB. With the log as the hand-off, every later stage can consume at its own pace and replay after a bug fix.

Users should be able to open a story and see all of its coverage

Two kinds of “sameness” hide in a news stream, and they need different tools. A syndicated copy is the same text: a PTI or Reuters report republished by fifty outlets under their own headlines. A story is the same event in different words: 300 newsrooms each writing about the landing. Manku, Jain and Das Sarma draw exactly this line: finding other sources that report the same event is a semantic similarity, unlike syntactic near-duplicates.

So add two stream consumers. A near-duplicate check compares each article’s text with the last 48 hours and puts copies in a syndication group — kept, because readers may prefer their own outlet, but represented by one copy in lists. A story clusterer then joins the article to the best-matching live story or starts a new one. Membership, the representative headline and chosen sources live in a Story DB, served by a Story API.

The full-coverage page of a story is deliberately the same for every reader. When Google relaunched its news product in 2018, its announcement described Full Coverage as an unpersonalised view of an event from a range of trusted sources, so that everyone starts from the same facts. Personalise which stories a reader sees, never which version of the facts inside a story.

Users should be able to browse top stories by edition and section

Top stories change by the minute but are the same for everyone in an edition, so compute them on a timer. Every 60 seconds a ranker scores the live stories for each edition and section — how many sources, how authoritative, how recent, how relevant to that country and language — and writes the ordered list as a versioned snapshot, such as in-en/top v1842. A classifier assigns sections, and stories naming a city go into its local list. An editorial console lets a small team pin, demote or label stories, as inputs to the ranker rather than edits to its output.

The Edition API keeps the latest snapshots in memory — 840 lists plus story cards is a few hundred megabytes. In front of it a CDN caches each page for 60 seconds, so a busy page costs the origin about one request a minute per cache location, not one per reader.

Signed-in users should be able to get a personalised “For you” feed

“For you” cannot be one list per edition, but it need not be computed from scratch either. A feed service starts from the reader’s edition list, adds stories suggested by the reader’s own signals, scores them all and returns the top 20. It reads three things, from memory or a fast key-value store:

• Edition lists, the Edition API’s snapshots, so every reader gets the important headlines.
• The user profile: followed topics and sources, home city, recent clicks and precomputed user clusters.
• Click statistics: for each live story, which stories its readers also opened, and how often each user cluster opened it.

Clicks arrive on a clicks topic. A stats updater applies them within seconds, so a click changes the reader’s next feed, and an offline clustering job rebuilds user clusters from months of clicks once a day. This mirrors the system Das, Datar, Garg and Rajaram describe in their 2007 Google News paper: offline clustering, a server that updates statistics on every click, and a server that scores candidates on request.

You now have a working aggregator: articles arrive, collapse into stories, get ranked per edition and personalised per reader. What remains is making each part fast, accurate and robust at the stated scale.

Deep dives

How do we find new articles within minutes without hammering 100,000 publishers?

Work backwards from the 5-minute promise. An article waits to be discovered, then about 12 seconds to be fetched, extracted and clustered, up to 65 seconds for the next ranking run to publish and push a list, and up to 60 seconds for the CDN’s copy to expire. Only the first wait is really ours to choose.

Bad: poll every feed every minute. Priority publishers make the deadline, but it costs 3,300 fetches a second and 1.3 Gbit/s of mostly unchanged feeds — a weekly blog gets polled 10,080 times between posts — and small publishers block the bot.

Good: adaptive polling with conditional GETs. Give each feed its own interval, learned from how often it changes. Keep an exponentially weighted estimate of new items per hour and set the next interval to about the time in which you expect one new item, clamped between 1 minute and 3 hours. A wire service posting 100 items an hour hits the 1-minute floor; a district paper posting 12 a day gets about 2 hours. RSS 2.0’s ttl (minutes a channel may be cached) and skipHours (GMT hours an aggregator may skip) are useful hints for the long tail, but do not let a stale ttl of 60 slow down a wire you know posts every two minutes.

Then make each poll cheap: send back the last ETag as If-None-Match and the last Last-Modified as If-Modified-Since, and an unchanged feed answers 304 Not Modified with no body. RFC 9110 says a server must ignore If-Modified-Since when If-None-Match is present, and judges the date on its own clock — so echo the server’s Last-Modified string, never your fetch time. At 150 fetches a second with 70% of answers being 0.5 KB 304s, traffic is 150 × (0.3 × 50 KB + 0.7 × 0.5 KB) ≈ 2.3 MB/s, down from 167 MB/s.

The poller is a job scheduler in miniature: each shard owns a set of hosts and keeps a min-heap of feeds by next-due time, so a newspaper’s 30 section feeds share one connection pool and one politeness delay. The gain is a 22× cut in fetches. The cost: when that district paper breaks a big local story, its 2-hour interval misses the target badly.

Great: push where offered, poll as a safety net, sitemaps and crawling for the rest. WebSub, a W3C Recommendation formerly called PubSubHubbub, turns polling around. A feed advertises a hub (rel="hub") and its canonical URL (rel="self") in Link headers or link elements. You subscribe once, the hub confirms the request came from you, and from then on it POSTs the updated feed to your callback as soon as the publisher notifies it.

The details that matter in production:

• Verify intent and payloads. Echo the hub’s random hub.challenge to confirm a subscription — that stops anyone pointing your callback at junk. With a hub.secret, each delivery carries an X-Hub-Signature HMAC of the body (sha256 is one defined algorithm); reject mismatches.
• Acknowledge fast. A 2xx should mean only “received”: queue the payload, reply, parse later.
• Renew leases before hub.lease_seconds runs out; renewing does not interrupt delivery.
• Keep polling slowly. Hubs retry failed deliveries only up to their own limits, then give up. A 30-minute background poll of pushed feeds catches the rare loss at 1/30 of the polling cost.

For publishers with a news sitemap, poll that instead of crawling pages. Google’s rules make it a compact, bounded diff: only articles from the last two days, at most 1,000 news entries per sitemap, each with a language code and a W3C-format publication date. Sites with neither get their section front pages crawled — every few minutes for priority sites, hourly for the rest — and unseen article-shaped links become new articles.

The gain is freshness in seconds for publishers that push and a polite, predictable load for everyone else. The cost is four intake paths, a public callback to secure and a per-feed rate model to watch. One trick for the long tail: when a story breaks, move every feed in its region or beat to a 1-minute interval for an hour.

How do we spot the same wire story republished by 50 outlets?

At 18:07 a news agency moves a 400-word report. By 18:30 fifty sites have published it, each with its own headline, some with an extra paragraph of local reaction, several cut after the fourth paragraph. Counted naively, one report looks like fifty sources, and top stories can show the same text five times. The goal: put copies in one syndication group, keep every copy reachable, show one, and credit the original.

Bad: an exact checksum of the body. Eight bytes per article catches pure reposts, but a changed byline or one extra sentence gives a new hash, so most syndicated copies slip through.

Good: 64-bit simhash with a Hamming-distance index. This is the web crawler’s tool. Charikar’s random-hyperplane construction makes each bit agree with a probability tied to the angle between two documents’ feature vectors, so similar texts get fingerprints that differ in few bits; Manku, Jain and Das Sarma found 64 bits with a 3-bit threshold worked on 8 billion web pages. It is tiny and proven. The cost is tuning: the threshold moves only in whole bits, and the fast lookup tables are laid out for one fixed threshold, so changing your idea of “a copy” means rebuilding the index — and it was tuned for whole pages differing in ads and footers, not short reports with a rewritten top.

Great: MinHash signatures with LSH banding. Represent each article as a set of shingles — overlapping runs of 4 words from the extracted body, after boilerplate removal (otherwise site templates dominate). Two sets’ Jaccard similarity is shared shingles over all distinct shingles: a copy keeping 80% of the original plus 10% new text scores 0.8 ÷ 1.1 ≈ 0.73. Comparing against 4 million recent articles directly is impossible, so compress and index:

1. MinHash. Hash the shingles with 120 different functions and keep each minimum. For one function, two articles share the minimum with probability exactly equal to their Jaccard similarity — Charikar cites min-wise hashing as an instance of a locality-sensitive hash. 120 four-byte minimums make a 480-byte signature.
2. Banding. Cut the signature into b = 20 bands of r = 6 values and hash each band to a bucket. Two articles sharing a bucket in any band are a candidate pair.
3. Verify. Estimate Jaccard as the fraction of the 120 values that match; call it a copy at 0.7 or above.

For similarity s, one band matches with probability s⁶, and at least one of 20 with probability 1 − (1 − s⁶)²⁰ — the S-curve from chapter 3 of Mining of Massive Datasets, whose threshold sits near (1/b)^(1/r) = (1/20)^(1/6) ≈ 0.61.

Work through it. At s = 0.8, one band matches with probability 0.8⁶ ≈ 0.26 and none of 20 with (1 − 0.26)²⁰ ≈ 0.002, so 99.8% of such copies become candidates; at 0.7, 92%; at 0.5, 27%; at 0.3, only 1.5% — and different newsrooms covering one landing share few 4-word runs. The band threshold (0.61) sits below the decision threshold (0.7) on purpose: banding errs towards recall, and the exact check removes false positives. Longer bands (16 × 8) would move the threshold to about 0.71 and catch only 61% of copies at 0.7.

The cost is small: 20 bucket lookups per article, under 500 a second, against a 2.6 GB in-memory table. The gain over simhash is control: b and r are knobs, and the threshold is in units of shared text.

Inside a group, the original is the agency’s own article if it publishes one, otherwise the earliest copy from the most authoritative outlet. The group counts as one source for clustering and ranking. The same index catches a quieter abuse: a publisher posting near-identical articles under twenty URLs to look like more coverage — they land in one group and count once.

How do we group a stream of articles into stories?

Stories are harder than copies: “ISRO’s lander touches down”, “चंद्रयान-4 की सफल लैंडिंग” and “What the landing means for India’s space budget” share almost no shingles but belong together, while articles about two different train accidents on the same day share many words and do not.

Bad: batch clustering every hour. K-means or hierarchical clustering over recent articles can be accurate, but a breaking story waits up to an hour to exist, you must guess k, and each run hands out new IDs, breaking story URLs, caches and click statistics.

Good: online single-pass assignment. Treat each arriving article as a query against the live stories:

1. Features. A multilingual sentence embedding of the headline and first paragraphs (see embeddings and semantic search), the linked entity IDs, and a TF-IDF vector of title and lead as a cheap lexical check.
2. Candidates. Live stories sharing an entity, from an inverted entity index, plus the 20 nearest story centroids from an approximate nearest-neighbour index (a vector database or in-memory equivalent), limited to stories updated in the last 72 hours.
3. Score. For example 0.7 × cosine(embedding, centroid) + 0.3 × Jaccard(entities), multiplied by 0.5^(gap ÷ 24 h), where gap is the time since the story’s last article.
4. Decide. Join the best story if it scores at least 0.70 and update its centroid as a running mean; otherwise start a new story. Tune the threshold on a few thousand pairs labelled by editors.

Each assignment takes a millisecond or two, so bursts need only a few cores. The cost is quality, in three predictable ways. Thin first reports split stories: “explosion heard in Pimpri” and “fire at Pune industrial unit, four hurt” share no entity and few words, so they start two stories. Drift chains stories together: a centroid following a long-running story (“monsoon”) can wander until it swallows unrelated articles. And order matters: the same articles in another order can cluster differently.

Great: online assignment plus a batch repair pass with stable IDs. Keep the online path for speed, and every 10–15 minutes re-cluster the last 48 hours properly — the classic split from batch vs stream processing. Build a similarity graph over articles with candidate pairs from the ANN index, then group it with average-linkage agglomerative clustering or a graph community method. Half an hour later, the Pimpri and Pune reports share the factory’s name and casualty count, and the batch pass merges them; a “monsoon” cluster with two dense halves splits.

The subtle part is reconciling IDs, because story IDs live in shared links, cached pages, editorial pins and click statistics. Each new cluster inherits the ID of the old story it overlaps most, if it holds more than half of that story’s articles. A story absorbed elsewhere becomes a redirect to the survivor, which inherits its click counts. Only genuinely new groups get new IDs.

Three more decisions finish the story layer:

• Languages. Cluster within each language first, where precision is highest, then link language-level stories that share entity IDs and have close multilingual embeddings. The Hindi edition shows Hindi sources first, with English coverage a tap away.
• Storylines. Stories on one ongoing subject — the Chandrayaan-4 mission from launch to landing — link into a storyline through shared entities over weeks, which powers timeline views; Google’s 2018 relaunch described organising articles into storylines in real time.
• Headline and sources. Take the headline from the article nearest the centroid among authoritative sources in the edition’s language, preferring original reporting and plain statements over question headlines. Show four or five sources, at most one per publisher or ownership group — national, local, specialist, and at most one wire copy — and change the headline only when the story materially moves on.

How do we rank top stories so the big news wins, and wins fast?

The ranker has 50,000 live stories for an edition and 30 slots. It must put the budget ahead of a celebrity wedding, a sudden landing ahead of the budget within minutes, and never let one publisher’s view dominate — for 840 lists every minute.

Bad: newest first, or most clicked. Newest first fills the page with minor items and repeated wire copy. Most clicked is late on breaking news because it needs clicks first, rewards whatever is already on top, is easy to game and drifts towards entertainment — Liu, Dolan and Pedersen saw a click-based recommender at Google News push popular entertainment stories even to readers who never clicked them.

Good: authority-weighted, time-decayed coverage. Let newsrooms vote with their reporting. Each distinct publisher covering the story adds its authority score, halved for every 4 hours since its article appeared:

score = Σ over publishers p of authority(p) × 0.5^(age(p) ÷ 4 h)

Counting publishers, not articles, and a syndication group once, makes the score hard to inflate by reposting. Authority comes from a publisher’s history — original reporting, corrections, how often its stories are corroborated — and is a reviewed policy decision. Recency is built in: a story stops rising when newsrooms stop writing about it.

Check the numbers. Story A’s 300 sources (authority 0.5) arrived 11:00–14:00, so at 18:00 they are 4–7 hours old and keep about 39% of their weight on average: 300 × 0.5 × 0.39 ≈ 58. Story B gains 4 sources a minute from 18:00; by 18:30 it has 120 sources of authority 0.6 keeping about 96%: 120 × 0.6 × 0.96 ≈ 69. The gain is a simple, explainable score that needs no clicks. The cost: one half-life does not fit every section (match reports stale in two hours, investigations relevant for a day), and coverage alone cannot tell a story that matters to Pune from national noise.

Great: score features, then apply constraints. Keep coverage as the backbone and add features, combined by a model trained on editors’ pairwise judgements (“which belongs higher?”), with engagement only as a minor, position-corrected signal:

• Velocity: distinct sources in the last 10 minutes against the usual rate for the story’s entities.
• Originality: the share of original reporting versus syndicated copies.
• Local relevance: entity and publisher locations matched to the edition and city.
• Section fit: a classifier’s score, with a decay half-life per section.
• Trust labels: fact-check labels and source trust, which can demote but never promote.

Then apply rules the model may not break: at most 2 headlines per publisher in the top 10, one story per storyline in the top 5, editorial pins and demotions, and hysteresis — swap neighbours only when scores differ by more than about 10%, so the list does not reshuffle every run. Recomputing everything each minute is fine at 700k cheap scores a second; design Reddit makes a different trade because millions of live posts favour a score that never needs recomputing.

Breaking-news detection puts a high bar on velocity. Count mentions per entity per minute in a count-min sketch — fixed memory across millions of entities — and keep exact publisher sets only for stories that pass that filter. Flag a story when, say, 40 distinct sources appear within 10 minutes, at least 10 times its entities’ usual rate, including three independent high-trust publishers. The flag boosts the story, adds a label, speeds up related feeds and can trigger alerts; the windowed counting is the subject of design top-K trending.

How do we personalise “For you” for 100 million readers?

The feed has about 10 ms of server time to pick 20 stories from tens of thousands. The 2007 paper by Das and colleagues describes the same pressure: Google News typically built its home page within a second, leaving a few hundred milliseconds for recommendations, and stories churned every few minutes, so a model a few hours old was already stale.

Bad: find similar users at request time. Compare the reader’s clicks with everyone else’s and recommend what the closest readers clicked. The paper explains why this fails: popular stories connect almost everyone, so even users with one click in common are too many, and precomputing all pairs offline is infeasible. It also has what Liu and colleagues call the first-rater problem: a story with no clicks cannot be recommended until hours after it breaks.

Good: a content-based profile. Keep decayed interest scores for the topics, entities and sources each reader reads, and match candidate stories against them. It works the moment a story is clustered and never pushes entertainment on someone who never reads it. The cost is narrowness: Liu, Dolan and Pedersen found interests shift with the news — national-politics clicks doubled during a US election campaign — so they blended each reader’s interests with the current trend in their country.

Great: several candidate sources, one light re-rank, statistics updated online. Gather about 800 candidates from five sources, then score them with a small model.

The two collaborative sources come straight from the Google News paper:

• Co-visitation. Two stories are co-visited when one reader clicks both within a few hours. Each live story keeps time-decayed counts of its co-visited stories, updated on every click; a reader’s candidates are the stories most co-visited with their last few hours of reading. It needs no long history, so the paper relies on it for new users, and it picks up a breaking story as soon as readers start opening it.
• MinHash user clusters. Treat each reader as the set of stories they clicked. Concatenating p MinHash values into a cluster key, q times over, puts each reader in q clusters; the paper used p of 2–4 and q of 10–20, computed by MapReduce over a few months of clicks. It is the duplicate check’s banding trick applied to people: with p = 3 and q = 15, readers whose histories overlap at Jaccard 0.6 share a cluster with probability 1 − (1 − 0.6³)¹⁵ ≈ 97%; at 0.3, about 34%; at 0.1, under 2%. Each story keeps a decayed click count per cluster, and a reader’s score for it sums over their clusters.

The paper combined MinHash, a second model (PLSI) and co-visitation linearly, tuned the weights with live experiments, and measured about 38% more clicks than recommending recently popular stories — though popularity still won when one irresistible celebrity story dominated. Liu, Dolan and Pedersen later added the trend-blended content profile on top, and reported 30.9% higher click-through on recommendations and 14.1% more visits per day.

The re-rank is deliberately light: about 30 features per candidate (source scores, topic affinity, the story’s edition score, freshness, source preferences, already shown) in a small tree ensemble. At about 5 µs each, 800 candidates cost 4 ms of CPU. A diversity pass caps stories per topic and publisher and fixes the opening mix — Google described For you as opening with five stories mixing top headlines, local news and the reader’s topics.

Three operational rules complete it. Update online, cluster offline: the stats updater batches clicks for 5 seconds, collapsing 1.2M increments a second into far fewer writes because hot stories repeat within a batch; clusters are rebuilt daily. Degrade gracefully: the paper separated the statistics updater from the recommender, so if updates stop, recommendations continue on slightly stale counts. Filter abuse: it ignored clicks from users with abnormally high click rates. New readers pick a few topics, and co-visitation takes over after their first clicks.

How do we serve 120,000 requests a second when news breaks?

Results day is predictable in date and brutal in shape: traffic doubles within minutes, everyone wants the same few pages, and those pages change every minute.

Bad: build every page on request. Reading the Story DB and ranking per request at 120k/s needs a huge fleet and melts the databases the moment counting starts.

Good: cache shared pages at the CDN. Edition, section and story pages are identical for every reader in an edition, so the CDN serves them with a 60-second TTL, and stale copies if the origin errors. Count the misses: about 2,000 cacheable list URLs (editions × sections × formats) across 300 edge locations, each refreshing once a minute, is up to 2,000 × 300 ÷ 60 = 10,000 origin requests a second, arriving in bursts as pages expire everywhere at once.

Great: a shield tier, a fallback ladder and versioned story pages. A regional shield tier of 10 caches cuts misses to 2,000 × 10 ÷ 60 ≈ 333/s, and by coalescing simultaneous misses for one URL it stops a story page that suddenly takes 30% of traffic from stampeding the Edition API (see hot keys and cache stampedes). Personal feeds take about 4 ms of CPU each, so a 16-core server at 50% load handles about 2,000 a second: 24 servers cover the 48k/s peak and about 50 give zone redundancy. When latency climbs, shed work in steps rather than failing — load shedding applied to quality, not requests.

Story pages need consistency while they change: a story on results night gains articles every few seconds and may be merged by the batch pass. Render each page from one story version, never half an update. Paginate the article list with a keyset cursor on (published_at, article_id) bound to that version, so new arrivals do not shift page 2 under the reader (see keyset pagination). Merged IDs answer with a 301 to the survivor. And for the 50 hottest stories, purge the CDN when the headline or a label changes rather than waiting out the TTL.

Before scheduled peaks — budget day, results day, a World Cup final — scale out ahead of time. The gain is a read path whose cost grows with distinct pages, not readers. The cost is up to a minute of staleness, which the requirements accept, and a fallback ladder that must be exercised regularly, because rungs that are never used quietly break.

The final design

Put the pieces together. The write path runs top to bottom on the left and centre; the read path runs along the bottom; the ranker’s snapshots are where they meet.

1. Discover. WebSub pushes arrive in seconds; adaptive pollers fetch feeds and news sitemaps with conditional GETs; a small crawler covers the rest.
2. Fetch and extract. Each new canonical URL is fetched once and becomes a clean record — text, language, UTC time, entities — on the articles topic.
3. Group copies. MinHash with 20 bands of 6 finds syndicated copies within 48 hours; a group counts as one source.
4. Cluster. The article joins the best live story above the threshold, or starts one, within seconds.
5. Rank. Every minute the ranker scores live stories per edition and section — decayed, authority-weighted coverage plus velocity, locality and trust, under diversity rules and editorial overrides — and publishes versioned snapshots.
6. Serve shared pages. The CDN and its shield answer almost all edition and story requests; the Edition API serves misses from memory.
7. Serve For you. The feed service merges edition, co-visitation, user-cluster, topic and local candidates, re-ranks about 800 in 4 ms and diversifies, falling back a rung at a time under load.
8. Learn. Clicks reach the stats updater, which updates decayed co-visitation and cluster counts in 5-second batches; user clusters rebuild daily.
9. Repair. Every 15 minutes a batch pass re-clusters the last 48 hours, keeping story IDs stable.

Check it against the requirements. A priority article is visible within about 3.3 minutes when polled and 2.4 when pushed, inside the 5-minute target. Ingestion costs about 150 fetches a second, not 3,300. The origin sees about 333 shared-page requests a second at a 120k/s peak, and personalised feeds need about 4 ms of scoring against a 200 ms budget. Wire copies count once, no publisher takes more than 2 of the top 10, story IDs survive re-clustering, and if ingestion stops, readers still get the last good snapshots.

What interviewers look for by level

Follow-up questions

What if a fake or misleading story starts spreading?

Make trust structural. A story reaches top stories or a breaking label only when several independent high-trust publishers corroborate it; a fast-growing story backed only by low-trust sources is held for review. Show fact-check labels, demote rather than delete without review, and keep Full Coverage unpersonalised so every reader sees the same range of sources. A story whose key facts flip within an hour gets a “developing” label.

What if a publisher spams near-duplicate articles?

The MinHash index already puts them in one syndication group, and ranking counts publishers, not articles, so the flood earns nothing. Track each publisher’s duplicate rate; above a threshold, poll it less often, lower its authority and alert a reviewer. Count sister sites under one owner as one voice.

What if we add a new language?

Each layer needs something. Extraction needs a tokeniser — Thai and Japanese put no spaces between words, and Indic scripts need correct handling of combining characters — and language identification that copes with Hinglish. Entity linking needs aliases and transliterations (“Bengaluru”, “Bangalore”, “ಬೆಂಗಳೂರು”). Multilingual embeddings let clustering work from day one; section classifiers wait for labelled data. Seed the publisher list by hand and keep editors close until quality settles.

What if users want push alerts for breaking news?

Reuse the breaking detector with a stricter bar — more independent sources, and a human check for the widest alerts — and fan out through a notification system. Target by edition, topics and city; cap alerts per user per day; send one per story, never per article; and stagger delivery over a minute or two so millions of taps do not hit the CDN in the same second.

What if we need full-text search over the archive?

Index every article — headline, extracted text, entities, publish time — into a search index fed from the articles topic, with a recency boost and results collapsed by story so one event does not fill the page. The indexing and ranking basics are in design a search engine like Google, and near-real-time indexing of a fast stream is the core of design post search.

How to say it in the interview

“I’ll design a Google News–style aggregator: about 100,000 publishers, 2 million articles a day, priority stories visible within 5 minutes, and 100 million readers with peaks around 120 thousand requests a second.

The write side is small, so I precompute there. Ingestion uses WebSub push where publishers offer it, adaptive polling of feeds and news sitemaps with conditional GETs — about 150 fetches a second instead of 3,300 — and front-page crawling for the rest. Each article is fetched once and extracted into a clean record with canonical URL, language, UTC publish time and entities, onto Kafka.

Two stream consumers turn articles into stories. MinHash with 20 bands of 6 groups syndicated wire copies, so a copy counts once. A clusterer assigns each article to a live story by entity overlap and embedding similarity, or starts a new one, and a batch pass every 15 minutes repairs merges and splits while keeping story IDs stable.

Every minute a ranker scores live stories per edition: authority-weighted coverage that halves every 4 hours, plus velocity, locality and trust, with per-publisher caps and editorial overrides. A velocity spike with three trusted sources raises the breaking flag. Lists are versioned snapshots; a CDN with a shield tier serves shared pages, so the origin sees a few hundred requests a second.

For you merges edition candidates with co-visitation and MinHash user clusters, as in the Google News paper, plus followed topics and local stories, then re-ranks about 800 candidates in 4 milliseconds. Clicks update decayed counts within seconds; clusters rebuild daily. Under load the feed falls back a rung at a time, down to the edition list.”

Concepts used in this design

• Design a web crawler — polite fetching, conditional GETs and simhash, reused for intake.
• Design a search engine like Google — indexing and ranking basics for archive search.
• Push vs pull architecture — WebSub push versus feed polling.
• Webhooks — WebSub deliveries are signed webhooks.
• Design a job scheduler — next-due heaps for feed polling.
• Kafka — the articles and clicks topics that decouple each stage.
• Batch vs stream processing — online clustering plus batch repair.
• Embeddings and semantic search — features for story clustering across languages.
• Vector databases — nearest-centroid lookups for candidate stories.
• Count-min sketch and HyperLogLog — per-entity velocity counts for breaking news.
• Design top-K trending — windowed counting behind breaking detection.
• Design a recommendation feed — candidate generation and re-ranking in depth.
• Content delivery network — serving shared edition pages at the edge.
• Hot keys, cache stampede and invalidation — coalescing misses on a viral story.
• Backpressure and load shedding — the For you fallback ladder.
• Design a notification system — breaking-news alerts.

References and further reading

1. Google News Personalization: Scalable Online Collaborative Filtering (Das, Datar, Garg and Rajaram, WWW 2007)research.googleMinHash user clusters, PLSI and co-visitation combined linearly; offline MapReduce plus online time-decayed counts; 38% more clicks than a popularity baseline.
2. Personalized News Recommendation Based on Click Behavior (Liu, Dolan and Pedersen, IUI 2010)static.googleusercontent.comWhy pure collaborative filtering lags on new stories, how interests follow the news cycle, and a hybrid that lifted recommendation CTR by 30.9%.
3. Detecting Near-Duplicates for Web Crawling (Manku, Jain and Das Sarma, WWW 2007)static.googleusercontent.com64-bit simhash with a 3-bit threshold on 8 billion pages, and the point that same-event news is a semantic, not syntactic, similarity.
4. Similarity Estimation Techniques from Rounding Algorithms (Charikar, STOC 2002)cs.princeton.eduLocality-sensitive hashing defined as Pr[h(x) = h(y)] = sim(x, y); min-wise hashing for Jaccard and random hyperplanes for cosine.
5. Mining of Massive Datasets, chapter 3: Finding Similar Items (Leskovec, Rajaraman and Ullman)mmds.orgShingling, MinHash signatures and the banding technique with its 1 − (1 − s^r)^b S-curve.
6. WebSub (W3C Recommendation)w3.orgHub discovery with rel=hub and rel=self, subscription with a verification challenge, leases, HMAC-signed content delivery and hub retries.
7. RSS 2.0 Specification (RSS Advisory Board)rssboard.orgItem guid for spotting new items, RFC 822 pubDate, and the ttl, skipHours and skipDays hints for aggregators.
8. News sitemaps (Google Search Central)developers.google.comOnly articles from the last two days, at most 1,000 news entries per sitemap, and W3C-format publication dates that need a time zone whenever they include a time.
9. RFC 9110, Section 13: Conditional Requestsrfc-editor.orgIf-None-Match takes precedence over If-Modified-Since, which is judged on the server’s clock; 304 Not Modified carries no body.
10. The new Google News: AI meets human intelligence (Google blog, 2018)blog.googleOrganising a constant flow of articles into storylines in real time, a five-story briefing in For you, and a Full Coverage view that is the same for everyone.