Design post search (Twitter / Facebook)

Post search is the box at the top of a social app. Someone types “ind vs aus” in the last over of a cricket final, “#diwali” on the festival evening, or “from:mumbaipolice waterlogging” on a flooded Monday morning, and expects posts written seconds ago, not last week. The plumbing looks like web search — an inverted index, partitions, scatter-gather, ranking in two phases — and this page will not re-derive it; design a search engine like Google covers posting lists, compression and tail latency. Post search earns its own design through four twists. The corpus grows by thousands of documents a second, and each must be searchable almost at once. Time is the strongest ranking signal. The popularity signals you rank by change every second. And every result passes a visibility check that is different for every viewer. Add traffic that arrives in tidal waves whenever the world does something interesting, and you have one of the most frequently asked search problems.

Understand the problem

Pin down which search this is in the first minute. Web search ranks a mostly static corpus by relevance and authority, and a page can wait hours to be indexed. Product search ranks a bounded catalogue — around 100 million listings — by attributes, facets and business rules; prices and stock move fast, but the set of listings grows slowly. Post search is a third shape: an append-only firehose where the newest matching post is often the best answer, popularity is measured in seconds, and two people typing the same words must see different results because they are allowed to see different posts.

Functional requirements

1. Users should be able to search posts by keywords, hashtags and phrases, with filters: author (from:), time range (since: and until:), has:media, language, and exclusions (-spoiler).
2. Users should be able to choose “Latest” or “Top”. Latest is reverse chronological. Top blends text match, engagement, author quality and recency.
3. Users should only ever see posts they are allowed to see. Deleted posts vanish within seconds. Followers-only and friends-only posts appear only for viewers in that audience. Authors the viewer has blocked or muted, and authors who blocked the viewer, never appear.
4. Users should be able to keep scrolling a live result set without seeing the same post twice or skipping one, while new posts keep arriving.

Non-functional requirements

• Ingest. About 500M new posts a day: 5.8k a second on average and around 20k a second for an hour during big events. Twitter reported its tweet rate rising 3–4× during big events; one-second bursts go higher still.
• Freshness. A new post is searchable in about 1 second typically and within 5 seconds at p99. Deletes and audience changes apply within seconds; an unfriend or a block applies at once.
• Corpus. About 600 billion searchable posts over more than ten years, growing by roughly 180 billion a year at today’s rate.
• Query load. 35k searches a second on average (about 3 billion a day) and 100k a second at peak.
• Latency. Server-side p50 under 100 ms and p99 under 400 ms.
• Signals. About 5 billion likes, reposts and replies a day, visible to ranking within seconds.
• Availability. 99.95%. Missing one partition’s posts for a moment is better than an error page.

Out of scope

• People and account search, and suggestions while typing — see design typeahead.
• Computing trending topics — see design top-K trending.
• Storing posts and building timelines — see design Twitter and design a news feed.
• Ads, and the full machine-learned ranking pipeline. We use one light model at the root and treat its training as a black box.
• Spam and abuse classification. We consume a spam score per post and per author.

The set-up

Plan the approach

There are three paths at three speeds: the write path turns a new post into postings within a second, the signal path turns millions of likes into updated counters within seconds, and the read path answers a query in a few hundred milliseconds. Build each requirement simply, then spend the deep dives on what is specific to posts: real-time indexing, partitioning by time, ranking with changing signals, visibility, hot events and pagination.

Core entities

• Post. Post ID (a time-sortable 64-bit ID from an ID generator), author ID, text, language, media flags, audience (public, followers, friends, a custom list) and state (live, edited, deleted). The post store is the source of truth.
• Index document. The search view of a post inside one partition: a local document ID, its postings, and a few fixed-width columns — post ID, author ID, audience, language, flags, a static quality score and engagement counters.
• Term. A normalised token: a word (diwali), a hashtag (#diwali), a mention, a link’s domain, or a filter term such as author:8841, lang:hi or has:media.
• Engagement counters. Likes, reposts, replies and quotes per post, updated in place.
• Viewer context. Who is searching: the accounts they follow, their friends, the authors they blocked or muted, and the authors who blocked them, plus language and country.
• Cursor. An opaque token: for Latest, the last post ID shown; for Top, a pointer into a saved ranked list.

API

Search. One read endpoint; the client never sees partitions or tiers.

GET /v1/search/posts?q=%23diwali%20sale%20lang%3Ahi&sort=top&limit=20

{
  "results": [
    {
      "post_id": "108636363041587203",
      "author": { "id": "8841", "handle": "rangoli_rani", "verified": false },
      "text": "Diwali sale haul! #diwali",
      "created_at": "2026-10-27T18:42:07.315Z",
      "counts": { "likes": 312, "reposts": 41, "replies": 17 },
      "media": [{ "type": "photo", "url": "https://media.example.com/p/9f2c.jpg" }]
    }
  ],
  "next_cursor": "c2Vzc2lvbl83YTk6MjA",
  "newest_id": "108636363041587203",
  "partial": false
}

For Latest, the next page is max_id (everything older than the last post shown) and the “new posts” banner polls with since_id:

GET /v1/search/posts?q=ind+vs+aus&sort=latest&limit=20&max_id=108636363041587202
GET /v1/search/posts?q=ind+vs+aus&sort=latest&since_id=108636545963573248&count_only=true

partial: true means a partition missed its deadline and the page was built from the rest.

Partition search, the internal call from a root to one replica of each partition (gRPC in production):

{
  "partition": 7,
  "query": { "must": ["#diwali", "sale"], "must_not": [], "filters": ["lang:hi"] },
  "sort": "top",
  "window": 2000,
  "k": 50,
  "max_doc": null,
  "viewer": {
    "id": "5512",
    "follows_bloom": "base64, about 240 bytes",
    "friends_bloom": "base64, about 160 bytes",
    "hidden_authors": ["9001", "4477"]
  },
  "deadline_ms": 120
}

Ingest events on Kafka, one topic for post changes and one for engagement:

{ "type": "post.created", "post_id": "108636363041587203", "author_id": "8841",
  "created_ms": 1793126527315, "audience": "public", "lang": "hi",
  "text": "Diwali sale haul! #diwali", "has_media": true }
{ "type": "post.deleted", "post_id": "108636363041587203", "at_ms": 1793127001210 }
{ "type": "engagement.delta", "post_id": "108636363041587203",
  "likes": 37, "reposts": 4, "window_end_ms": 1793127005000 }

Back-of-envelope estimates

These numbers decide the partition count, the replica count and what lives in RAM. (Method: back-of-envelope estimation.)

Writes. 500M posts ÷ 86,400 s ≈ 5.8k posts/s; at about 3.5× during a big event, 20k posts/s.

Signals. Assume 10 likes, reposts or replies per post: 5B a day ÷ 86,400 ≈ 58k events/s, about 200k/s at peak.

Reads. 35k searches/s × 86,400 ≈ 3B a day; 100k/s at peak. A result cache with 30% hits leaves 70k/s for the index.

Real-time tier. Keep 7 days in RAM: 7 × 500M = 3.5B posts. A post yields about 30 text terms plus 10 filter terms, so 40 postings. At about 5 bytes per posting (a mix of write-friendly and compacted segments), that is 200 B; fixed columns add 48 B (post and author IDs 16 B, four counters 16 B, audience, language, flags, quality score and a 4-byte duplicate signature 16 B); dictionary and allocator slack about 50 B. Total ≈ 300 B per post, and 3.5B × 300 B ≈ 1.05 TB per copy. As a sanity check, Earlybird measured a full write-friendly segment at about 420 B per tweet (6.7 GB for 16 million), shrinking by about 55% once compacted — around 190 B — so 300 B for longer posts is in range.

Partitions. Put about 52 GB of index on a 128 GB server, leaving room for per-query memory, garbage collection and the OS: 1.05 TB ÷ 52.5 GB = 20 partitions of 175M posts.

Ingest per partition. 20k ÷ 20 = 1,000 posts/s at peak, far below the 7,000 tweets/s Earlybird indexed on one 2011 server while it also served queries at full CPU.

Replicas. Earlybird served about 5,000 queries/s over 144 million tweets on an 8-core server, with p99 at 170 ms. Assume our 32-core servers, with longer posts and heavier scoring, do 3,000 queries/s at about 50% CPU. Then 70k ÷ 3,000 ≈ 24 copies of each partition must serve at once. Three regions sized to survive losing one need 12 copies each (two regions × 12 × 3,000 = 72k): 36 copies per partition, 720 servers, each at about 70,000 ÷ 36 ≈ 1,950 queries/s normally.

Fan-out. 70k queries/s × 20 partitions = 1.4M partition calls/s.

Viewer filters. A Bloom filter at 1% false positives needs about 9.6 bits per member. 200 followed accounts × 9.6 bits ≈ 240 B, 130 friends ≈ 160 B, plus 20 hidden authors × 8 B = 160 B: ≈ 0.6 KB per call, so 1.4M × 0.56 KB ≈ 0.8 GB/s of filter bytes across the fleet. Sending the 350 raw 8-byte IDs instead would be 2.8 KB per call, about 3.9 GB/s.

Archive. Compressed postings at about 1.5 B each and 40 B of columns make about 100 B per post: 600B × 100 B = 60 TB per copy. At 3 TB per SSD server, 20 servers per copy; two copies per region, 120 servers. If 5% of index queries reach the archive (3.5k/s), each copy sees about 580 queries/s.

Engagement columns. 3.5B × 16 B = 56 GB in total, 2.8 GB per partition — small enough to sit next to the postings.

Latency budget.

High-level design

Build it one requirement at a time; the result works for one region at moderate traffic, and the deep dives make it fast, fresh and safe at full scale.

Users should be able to search posts by keywords, hashtags and filters

Two paths meet at the index. On the write path, the post service commits a post to the post store and publishes a post.created event to Kafka. An indexer consumes the stream, detects the language, tokenises the text and turns everything searchable into terms: words, #hashtags (kept whole and also split, so #IndVsAus matches “ind vs aus”), @mentions, the domain of each link, and filter terms such as author:8841, lang:hi and has:media. It hashes the post ID to pick one of 20 partitions and publishes the document to the Kafka partition that feeds it. Every replica of that index partition consumes the same stream and appends the post to its newest segment.

On the read path, the search API parses #diwali sale lang:hi -giveaway into a boolean query. A root sends it to one replica of each partition, each partition returns its best matches, and the root merges them and hydrates the winners from the post store.

Three choices keep this simple and fast:

• Partition by post ID, not by term. A hot hashtag spreads evenly over all 20 partitions instead of melting one, and each post is written to exactly one place. The cost is that every query visits every partition — fine at 20, and the web-search sibling explains why the same choice holds even at 1,000.
• Filters are just terms. from:rangoli_rani becomes the term author:8841, and has:media is a term too, so a filter is one more posting list to intersect. Time ranges are cheaper still: document IDs follow creation time (more on this below), so since: and until: become a range of document IDs to skip to.
• The index stores IDs, not posts. Partitions return post IDs and scores; text, author names and media come from the post store (behind a cache) for the 20 winners only.

Users should be able to choose Latest or Top

Twitter’s Earlybird paper calls this the dominance of the temporal signal: for real-time search, the timestamp is often the most important signal, so postings should be traversed newest-first — the opposite of a web index sorted by document ID or quality. If postings are stored in time order and read from the newest end, Latest needs no sorting at all: walk backwards, keep matches that pass the filters, stop after 20. Twitter’s 2020 write-up calls this early termination critical to serving very high request rates, and Elasticsearch offers it as index sorting: sort segments by timestamp, and a top-N-by-time query stops after N documents per segment if you do not ask for the total hit count.

Top cannot stop at the first 20, because the 21st-newest post might have 10,000 likes. But it does not need every match either. Each partition scores a bounded window — the newest 2,000 matches — plus every match in a small side index of posts that have already proved popular, and returns its best 50.

The root merges what the partitions return: for Latest, a 20-way merge by post ID; for Top, the best 50 from each, re-scored with a light model. How the signals get into the index, and why the side index exists, is the third deep dive.

Users should only ever see posts they are allowed to see

Every document carries its audience as a term (aud:public, aud:followers, aud:friends, aud:list) and its author ID as a column. The root loads the viewer’s context once per query — Bloom filters of the accounts they follow and their friends, plus the short list of authors hidden from them — and sends it with the query. Inside the partition, a document matches only if it is public, or its audience term agrees with the viewer’s filters, and its author is not hidden. Deleted posts are skipped through a per-segment tombstone bitmap, set within about a second of the post.deleted event.

That filter is fast but approximate: a Bloom filter has false positives, and the viewer context may be a few seconds old. So the root runs one more, exact check on the finalists against the authoritative visibility service, just before hydration.

Hydration is the last safety net: if the post store says a post was deleted a moment ago, it is dropped even though the tombstone has not reached the partition yet. The page comes back one short, which nobody notices; a deleted post on screen is something everybody notices.

Users should be able to keep scrolling a live result set

Offset paging — “skip 20, take 20” — breaks the moment new posts arrive: five new matches push five posts from page 1 onto page 2, and the user sees them twice. A deletion does the opposite and silently skips one. It is also expensive for deep pages, because every partition must produce offset + limit results; Elasticsearch caps from + size at 10,000 hits by default for exactly this reason.

For Latest, use the post ID as a keyset cursor. Post IDs sort by creation time, so “page 2” means “the 20 newest matches with ID below 108636363041587202”. New posts get bigger IDs and cannot disturb older pages; they wait above, counted by a since_id poll that powers a “5 new posts” banner.

Each partition turns max_id into a document-ID bound and starts its backwards walk just below it, so page 50 costs the same as page 1. Top is harder, because scores change while you scroll; the last deep dive deals with it. The general pattern is in offset vs keyset pagination.

You now have a working post search: posts indexed from a stream, a fan-out query with Latest and Top, per-viewer filtering with a final check, and stable cursors. The deep dives are about making each part hold up at 20,000 posts and 100,000 queries a second.

Deep dives

How do we make a post searchable within a second?

A post index must take in thousands of documents a second and answer tens of thousands of queries a second, on the same data at the same time — which, as Earlybird’s authors note, rules out the build-and-swap approach of web search.

Bad: rebuild in batches. Re-index every few minutes and swap. It is simple and compact, but a post from the first over stays invisible until the next build, and each build costs as much as the corpus, not the change.

Good: a general-purpose engine with near-real-time refresh. Elasticsearch buffers new documents and, on each refresh, writes them into a new small segment that searches can open without a full commit to disk. By default that happens every second, on indices searched in the last 30 seconds. For most products that is enough, and you should stop there. At 20,000 posts a second with spikes, you pay for it: a new segment per shard per second, constant background merging, and every update to a document — including a like count — implemented as delete plus re-insert. Merges, garbage and queries compete for the same CPU at exactly the moment a final over drives both writes and reads up.

Great: an index built for append-only, time-ordered documents. This is Twitter’s Earlybird design, built on Lucene, and the core ideas carry over to any large post search:

• Segments that fill and then freeze. Each partition keeps a series of segments. Only the newest is writable; when it fills, it is converted in the background into a compact read-only form. Earlybird used 12 segments of 2²³ ≈ 8.4 million tweets each, and the compacted form took about 55% less memory.
• One writer, many readers, no locks. A single thread indexes into the active segment; query threads read it concurrently. The writer finishes all postings for a post and then advances one volatile counter (maxDoc), which acts as a memory barrier. Readers read the counter first and ignore anything beyond it, so they never see half a post.
• Postings as plain integers, in time order. In Earlybird’s writable segment each posting was a 32-bit integer: 24 bits of document ID and 8 bits of position — enough for a 140-character tweet. Lists grow through pools of slices of 2, 16, 128 and 2,048 integers, so a growing list is never copied and leaves no garbage for the JVM to collect.

The paper reported tweets searchable within about 10 seconds and queries answering in about 50 ms on average. The design Twitter walkthrough draws this structure in detail; what matters here is what Twitter did next, because it is the difference between 10–15 seconds and 1.

Until mid-2019 Twitter’s indexing latency was around 15 seconds, and two waits caused it. First, some fields are not ready when a post is created — a fully resolved URL is a far better signal than a shortened link, but resolving it takes time — and the old pipeline indexed all fields at once, so every post waited for its slowest field. Second, the index assumed posts arrived in creation order, so the pipeline held every post in a buffer for a few seconds and sorted it.

The fixes are worth copying:

1. Index now, patch later. Send the post to the index the moment it exists with the fields most queries need, and send a second, partial update when the slow fields arrive. Very new posts briefly lack a resolved URL; few queries care.
2. Derive the document ID from the creation time. Lucene document IDs are positive 31-bit integers. Twitter gave 27 bits to the millisecond within the segment — enough for a bit more than 37 hours, against segments that are writable for about 12 — and 4 bits to a counter for posts in the same millisecond. A post that arrives late still gets the ID its creation time deserves.
3. Insert, don’t append. A prepend-only list cannot take a post in the middle, so Twitter replaced it with skip lists in which each entry is promoted to the next level with 20% probability. A late post is inserted in its place in O(log n) time, and a saved search finger makes the next seek in a conjunction cost O(log d), where d is the distance skipped.
4. Publish documents, not postings. A searcher takes a snapshot of a published pointer into each allocation pool and ignores anything allocated after it. Without that, “dogs -hot” could match a post whose dogs posting is written but whose hot posting is not.

Skip lists made long lists slower to scan but conjunctions and phrases faster, leaving average query speed nearly unchanged. With the waits gone, Twitter reported posts searchable within one second. The rollout is a lesson too: the new pipeline ran beside the old one, a staging cluster took dark reads of production queries, and the servers refused to return anything younger than 15 seconds until every client was confirmed not to depend on the delay.

Two more pieces make the write path robust:

• Kafka is the replication log. Each of the 20 index partitions has its own Kafka partition. All 36 replicas consume it independently, so replicas need no protocol of their own, and the single writer per replica gets posts in a stable order. (See design a distributed message queue.)
• Recovery is a snapshot plus a replay. Every 12 hours, when a segment freezes, write it to blob storage with the Kafka offset it covers. A replacement replica loads the frozen segments and replays the topic from that offset — minutes, not a rebuild. Facebook’s Unicorn does the same with a twist: each index server remembers the timestamp of the last update per category, and the tailer resends everything after it.

How do we partition hundreds of billions of posts?

The real-time tier holds a week in RAM. The other 600 billion posts are 60 TB even compressed, and they keep growing by billions a week. How should they be split?

Bad: partition by term. One partition owns #indvsaus, another owns sale. A query touches only the partitions of its terms, but during a match one partition receives most of the traffic while the others idle, every new post must be written to about 40 partitions, and intersecting two lists means shipping one across the network. The web-search sibling makes the full case; for posts, hot terms make it worse.

Good: hash every post across one flat set of partitions. Load is even and each post lives in one place. But a ten-year index that only grows makes this painful in two ways. Every query, even “Latest”, touches partitions full of posts from 2016 that it will never return. And when the corpus outgrows the cluster, adding hash partitions means reshuffling data that is already serving, which Twitter’s 2014 write-up describes as significant operational work.

Great: time tiers, hash partitions inside each tier. This is how Twitter built its complete index of about half a trillion tweets with an average latency under 100 ms. Split the corpus first by time into tiers, then hash each tier into partitions, group a partition’s segments onto servers, and replicate servers for capacity. The real-time tier, about a week, stays in RAM; the full index, over 100 times larger, sits on SSDs, which cost far less per terabyte than RAM.

The root hides all this behind one endpoint. Twitter’s roots run a two-level scatter-gather — across tiers, then across partitions — and skip tiers that cannot contain results for the query’s time range. For Latest, query the tiers in order: the real-time tier answers almost every query alone, and only a rare term (“#chandrayaan3 landing replay”) falls through to the archive. For Top, query the real-time tier and the popular side index, and add archive tiers only for explicit old ranges or too few results. A daily batch job builds the archive tiers; Twitter’s shared preprocessing code with the real-time path and ran over a thousand single-threaded index builders in parallel, rebuilding nearly half a trillion tweets in about two days.

Within a tier, replicas buy throughput and survival. Twenty partitions keep the fan-out modest — nothing like web search’s thousand — but hedge any partition call still pending at the 95th percentile, and return partial: true rather than wait for a dead replica. Unicorn chose to shard by result ID for exactly this reason: when a machine dies, returning some of the results beats returning none.

How do we rank “Top” when likes change every second?

Top is where post search differs most from web search. Text match still matters, but so do age and popularity — and popularity moves: a post can go from 3 likes to 30,000 in an hour.

Bad: rank every match by engagement. Say 2% of posts mention #diwali during Diwali week: 70M matches, 3.5M per partition. Scoring all of them blows the 150 ms partition budget, and ranking purely by likes pins last Tuesday’s viral post to the top for good.

Good: score a bounded, recent window with a cheap formula. Earlybird’s ranking used three kinds of signals, and they are still the right split:

• Static, computed at indexing time: author reputation and spam score, language, media, links, text quality.
• Resonance, which change after indexing: likes, reposts, replies and quotes.
• Searcher, sent with the query: whether the viewer follows the author, and the viewer’s languages. Twitter compressed the useful part of each searcher’s graph into a Bloom filter for exactly this test.

Each partition walks the newest 2,000 matches and computes something like text × quality × log10(1 + engagement) × 2^(−age ÷ 6 h), boosted when the viewer follows the author. The logarithm stops 20,000 likes from counting 66 times as much as 300; the half-life decides how fast fame fades.

The half-life is a product decision, and Earlybird’s authors left it open: is a “perfect” tweet from five minutes ago better than a “good” one from five seconds ago? A sensible default is a shorter half-life for queries spiking right now (match day) and a longer one for slow topics (a recipe hashtag).

Keep fast-changing counts out of the immutable index. In a Lucene-style index, updating a field means deleting the document and inserting it again; at 200,000 engagement events a second that is a storm of re-inserts into segments meant to be frozen. Instead, store counters as fixed-width columns indexed by document ID, beside the postings, and overwrite them in place. Twitter built the same idea: an updater pushes resonance signals to Earlybird, which was extended to take updates to arbitrary annotations without locks or slowing searches. A stream job first sums events per post over 5-second windows, so 2,000 likes in a window cost one update. If about 100,000 distinct posts receive engagement in a peak window, that is 20k updates a second, 1k per partition.

The counts in the index only need to be good enough to rank. The numbers on screen come from the counter service during hydration, so a stale column never shows anyone a wrong like count.

Great: two candidate sources, two phases, and a final pass at the root.

• A popular side index. A recent window misses the post from four days ago that everyone is still sharing. When the aggregator sees a post cross a threshold, say 100 likes, it adds the post to a small side index of proven posts — about 1% of the tier — which Top reads in full. Twitter’s first historical index worked in a similar spirit: about two billion top tweets, chosen by a score.
• A light model at the root. Each partition returns its best 50 with a compact feature vector. The root merges 20 × 50 = 1,000 and re-scores them with a gradient-boosted model using features too costly per match: author–viewer affinity, engagement velocity (likes in the last 10 minutes), media quality. At about 20 µs per candidate, 1,000 candidates cost 20 ms of CPU, split across cores.
• Diversity and duplicates. Cap posts per author per page and collapse near-copies, which multiply during big events. Twitter computed MinHash signatures compressed to 4 bytes per tweet, enough to catch most duplicates.

How do we make sure nobody sees a post they should not?

A privacy failure is the worst bug search can have: a friends-only post about a diagnosis shown to a colleague, or a harasser finding the person who blocked them. And privacy state changes constantly.

Bad: index only public posts, or filter in the app. Public-only search is safe but cannot find a friend’s friends-only post. Filtering on the device means private posts leave the server — a leak by design.

Good: keep privacy out of the index and check every result. Unicorn, Facebook’s graph search engine, holds no privacy information at all. Each result carries its lineage — the edges that produced it — and the caller, Facebook’s PHP frontend, asks the authoritative privacy service whether enough of them are visible. The paper’s reasons: after an unfriend, friends-only content must vanish immediately (even a 30-second lag was unacceptable), keeping a real-time index that strict was too costly, and the privacy logic already lived in one place. You gain exactness and one source of truth. You pay in wasted work and short pages: if 1 candidate in 25 is visible to this viewer, a page of 20 needs 500 candidates.

Great: filter cheaply in the index and exactly at the end. The high-level design already does this; here is what each layer is for.

Three details make the layers hold up:

• Author-level changes never touch postings. When an account with 80,000 posts goes protected, is suspended or deactivates, one row changes in a small author-state table that Kafka replicates to every partition, and matches consult it. Re-indexing 80,000 documents would take minutes and churn segments.
• Blocks work in both directions. Hide authors the viewer blocked or muted, and authors who blocked the viewer. Muted words become a query-time must_not.
• Deletes are tombstones first, purges later. A delete sets a bit in the segment’s tombstone bitmap within about a second; the posting itself disappears when the segment is compacted or rebuilt. Legal takedowns also purge result caches by post ID.

How do we survive a cricket final?

In the last over of an India–Australia final, a huge share of all searches are the same few queries. Assume 40% of a 100k/s peak — 40,000 searches a second — are variants of “ind vs aus”. The postings cope: hash partitioning spreads #indvsaus over all 20 partitions, and Latest reads only the newest few. What hurts is the fan-out: 40,000 × 20 = 800,000 partition calls a second for one question whose answer changes a few times a second.

Bad: add servers when the spike starts. A new replica must load 52 GB and replay Kafka, which takes minutes; the over is done by then.

Good: a short-TTL result cache in front of the roots. Cache the merged candidates for 1 second for Latest and a few seconds for Top. Nobody can tell a 1-second-old result from a fresh one, and each key now costs one computation per second instead of thousands.

Great: normalise, coalesce, personalise after the cache, and shed load in a planned order.

• Normalise the key. Fold case and spacing, treat “v” and “vs” alike, and map #INDvAUS to its tokens, so dozens of spellings collapse into about 10 keys.
• Coalesce misses. When an entry expires, one request per key per region recomputes it while the others wait a few milliseconds or take the stale copy. Route keys to cache nodes by consistent hashing so each hot key has one owner. With 10 keys refreshed once a second, the index sees 200 partition calls a second per region, 600 across three, instead of 800,000. The general pattern is in hot keys and cache stampedes.
• Personalise after the cache. Cache the top 200 public candidates. Per viewer, drop hidden authors and apply the follow boost in about a millisecond, and fetch followers-only posts from followed accounts with a small, highly selective separate query.
• Scale before, not during. A final is on the calendar. Add replicas the day before, and warm the caches with the queries you expect.
• Shed load in a planned order when you are still over budget: archive tiers first, then expensive shapes (long OR lists, deep pages), then shrink Top’s window from 2,000 to 500, then serve Top as Latest, and finally serve stale cache entries. See backpressure and load shedding.

Unicorn handled a different kind of heat the same way: its friend: and likers: terms were especially hot, so Facebook gave just those term families a small, highly replicated tier — and could then reduce replication elsewhere.

How do we page through “Top” when scores keep changing?

max_id works for Latest because the order is time, and time does not change. Top is ordered by score, and scores change with every like. A keyset cursor on (score, post_id) breaks: the last post on page 1 scored 1.47; by the time you ask for page 2 it scores 1.9 and some page-1 posts have fallen below it, so you get repeats and misses.

Bad: re-run the query with an offset. Repeats, gaps, and a cost that grows with depth.

Good: freeze the order for the session. On the first Top request, the root stores the ranked list of the best 400 post IDs in a cache under a search-session ID for 10 minutes, and the cursor is (session, position). Later pages are slices of the frozen list, hydrated with current counts and re-checked for visibility, so a deleted post still drops out. The cost is small: 400 × 8 B = 3.2 KB per session. If 20% of peak searches go to page 2, that is 20k new sessions a second × 600 s = 12M sessions × 3.2 KB ≈ 38 GB across the cache cluster. Elasticsearch offers the same idea as a point in time used with search_after and a unique tiebreaker, because otherwise a refresh between pages can reorder results.

Great: freeze, then continue by time. When the frozen 400 run out, switch to a time-ordered walk below the oldest post in the snapshot, skipping IDs already shown. Someone on result 401 wants more, not a slightly different best.

The final design

Each region runs the full read path and its own replicas of every tier; Kafka streams and the daily batch feed all three regions.

1. Publish. The post service commits the post and publishes post.created (or an edit or delete) to Kafka: 5.8k events a second, 20k at peak.
2. Index. The indexer tokenises the post, adds filter and audience terms, and sends it to its partition’s stream. Each of the 36 replicas inserts it by its time-derived ID and publishes it, about a second after posting. Slow fields follow as a partial update; deletes set a tombstone bit.
3. Signals. Likes and reposts are summed per post over 5-second windows and written into counter columns in place; posts that cross 100 likes join the popular side index.
4. History. A daily batch builds compressed segments for the SSD archive tiers.
5. Query. The search API parses and normalises the query, and answers hot public queries from a 1-second result cache with coalesced misses.
6. Fan out. On a miss, a root loads the viewer’s filters and queries one replica of each real-time partition, plus the popular side index for Top and archive tiers only when needed. Partitions walk newest-first and answer within 120 ms; slow calls are hedged.
7. Check. The root merges, re-ranks with a light model, removes duplicates, and sends about 60 finalists to the visibility service for an exact check.
8. Hydrate. The root loads the 20 winners from the post cache, drops anything just deleted, and returns them with a cursor: max_id for Latest, a frozen-list position for Top.

Check it against the requirements. Freshness: about 1 second, with no sort buffer and no wait for slow fields. Latency: about 75 ms typical and about 330 ms at p99 against a 400 ms target. Throughput: 70k index queries a second over 720 real-time servers at about 1,950 queries a second each, and 1,000 posts a second per partition at peak. Scale: 1 TB per copy of the last week in RAM, 60 TB per copy of history on SSD, growing by adding tiers. Privacy: cheap filtering where data lives, an exact check before anything is shown. Spikes: the hot question costs the index 600 calls a second instead of 800,000.

What interviewers look for by level

Follow-up questions

What if users want to search only their friends’ posts?

Restrict by author: add author ∈ friends(viewer) to the query. With about 130 friends on average (Unicorn’s figure), an OR of 130 author: terms intersected with the keywords is small and selective. Unicorn generalises this as the apply operator: run an inner query (friend:5), turn its results into terms, and run the outer query, with the inner results typically capped around 5,000. Friends-of-friends shows why you compute such sets at query time: each user has about 48,000 on average, and precomputing those lists for a billion users would need over 178 TB, against 484 GB for plain friend lists.

What if we need semantic search?

“Mumbai rain” should find “barish ne phir se local band kar di”. Embed each post at ingest into a vector of, say, 256 one-byte dimensions and keep an approximate nearest-neighbour index beside each real-time partition: 3.5B × 256 B ≈ 900 GB, about the size of the whole lexical tier, so many teams embed only posts that pass a quality bar. The root merges vector and lexical candidates before re-ranking, and keyword matching stays for exact tokens such as hashtags and handles. See embeddings and semantic search.

What if a post is edited after it went viral?

Keep the post ID, so counters, cursors and caches stay valid. Index the new text as a new document and tombstone the old one. Most edits happen soon after posting, while the post’s segment is still writable, so the new document goes where its creation time puts it. A rare edit to an older post goes into a small mutable side segment per partition, searched alongside the others, until the next compaction folds it in. Hydration always shows the latest version.

What if a country requires some posts to be hidden there?

Treat it as visibility. Add a term such as withheld:IN to affected posts, and -withheld:<viewer country> to every query from that country, with the country derived server-side. Orders arrive after posting, so the term comes as a partial update over Kafka; the exact check at the root consults the same rule list, and caches are keyed by country for affected queries. Log every suppression for audit.

What if we must support 20 languages?

Language identification on short posts is hard — the Earlybird paper notes how little text a tweet offers and how often people switch languages mid-sentence — so do not trust one label. Index a language-neutral field (Unicode normalisation, case folding, script-aware tokenising) plus a stemmed field for confident detections. Chinese and Japanese have no spaces: index character bigrams, at the cost of more postings, or use a dictionary segmenter. For Indic scripts, normalise variant spellings rather than stem, and expand Hinglish queries with transliterations, as the web search design describes.

How to say it in the interview

“I’ll design keyword search over a social network’s posts: about 500 million posts a day, 20 thousand a second during big events, searchable within a second, and up to 100 thousand searches a second with a p99 under 400 milliseconds.

The write path is a stream. Posts go through Kafka to an indexer that routes each by hash of post ID to one of 20 partitions of an in-memory index holding the last week — about a terabyte per copy. Each replica has one writer and many lock-free readers, and document IDs come from creation time, so late posts slot into place without a sort buffer.

Older posts live in time tiers on SSD, hash-partitioned within each tier. A root asks the newest tier first and only goes back when it must, so growth just adds a tier.

Latest walks postings newest-first and stops after 20 hits. Top scores the newest 2,000 matches per partition plus a small index of popular posts, using static signals, recency decay and like counts updated in place from 5-second aggregates; the root re-ranks a thousand candidates with a light model.

Visibility has three layers: audience terms and Bloom filters of the viewer’s graph inside partitions, an exact check on about 60 finalists, and a final check at hydration, so an unfriend or a delete takes effect immediately.

For a cricket final, I normalise queries, cache public results for a second with coalesced misses, personalise after the cache, pre-scale, and shed load in a planned order. Latest pages with max_id cursors; Top pages through a frozen ranked list.”

Concepts used in this design

• Design a search engine like Google — posting lists, compression, scatter-gather and tail latency, which this design reuses.
• How search engines work — the inverted index and query evaluation in general.
• Search data structures — inverted indexes, skip lists and other structures behind fast seeks.
• Elasticsearch — near-real-time refresh, index sorting and search_after, and where they stop scaling.
• Kafka — the post and engagement streams, and the log that keeps replicas in step.
• Design a unique ID generator — time-sortable post IDs for cursors and time tiers.
• Design Twitter — the posting pipeline and Earlybird’s segment internals.
• Database sharding — hash partitions inside each time tier.
• Data replication — 36 replicas per partition for throughput and survival.
• Bloom filters — compact viewer filters sent with every partition call.
• Consistent hashing — giving each hot cache key a single owner.
• Hot keys, cache stampede and invalidation — coalescing misses on the hottest queries.
• Backpressure and load shedding — the planned order for degrading under overload.
• Offset vs keyset pagination — why max_id cursors beat page numbers.
• Strong vs eventual consistency — a fast, slightly stale index guarded by exact checks.
• Embeddings and semantic search — adding meaning-based retrieval beside keywords.

References and further reading

1. Earlybird: Real-Time Search at Twitter (Busch et al., ICDE 2012)notes.stephenholiday.comRequirements of real-time search, newest-first postings, 12 segments of 2^23 tweets, 32-bit postings, one writer with a volatile maxDoc, static and resonance signals, 10 s indexing and 50 ms queries, and per-server benchmarks.
2. Reducing search indexing latency to one second (Twitter Engineering, 2020, archived)web.archive.orgFrom about 15 s to 1 s: time-derived 31-bit doc IDs, skip lists instead of prepend-only lists, a published pointer for document atomicity, and indexing slow fields as a later update.
3. Building a complete Tweet index (Twitter Engineering, 2014, archived)web.archive.orgAbout half a trillion Tweets on SSD, time tiers times hash partitions, roots that skip irrelevant tiers, a daily batch pipeline, and growth by adding tiers and replicas.
4. The engineering behind Twitter’s new search experience (Twitter Engineering, 2011, archived)web.archive.org2,200 Tweets/s and 18,000 queries/s, 3–4× spikes during big events, an updater for engagement signals, 4-byte MinHash signatures for duplicates and a Bloom filter of the searcher’s graph.
5. Unicorn: A System for Searching the Social Graph (Curtiss et al., VLDB 2013)vldb.orgFacebook’s in-memory graph search: sharding by result ID, mutable layers over immutable ones, the apply operator, WeakAnd, and privacy checked by the caller from per-result lineage.
6. Near real-time search (Elasticsearch documentation)elastic.coRefresh opens new segments without a full commit; by default every second, and only on indices searched in the last 30 seconds.
7. Index sorting settings and early termination (Elasticsearch documentation)elastic.coSorting segments by timestamp so a top-N-by-time query can stop after N documents per segment when total hits are not tracked.
8. Paginate search results (Elasticsearch documentation)elastic.coThe 10,000-hit limit on from/size, search_after with a unique tiebreaker, and point-in-time views so refreshes do not reshuffle pages.