5.5 min
to push one tweet to 100M followers
Pretend all of 2012’s 300k deliveries a second went to that one tweet. A million followers take 3.5 s; the target is 5 s.
Skip giants when writing. Merge them in when you read.
Fan-out skips accounts above about a million followers. Your read is one Redis range plus a few cached giants, merged by tweet ID.
Who hands out 143k tweet IDs a second?
Bad: One database counter
One master issues every ID and chokes when traffic spikes.
Good: Snowflake IDs
Time, machine and sequence in 64 bits. No coordination; IDs sort by time.
Thousands of tweets a second, searchable in 10 s, while queries hit the same index.
The trick
Partition by tweet. Write one segment with one thread, and read postings backwards, newest first, no locks.
“#GoodMorning” has more tweets. “#IPLFinal” is the trend.
Trends score velocity, not volume, how far a term sits above its own baseline. #IPLFinal scores 357; #GoodMorning 1.1.
143,199
tweets in one second, 3 August 2013
About 25× the 5,700-a-second average, during a TV airing of Castle in the Sky. Users saw no blip.
Split “now” from “soon”
- Validate, ID, store, ack: sized for 25×
- Fan-out, search, trends and counts queue up and drain
- Partition stores by hash, not by time
- Uniform back-pressure, and switches to shed work
Remember
Snowflake IDs sort by time. Push IDs to cached timelines, merge giants on read, index by tweet, trend on velocity, and queue everything but posting.