1B
distinct items a day, almost all tiny
Exact counters for one 24-hour window would need about 125 GB, nearly all of it for items that will never rank.
A 76 MB grid counts a billion items
Hash an item to one cell per row; add 1. Its estimate is the smallest of those cells. Collisions only add, so it never under-counts.
Three servers send their top 2. The real #1 never arrives.
C is third everywhere, so nobody sends it, yet its 300 beats A’s 200. Fix it by routing each item’s events to one task.
600k
increments a second, all on one task
During an IPL final #iplfinal might get 200k events/s × 3 scopes, 24× a task’s budget. Sum per second in 200 upstream tasks: 600 messages/s.
#love: 900 uses in five minutes, usually 800. #raininmumbai: 120, usually 1.5. Which is trending?
Top isn’t trending
#raininmumbai, at 80× its usual against 1.1×. Score share × ln(share ÷ baseline), and it scores about 5× higher.
Slide the last hour with one add and one subtract
Keep 60 minute buckets per active item. Each minute, add the newest and subtract the one that fell out. Constant work, however busy.
Reads never touch the counters
- Precompute all 3,450 lists: 138 MB in total
- Swap versions with a pointer flip
- CDN caches 15 s: 75k reads/s become 1k
- Pipeline stalled? Serve the last good list
Remember
Gate the long tail with a small summary, key by item, pre-sum hot keys, score trends against a baseline, recount exactly nightly.