3B
notifications a day, for 300M users
1B transactional that must land in seconds, 2B from campaigns that can take minutes. Transactional alone peaks near 50k a second.
Where does your OTP wait?
Bad: One SMS queue
Behind 20M sale SMS at 10k/s: 2,000 s. It expires in 5 minutes.
Good: Its own lane
sms.p0: own workers, reserved SMS capacity. It waits milliseconds.
5.7×
FCM’s default allowance, for one campaign
100M users in 30 minutes is 3.4M FCM sends a minute; the default is 600k. Every order update shares the same allowance.
100M pushes: all at once, or paced?
All at once ends ten minutes sooner by spending the quota order updates need. Paced at 55k/s, it ends at minute 31.
Expand 100M users at once and a cancel must purge 100M queued messages.
The trick
Release 10k-user chunks only as fast as you send. Cancel stops the releases: about 50k users in flight.
FCM said 200 OK, then the worker crashed. Send again?
Nobody recorded it, so exactly-once to a phone is impossible. Each type decides. An OTP is never resent; a receipt retries with a collapse ID.
No loss, rare duplicates
- Producer: outbox row in the same transaction
- API: idempotency key, same ID on retry
- Worker: dedupe key in Redis before sending
- Retries end at the type’s TTL
Remember
Accept durably, decide once per user, send through lanes by priority and channel. Priority comes from the type, with its own workers and quota, so an OTP never waits behind a campaign.