6M
jobs assumed due at 09:00:00 IST
At a dispatch budget of 100k a second, that’s 60 s of backlog. A seat-hold release due at 09:00:00 may wait at the back.
What’s due right now?
Bad: Scan the table each second
fire_at <= now() reads billions of rows to fire a few thousand.
Good: Bucket timers by minute
Read each bucket once, minutes ahead: ≈ 680 timers, 68 KB.
A job due next March needs no attention until next March
Timers wait on disk in minute buckets. Five minutes ahead, each bucket is read once into Redis; every 100 ms, what’s due fires.
Two scheduler nodes briefly both own partition 417. Both pop the same due job.
The trick
Firing is one conditional write, SCHEDULED → QUEUED only at the timer’s version. Exactly one node sees “1 row updated”.
A worker debits ₹499, then dies before it acks
Its lease expires and attempt 2 runs, at-least-once. Both calls carry key K, so the payments API just returns txn 91.
Back to 09:00: precise jobs get their own lane
Drained first, 50k precise jobs take half a second. 5.95M flexible jobs spread over their 10-minute window, ≈ 9.9k a second.
New York, 8 March 2026: 02:30 never happens
- Store the zone,
America/New_York, never an offset - Compute in that zone, fire in UTC
- A skipped run fires right after the gap
- A repeated hour runs once
Remember
Separate deciding when from doing the work. Bucketed timers, preloaded into Redis, fire once by conditional write. Leased workers run each job at least once, with one idempotency key.