First ask: what if two workers both hold this lock?
The answer
A duplicate SMS? One Redis node will do. A merchant paid twice? No lock can stop that alone; the database must check.
200k/s
renewals if every lock had its own TTL
A million held locks renewed every 5 s: four times the 50k budget. So locks hang off one lease per process: 20,000 ÷ 5 s = 4k keepalives.
Worker A froze for 20 s, then deleted B’s lock
Its plain DEL looked reasonable; now B and C both hold it. Release only if the value is still your token, in one script.
Where do locks live?
Bad: Primary + async replica
Acks your lock, crashes before copying it; the promoted replica grants it again.
Good: Consensus group
Granted only once 3 of 5 replicas have it. No failover forgets it.
A paused holder wakes up and writes. Only storage can stop it.
Every grant carries a token that only rises. Storage remembers the highest it has accepted, 34, and refuses 33.
Stop on your own clock, before the leader frees the lock
Last good renewal sent at 5 s, 15 s lease, 3 s margin: stop at 17 s. The leader frees it at 20.
A hot lock admits ≈ 38 holders a second. Design it away:
- Batch: take it once, allocate 200 bookings
- Use the database’s own row lock or conditional update
- One writer per key: partition the stream by merchant
Remember
Ask what two holders would cost. For correctness, use a consensus store, session leases and fencing tokens. A lease means “probably alone”; only the database, refusing stale tokens, makes it certain.