Signals that it’s a long-running job
- Work measured in seconds to hours
- Timeouts on the path: 29 s, 60 s
- Nobody needs to watch a spinner
- The work must outlive a deploy or crash
Raise the timeout?
Bad: A longer timeout
A 29 s or 60 s limit still lurks, and retries double the work.
Good: 202 Accepted + status URL
Accept in milliseconds; a retried submit returns the same job.
Climb the ladder. The queue is only rung 2.
Change the contract first, then how the work runs. Most of the interview is about dead workers, huge jobs and one tenant flooding the queue.
Lease for 2 minutes, heartbeat every 30 seconds
A crashed worker stops heartbeating, so its job reappears within one lease. Every write carries the attempt number, so a woken zombie can’t overwrite the retry.
86
worker slots for 0.5 exports a second
Little’s law: 0.5 jobs a second × 120 s = 60 running at once, or 86 slots at 70% utilisation. Then check the database: 860,000 rows a second.
The 10-minute export crashes at 90%, and the retry starts from zero.
The fix
Checkpoint every million rows, or split into 100,000-row chunks. A crash then costs one interval or one chunk, not the whole job.
Split the giant job into chunks
50 chunks of 100,000 invoices on 20 workers take 3 waves of 10 s, about 30 s instead of 500. A counter reaching zero triggers the merge.
Say it like this
Accept with 202 and a status URL. Run on leased, idempotent workers that heartbeat, retry and dead-letter. Checkpoint long work, push progress, and hand over a signed URL.