Long-running jobs

Long-running jobs is the playbook for any work that takes longer than a request should: transcoding a video, exporting millions of rows to a CSV, generating 20 lakh exam-result PDFs, sending a campaign to 2 million users, judging a code submission. The answer has two halves. The contract with the client changes from “wait for the answer” to “here is a ticket, check back”. The execution moves to workers that survive crashes and deploys, retry safely, report progress and share capacity fairly. Weak answers stop at “put it on a queue”.

Picture it first: the dry cleaner in wedding season

Two weeks before a family wedding you drop a sherwani and six sarees at the dry cleaner. Nobody expects you to stand at the counter until they’re done.

1. The counter hands you a token slip and says “come back Thursday”.
2. Your order goes in the register, so it survives the shop closing for the night.
3. In the back, whoever is free takes the next bundle. If a worker goes home sick, someone else picks the bundle up.
4. You can phone and ask how it’s going, or they SMS you when it’s ready, and you collect with the slip.
5. A hotel’s order of 400 tablecloths is split across workers, and if the steam press breaks halfway, the finished ones aren’t washed again.
6. In wedding season, an express counter and a rule that one hotel can’t take every press keep ordinary customers moving.

Map it back: the slip is 202 Accepted with a status URL; the register is a durable job table and queue; whoever is free is a worker pool with leases and retries; phoning and the SMS are polling, push and notifications; the tablecloths are chunking and checkpoints; the express counter and the one-hotel rule are priorities and per-tenant fairness.

Spotting the pattern

Listen for these cues:

• Work measured in seconds to hours: video transcoding, a CSV export over millions of rows, PDF generation, bulk imports, batch ML inference, a campaign to 2 million users, code judging, data backfills.
• A timeout on the path. An API Gateway REST integration stops at 29 seconds, an Application Load Balancer closes a connection idle for 60 seconds by default, a Lambda function runs at most 15 minutes, and mobile networks drop long requests far sooner.
• Nobody needs to watch. The user wants the result, not a spinner; an email or a bell icon will do.
• The work must outlive a server: a deploy, a crash or a scale-in must not lose it.
• Bursty demand: deadline days, campaign launches and month-end, where a queue smooths the peak.

It does not apply when:

• The work fits in the request, under a second or two at p99. A queue only adds latency and moving parts.
• The user can’t continue without the answer, as with a payment authorisation or a seat hold. Make it faster instead.
• The result is a stream, such as an LLM answer token by token: that is real-time updates.
• It is a chain of steps with waits and undo (reserve, pay, ship): that is a multi-step workflow.

The running example is a GST report export on an Indian marketplace’s seller dashboard. On the evening before the monthly filing deadline, 1,800 sellers an hour (0.5 a second) request their invoice report. Most reports take seconds and the average is 2 minutes. The largest seller has 5 million invoices: a 2 GB CSV that takes about 10 minutes, since reading 5,000,000 rows at 10,000 a second takes 500 seconds before compression and upload. Targets: no request waits more than a second, no report is lost or generated twice, and a seller who closes the tab still gets the file.

Level 0: do it in the request

The handler queries the invoices, writes the CSV and streams it back. For a seller with 2,000 invoices that takes under a second, and it is the right design. For the seller with 5 million it fails three ways:

• Timeouts. API Gateway gives up at 29 seconds and the load balancer at 60 seconds of silence. A 10-minute export never finishes, and the work done so far is thrown away.
• Retries double the work. The client sees a 504 and retries, so two exports run, then three. On deadline night, retries multiply load exactly when the database is busiest.
• Deploys kill it. An afternoon deploy ends whatever is in flight, with no record that it ever started.

You gain simplicity: one request, one response, nothing new to run. You pay with a thread and a connection held for the whole job. It breaks at the first timeout on the path: 29 seconds behind API Gateway. Say: “Anything that can exceed a few seconds at p99 leaves the request path.”

Level 1: accept now, answer later

Change the contract. The submit endpoint validates the request, records a job and answers at once:

POST /v1/reports/gst-export HTTP/1.1
Idempotency-Key: 5f1c9a7e-48812-2026-10
Content-Type: application/json

{"seller_id": "48812", "period": "2026-10"}

HTTP/1.1 202 Accepted
Location: /v1/jobs/j_81
Retry-After: 10

RFC 9110 calls 202 intentionally noncommittal: the request was accepted for processing, not completed, and the response should point to a status monitor. Azure’s Asynchronous Request-Reply pattern fills in the rest:

• Validate before accepting. A bad period gets a 400 now, not a failed job later.
• Location names a status resource; Retry-After paces polling. The status body carries the state (pending, running, succeeded, failed or cancelled), timestamps, a percent complete and a structured error.
• Finish with 303 See Other pointing at the result. A 303 makes the client fetch it with GET; after a 302, some clients replay the original POST.
• DELETE on the status resource cancels, and results get a retention period.
• An Idempotency-Key on submit. The client can’t tell a lost response from a lost request, so it retries; a unique index on (seller, key) returns the existing job instead of starting a second.

Polling is cheap if you pace it. With 30,000 jobs open on deadline night, polling every 5 seconds is 30,000 ÷ 5 = 6,000 requests a second; Retry-After: 30 cuts it to 1,000, each a primary-key read of one small row.

You gain a fast, honest API: timeouts stop mattering and retries are safe. You pay for a job table, a status endpoint, polling traffic and a client that must poll. It breaks when the work still runs inside the API process: a deploy kills it, nothing retries it, and a burst of exports eats the API’s CPU. Say: “Submit returns 202 with a job URL in milliseconds, and an idempotency key makes a retried submit return the same job.”

Level 2: a durable queue and a worker pool

Separate accepting from doing. The API writes the job row and enqueues a message, ideally through a transactional outbox so the two can’t disagree; a separate pool of workers takes messages from a message queue. The job row is the source of truth (status, attempt, lease, progress, result, error); the queue is only the doorbell. A PostgreSQL table claimed with SKIP LOCKED can play both roles at modest scale.

Leases, not locks. In SQS a received message becomes invisible for the visibility timeout, 30 seconds by default; if the worker hasn’t deleted it by then, it reappears for someone else. A 10-minute export can’t live on a 30-second lease, and a 12-hour lease would strand a crashed worker’s job for 12 hours. So take a short lease, say 2 minutes, and heartbeat: extend it every 30 seconds with ChangeMessageVisibility, as the SQS docs recommend. A dead worker’s job is back within 2 minutes. SQS caps visibility at 12 hours from the first receipt, however often you extend it; anything longer must be chunked.

At-least-once means idempotent workers. SQS warns that a standard queue can deliver a message twice even within the timeout, and a paused worker can wake up after its lease has expired. So:

• Write output to a key derived from the job, such as exports/j_81.csv.gz, so a second run overwrites instead of duplicating.
• Make state changes conditional, for example UPDATE jobs SET status = 'succeeded' WHERE id = 'j_81' AND attempt = 3. The attempt number acts as a fencing token, so a zombie from attempt 2 can’t overwrite attempt 3 (idempotency).
• Send the “report ready” email only on the transition into succeeded.

Retries, then a dead letter. Retry transient errors such as a database timeout after 30 seconds, 2 minutes and 8 minutes with jitter; fail permanent ones, such as a malformed period, at once. SQS’s redrive policy moves a message to a dead-letter queue after maxReceiveCount receives, so a poison job that crashes every worker stops circling. Alert on the dead-letter queue’s depth, fix the cause, then redrive.

Size the pool with Little’s law: jobs in progress = arrival rate × average duration = 0.5 a second × 120 seconds = 60 running jobs. At a 70% target utilisation that is 60 ÷ 0.7 ≈ 86 worker slots. Then check what the workers lean on: 86 exports reading 10,000 rows a second each is 860,000 rows a second against the read replica. The downstream, not the worker count, often sets the real limit.

You gain work that survives deploys and crashes, automatic retries, and a pool sized and scaled apart from the API. You pay for a queue, a job table, idempotent workers and dead-letter operations. It breaks when users can’t tell what is happening and the 2 GB file still streams through your API. Say: “Workers lease a job for 2 minutes and heartbeat every 30 seconds, so a crashed worker’s job reappears within 2 minutes; five failures send it to a dead-letter queue.”

Level 3: tell the user

A seller staring at “running” for 10 minutes will submit again or call support. Three additions fix that.

Progress on the job row. The worker records rows done and percent complete every 100,000 rows: 50 small writes for the biggest export instead of 5 million. The status endpoint just reads the row.

Push when someone is listening, notify when nobody is. An open dashboard tab can subscribe to the job’s status over Server-Sent Events (real-time updates), with polling as the fallback. A seller who closed the tab gets an email or in-app notification with the link. API customers get a webhook such as job.succeeded, signed and retried, so they stop polling altogether.

Results live in object storage. The worker uploads the file to blob storage, and the status resource hands out a pre-signed URL valid for, say, 24 hours. The 2 GB download then goes straight from the store, instead of tying up an API server and its bandwidth for minutes. Delete results after 7 days with a lifecycle rule, and say so in the status response, since Azure’s pattern recommends a stated retention window.

You gain users who aren’t watching a spinner, fewer polls, and downloads that never touch your servers. You pay for a push channel, a notifier, webhook retries and signed-URL plumbing. It breaks when a 10-minute job crashes at 90% and starts again from zero. Say: “Progress goes on the job row every 100,000 rows; an open tab gets it over SSE, and the seller gets an email with a 24-hour signed link.”

Level 4: survive long work

Leases and retries make a job eventually finish, but a retry starts from the beginning. For a 10-minute job that hurts; for a 6-hour backfill it may never finish. Two tools bound the loss.

Checkpoints. Read invoices in ID order with keyset pagination (WHERE invoice_id > :last ORDER BY invoice_id LIMIT 10000), write a part file every million rows (about every 2 minutes with compression and upload), and record the last invoice ID and the finished parts on the job row. A retry reads the checkpoint and carries on. Temporal builds the same idea into heartbeats: a heartbeat can carry progress, and the next attempt after a missed heartbeat receives it. Also pin the data to the moment the job started, with an as_of timestamp or a database snapshot, so a resumed export doesn’t mix old and new invoices; a GST report’s totals must match the filing.

Chunks. Some jobs are better split than resumed. The coordinator cuts the 5 million invoices into 50 chunks of 100,000 by ID range, each its own queue message. One worker would take 50 × 10 s = 500 seconds; 20 workers take 3 waves × 10 s = 30 seconds, plus the merge. A counter of remaining chunks on the job row, decremented atomically as each finishes, triggers a finaliser that stitches the part files and checks the row count. A failed chunk retries alone.

Two cautions. Parallelism multiplies load on the source: 20 workers at 10,000 rows a second is 200,000 rows a second from one seller’s data, so cap the fan-out per job and per tenant. And decide what partial failure means. A GST report missing one chunk is worse than no report, so the job fails and retries the chunk; a campaign to 2 million users reasonably finishes “with errors” and a list of the 1,200 numbers that bounced.

You gain a bounded loss on any crash, minutes instead of hours, and retries at chunk size. You pay for checkpoint state, a coordinator, a fan-in counter, a merge step and more load on the source. It breaks when one tenant’s huge backfill floods the shared queue and everyone else waits. Say: “The export checkpoints every million rows, so a crash costs at most two minutes; really big jobs fan out into 100,000-row chunks with a fan-in counter.”

Level 5: operate at scale

On deadline night the pool is shared by thousands of sellers, nightly backfills and scheduled jobs. Running it well takes six habits.

• Priorities. Put interactive jobs (a seller is waiting) and bulk jobs (backfills, re-exports) on separate queues, and have workers poll them in a weighted ratio, say 4 to 1, so bulk work still moves.
• Fairness per tenant. One seller submitting 2,000 re-exports must not push everyone else’s report back by an hour. Cap concurrent jobs per tenant (say 3) and schedule fairly (noisy neighbours and fairness). SQS fair queues do this from a tenant ID in MessageGroupId: when one tenant has far more messages in flight than the rest, SQS hands out other tenants’ messages first, yet still lets the busy tenant use spare capacity.
• Timeouts and cancellation. Give every job type a maximum runtime, say 3 × its p99, after which it fails instead of running forever. A DELETE marks the job “cancelling”, and the worker checks between chunks. Temporal delivers cancellation with heartbeats, so an activity that never heartbeats can’t be cancelled.
• Scheduled jobs, once. If three scheduler replicas all fire the 00:00 run, insert it with a unique key of job name plus scheduled time so exactly one row wins (design a job scheduler).
• Autoscale on backlog, not CPU. An export worker waiting on the database shows low CPU while the queue grows. Scale on queue age or backlog: 600 queued jobs × 120 s = 72,000 worker-seconds, and draining them in 5 minutes needs 72,000 ÷ 300 = 240 slots, capped by what the replica can serve (autoscaling done right).
• Durable buffers and backpressure. Slack’s job queue runs over 1.4 billion jobs a day at a peak of 33,000 a second. Its original Redis-based queue failed in a database slowdown: enqueues outran dequeues until Redis ran out of memory, and then it couldn’t even dequeue, because moving a job to the in-flight list needed memory too. Slack put a durable Kafka log in front. Reject submits with 429 once the backlog passes what you can drain, rather than accepting work you will never finish (backpressure and load shedding).

Cost is the last lever. Checkpointed, idempotent workers can run on cheap interruptible capacity (spot or preemptible VMs), because losing one costs a lease timeout and a checkpoint interval; keep a small on-demand floor for interactive jobs.

You gain predictable waits under load, no tenant starving the rest, and a bill that follows demand. You pay for multiple queues, per-tenant counters, scheduler deduplication and autoscaling policy. It breaks when a job is really several steps that wait on people or outside systems, or must be undone. Say: “Interactive and bulk jobs get separate queues, each seller is capped at three running jobs, and the pool scales on the age of the oldest job.”

Level 6: graduate to a workflow engine

Sometimes the job grows into a process: export the report, wait up to two days for the seller to approve it, file it with the tax portal, retry the portal for hours, email a receipt, and reverse the filing if payment fails. Hand-rolled job chains end up rebuilding state, timers, per-step retries and compensation, badly.

A workflow engine persists every step. In Temporal, each step is an activity run by a worker. Temporal recommends always setting a Start-To-Close timeout, because the server can’t otherwise tell that a worker crashed; long activities add a heartbeat timeout, and a heartbeat can carry progress for the next attempt. The SQS docs point to AWS Step Functions once processing needs more than 12 hours. The full lesson is multi-step workflows, and the engine itself is in workflow engines.

You gain durable state, timers and retries for every step, with history you can inspect. You pay for an engine cluster or service, deterministic workflow code and a new programming model. Say: “A single export is a job; export, approve, file and refund is a workflow, so it goes on Temporal.”

Comparing the rungs

Which classic problems use it

Common mistakes

• Raising the timeout instead of changing the contract. Somewhere on the path a 29-second or 60-second limit still waits.
• A background thread in the API server. The 202 is correct, but a deploy silently kills the work.
• No idempotency key on submit. Every retried POST starts another 10-minute export.
• A lease shorter than the job and no heartbeat. The job reappears mid-run, two workers process it and the seller gets two emails.
• No attempt limit. A poison job retries forever and quietly eats workers; cap attempts and use a dead-letter queue.
• A progress write per row: 5 million updates to say “40%”.
• Streaming the 2 GB result through the API instead of handing out a signed URL.
• One FIFO queue for every tenant, so one big backfill delays everyone’s reports.
• Autoscaling on CPU while the backlog grows behind a slow database.

Cheat sheet

• Jobs in progress = arrival rate × average duration (Little’s law); slots = that ÷ target utilisation.
• Poll rate = open jobs ÷ poll interval.
• Heartbeat several times per lease; a dead worker’s job returns within one lease.
• Work lost per crash ≤ one checkpoint interval (plus the lease).
• Fan-out time ≈ ⌈chunks ÷ workers⌉ × chunk time + merge.
• Slots to drain a backlog = backlog × average duration ÷ target drain time.

How to say it in the interview

“A big GST export takes about 10 minutes, past API Gateway’s 29 seconds, so submit validates, stores a job row and returns 202 with a status URL in milliseconds; an idempotency key makes a retried submit return the same job. A queue wakes workers, which lease a job for 2 minutes and heartbeat every 30 seconds, so a crashed worker’s job reappears quickly. Output is keyed by job ID, every write carries the attempt number, and five failures send a job to a dead-letter queue. The worker checkpoints every million rows and uploads the file to S3; the client polls with Retry-After or gets SSE updates, and the seller gets an email with a 24-hour signed link. I size the pool with Little’s law, scale on queue age and cap each seller at three running jobs.”

What if the client retries the submit? The Idempotency-Key header hits a unique index on (seller, key), so the second POST gets the same 202 and job URL instead of a second export.

A worker dies at 90%. What happens? It stops heartbeating, the lease expires within 2 minutes, and another worker takes the job as attempt 2. It resumes from the last checkpoint, at most a million rows back; the job-keyed output and conditional writes make any overlap harmless.

How many workers do you need? Little’s law: 0.5 jobs a second × 120 seconds = 60 running at once, so about 86 slots at 70% utilisation. Then check the database, since 86 readers at 10,000 rows a second is 860,000 rows a second.

How does the user find out it’s done? The status URL says so, polled at the pace of Retry-After or pushed over SSE while the tab is open; otherwise an email or webhook. The result is a signed URL to object storage, valid for 24 hours.

When would you use a workflow engine instead? When the job becomes steps that wait on people or other systems, need timers of hours or days, or must be undone on failure. A single export stays a job.

Where to go next

• Message queues: the doorbell between the API and the workers.
• Delivery guarantees: at-least-once, exactly-once and DLQs: why workers must be idempotent.
• Idempotency: keys on submit and safe reruns.
• Timeouts, retries and backoff with jitter: retrying without a stampede.
• Real-time updates: pushing job progress to an open tab.
• Large file uploads: the inbound twin of signed-URL downloads.
• Multi-step workflows: when a job becomes a process.
• Serverless architecture: workers that fit inside 15 minutes.
• Design a job scheduler: this pattern as a full design problem.

References and further reading

1. Azure Architecture Center: Asynchronous Request-Reply patternlearn.microsoft.com202 with Location and Retry-After, a status resource with status and percent complete, 303 to the result, DELETE to cancel, an Idempotency-Key on submit, and a retention window for results.
2. RFC 9110: HTTP Semantics (202 Accepted, 303 See Other, Retry-After)rfc-editor.org202 as an intentionally noncommittal acceptance that points to a status monitor, 303 for an indirect response fetched with GET, and Retry-After as seconds or a date.
3. Amazon API Gateway quotasdocs.aws.amazon.comREST API integration timeout of 50 ms to 29 seconds for every integration type; Regional and private APIs can raise it at the cost of throttle quota.
4. Elastic Load Balancing: Application Load Balancer attributesdocs.aws.amazon.comThe connection idle timeout, 60 seconds by default; send data before it expires on lengthy operations.
5. AWS Lambda quotasdocs.aws.amazon.comA maximum function timeout of 900 seconds (15 minutes).
6. Amazon SQS visibility timeoutdocs.aws.amazon.comA 30-second default, extending it with ChangeMessageVisibility as a heartbeat, the 12-hour maximum from first receipt, possible duplicate delivery, and about 120,000 in-flight messages per standard queue.
7. Amazon SQS dead-letter queuesdocs.aws.amazon.commaxReceiveCount in the redrive policy, messages received 3 or more times moved to the back of the queue, longer DLQ retention, and redrive after a fix.
8. Amazon SQS fair queuesdocs.aws.amazon.comA tenant ID in MessageGroupId lets SQS favour quiet tenants when one tenant builds a backlog, without capping anyone’s throughput.
9. Temporal documentation: Detecting Activity failuresdocs.temporal.ioAlways set a Start-To-Close timeout to catch crashed workers, heartbeat timeouts for long activities, heartbeat payloads that let a retry resume, and cancellation delivered on heartbeat.
10. Scaling Slack’s Job Queue (Slack Engineering)slack.engineering1.4 billion jobs a day at a peak of 33,000 a second, jobs from milliseconds to minutes, and a Redis queue that ran out of memory and then could not even dequeue.