Design a job scheduler

A job scheduler is the alarm clock of a backend. Teams hand it work to do later — “send this EMI reminder at 09:00 IST on the due date”, “release these unpaid seat holds in 15 minutes”, “retry this webhook in 2 minutes”, “build every merchant’s settlement report at 02:00” — and it promises to start each one on time, on some worker, even if machines die in between. On one server that is cron and a table. It becomes a real design problem when a billion jobs a day share one platform, when half the company picks 09:00:00 as its favourite second, when a worker dies halfway through a payout, and when a customer in New York schedules something for 02:30 on the one night that time does not exist. In this problem you design a shared, multi-tenant scheduler that fires one-off, delayed and recurring jobs within a second of their time, runs each at least once, and never loses one.

Understand the problem

Two words in the prompt hide most of the design: when and run. “When” sounds simple until you ask how late is too late. A seat hold released a minute late blocks a sale during a tatkal rush; a settlement report ten minutes late bothers nobody. “Run” sounds simple until a worker dies after charging a card but before saying so. Pin both down before drawing boxes, and say early that this is a scheduler of time, not of machines.

Functional requirements

1. Tenants should be able to schedule one-off and delayed jobs — at an absolute time (“09:00 IST on 5 November”) or after a delay (“in 15 minutes”), up to a year ahead — and update or cancel them until they start.
2. Due jobs should run at least once on a worker, either as an HTTP callback or as a task pulled by the tenant’s own workers, with timeouts, retries with backoff, priorities, and a dead-letter queue for jobs that keep failing.
3. Tenants should be able to define recurring jobs with a cron expression and a time zone, pause and resume them, and choose what happens to runs missed during downtime and to runs that would overlap.
4. Tenants should be able to see each job’s status and history — the current state, every attempt with its timing and error — and alert on lateness and failures.

Non-functional requirements

• Scale. About 1 billion jobs scheduled a day across a few thousand tenants, roughly 5 billion waiting at any moment, and about 10 million cron schedules.
• Spiky triggers. On average about 12k jobs fall due each second, but the top of the hour — 09:00:00 IST above all — brings peaks of around 100k triggers a second, with millions of jobs due in the same second.
• Timing. Precise jobs start within 1 second of their time at p99. Flexible jobs declare a window (“any time in the 10 minutes after 09:00”) and start inside it.
• Range. Delays from zero to one year; executions from 100 ms to 1 hour; payloads up to 64 KB, with anything bigger kept in blob storage and passed by reference.
• Durability. Once the API answers 201, the job is never lost — not to a crash, a failover or a full queue.
• Availability. 99.99% for both scheduling and firing: 525,600 minutes a year × 0.0001 ≈ 53 minutes of downtime a year.
• At-least-once execution. Duplicates are allowed but rare, and every attempt carries the same idempotency key.
• Isolation. One tenant’s 50 million jobs must not delay another tenant’s seat-hold releases.

Out of scope

• Workflow orchestration. Graphs of dependent steps, fan-in, human approvals and long sagas belong to workflow engines such as Temporal or Airflow. Our scheduler can start a workflow run; it does not run the graph. A follow-up covers this.
• Placing work on machines. Packing jobs onto servers by CPU, memory or GPUs is cluster scheduling, the job of Kubernetes or Borg. We decide when; they decide where.
• Sandboxing tenant code. Tenants run their own worker pools or expose HTTP endpoints; we do not build a container platform.
• Sub-second precision. Trading timers and game ticks need timers inside the process that owns the state, not a shared service.
• Billing for job runs.

The set-up

Plan the approach

Build a simple, working scheduler first — an API, a durable store, something that notices due jobs, a queue and workers — then spend most of the time in the deep dives, where this problem is won. State the core idea early: the scheduler only moves jobs onto queues at the right moment; workers, owned and scaled by tenants, do the work.

Core entities

• Job. One unit of work to run once: job_id, tenant, queue, fire time in UTC, timing class (precise, or flexible with a window), target (an HTTP callback or a task queue), payload, retry policy, timeout, status, attempt number, and a version that changes on every update.
• Timer. A tiny index entry meaning “look at job X, version v, at time t”, stored by partition and minute bucket. Firing, retries and safety checks are all timers.
• Schedule. A recurring definition: cron expression, IANA time zone (Asia/Kolkata, America/New_York), job template, missed-run policy, overlap policy, a paused flag and the next occurrence.
• Attempt. One execution: attempt number, worker, lease expiry, start and end times, outcome and error. Attempts make up the history.
• Queue and tenant settings. Timing lanes, fairness weights, concurrency caps and quotas.
• Partition lease. Which scheduler node owns which of the 1,024 timer partitions, under which assignment version.

API

Scheduling a payment reminder for 09:00 IST, allowed to go out any time in the following ten minutes:

POST /v1/jobs
Idempotency-Key: reminder-inv_8842-2026-11-05
Content-Type: application/json

{
  "queue": "payment-reminders",
  "run_at": "2026-11-05T09:00:00+05:30",
  "timing": { "class": "flexible", "window_s": 600 },
  "target": { "type": "http", "url": "https://reminders.internal/v1/send", "timeout_s": 30 },
  "payload": { "invoice_id": "inv_8842", "user_id": "u_1931" },
  "retry": { "max_attempts": 6, "initial_backoff_s": 10, "max_backoff_s": 600 }
}

{
  "job_id": "job_7Qk2",
  "status": "SCHEDULED",
  "run_at": "2026-11-05T03:30:00Z",
  "fire_at": "2026-11-05T03:34:12Z",
  "version": 1
}

The API converts the local time to UTC and, because the job is flexible, picks a fire time inside the window from a hash of the job ID — here 4 minutes 12 seconds after 09:00. A seat-hold release would instead send "delay_s": 900 with "class": "precise". The Idempotency-Key header makes a retried create return the same job instead of making a second one.

The rest of the job API:

PATCH  /v1/jobs/job_7Qk2               new time or payload; only while SCHEDULED, otherwise 409
DELETE /v1/jobs/job_7Qk2               cancel: 200 if it had not started, 202 if a running attempt must be asked to stop
GET    /v1/jobs/job_7Qk2               status, attempt number, next fire time
GET    /v1/jobs/job_7Qk2/attempts      every attempt with timings, worker and error
GET    /v1/jobs?queue=payment-reminders&status=DEAD&cursor=…   keyset-paginated listing
POST   /v1/queues/payment-reminders/dead-letters:redrive       run dead jobs again after a fix

A recurring job is a schedule plus a job template:

POST /v1/schedules
Content-Type: application/json

{
  "name": "merchant-settlement-report",
  "cron": "0 2 * * *",
  "time_zone": "Asia/Kolkata",
  "job_template": { "queue": "settlements", "target": { "type": "task" }, "payload": { "report": "daily" } },
  "missed_runs": { "policy": "run_latest", "starting_deadline_s": 3600 },
  "overlap": "forbid"
}

Workers that pull tasks use three calls; the shared HTTP executor uses the same ones internally on behalf of callback jobs:

POST /v1/queues/settlements:claim      max 10, lease 45 s → jobs with attempt, lease_token, idempotency_key
POST /v1/attempts/{token}:heartbeat    extends the lease; the reply says whether a cancel was requested
POST /v1/attempts/{token}:complete     outcome: succeeded | retryable_failure | fatal_failure, plus error

Every callback and task carries an attempt number and Idempotency-Key: job_7Qk2@2026-11-05T03:34:12Z — the job ID plus the occurrence’s fire time, the same on every retry of one run. Dropbox’s ATF asks handlers for the same three outcomes, because only the handler knows whether an error is worth retrying. For HTTP callbacks the status code says it: 2xx is success, 5xx and timeouts are retryable, 429 and 503 retry after their Retry-After, and other 4xx answers are fatal.

Back-of-envelope estimates

These numbers size the store, the scheduler fleet and the workers (method: back-of-envelope estimation).

Creates. 1B jobs a day ÷ 86,400 s ≈ 11.6k creates/s; × 3 for the daytime peak ≈ 35k/s.

Triggers. Each job fires once, plus a few percent of retries: about 11.6k fires/s on average. The 09:00:00 IST second is the problem. Assume 6M jobs fall due in it — reminders, statements, reports and cache warm-ups, of which about 50k are precise, such as sale openings and seat-hold releases. At a dispatch budget of 100k/s, 6M ÷ 100k = 60 s of backlog. That single division is why precise and flexible jobs cannot share one line.

For scale, Slack has reported over 1.4 billion jobs on its busiest days at peaks of 33,000 a second: large, but not exotic.

Waiting jobs. Little’s law says waiting = arrival rate × average wait. Assume an average wait of 5 days (seconds for most jobs, months for some reminders): 1B/day × 5 days = 5B waiting jobs. At about 1 KB each (metadata plus a typical 500-byte payload) that is 5 TB, 15 TB with three replicas. Timers are about 100 bytes: 5B × 100 B = 500 GB.

Buckets. Timers live in 1,024 partitions × one bucket per minute. An average minute brings 1B ÷ 1,440 ≈ 694k jobs, so one (partition, minute) bucket holds 694k ÷ 1,024 ≈ 680 timers, about 68 KB — a single read. The 09:00 bucket holds 6M ÷ 1,024 ≈ 5,860 timers, about 590 KB — still one paged read. A year ahead is 525,600 minutes, so up to 1,024 × 525,600 ≈ 538M buckets could exist; only non-empty ones take space.

Near-term tier. Five minutes ahead at the average rate is 11.6k/s × 300 s ≈ 3.5M timers; add 6M just before the hour and you get 9.5M entries × about 200 B ≈ 2 GB of Redis. Memory is not the problem.

Store writes. Per job: create (job row and timer) = 2, fire (a safety-net timer and a conditional write) = 2, result = 1, so 5 writes per job. On average 11.6k × 5 ≈ 58k writes/s. In the minute after 09:00: 100k fires/s × 2 + about 100k results/s + 35k creates/s × 2 ≈ 370k writes/s. With three replicas that is about 1.1M replica writes/s; at about 30k per node on NVMe, you need about 40 nodes of a wide-column store, each holding (5 TB + 0.5 TB) × 3 ÷ 40 ≈ 410 GB.

Scheduler nodes. A node that batches its conditional writes can fire about 5k jobs/s: 100k ÷ 5k = 20 nodes. Run 32, each owning 32 of the 1,024 partitions, so losing one node moves only about 3% of the timers.

Workers. Little’s law again: jobs in flight = rate × duration. Assume 90% are short callbacks of about 300 ms and 10% are tasks averaging 30 s: mean 0.9 × 0.3 + 0.1 × 30 ≈ 3.3 s. On average, 11.6k/s × 3.3 s ≈ 38k executions in flight. Callbacks alone are 11.6k × 0.9 × 0.3 ≈ 3.1k on average, plus 100k × 0.3 = 30k at 09:00. At about 2,000 concurrent I/O-bound calls per executor host, that is 33k ÷ 2,000 ≈ 17 hosts at peak; run 30. Task workers belong to tenants and autoscale on queue depth.

Lateness budget for a precise job. Dispatcher tick up to 100 ms + conditional write about 10 ms + queue push about 5 ms + worker pickup by long poll about 100 ms ≈ 0.2–0.3 s typical, and a p99 under 1 s while workers have free slots. Dropbox’s ATF, by contrast, polls its task store every two seconds and aims for 95% of tasks to start within five seconds; Airbnb’s Dynein targeted a p95 deviation under ten seconds. A one-second p99 needs an in-memory tier.

High-level design

Build the design one requirement at a time; by the end it works for one shard at moderate traffic.

Tenants should be able to schedule one-off and delayed jobs

The stateless scheduler API authenticates, checks quotas, deduplicates and normalises the time. Then it writes two records to the job store: the job row, keyed by job_id, and a timer in the bucket for its partition (a hash of the job ID, modulo 1,024) and its fire minute. Only when both writes are acknowledged by a quorum of replicas does it answer 201 — that is the durability promise.

Splitting job and timer makes changes cheap. Cancel sets CANCELLED and bumps the version; reschedule bumps the version and writes a new timer. The old timer names version 1 while the job is at version 2, so it is dropped when it fires. Every change is one or two writes, however far ahead the job sits — whereas Resque Scheduler, which Airbnb used before Dynein, had to search the entire backlog to dequeue a delayed job.

Jobs due right away skip timers and go straight onto a ready queue; Dynein relays anything due within 15 minutes straight to SQS.

Due jobs should run at least once on a worker

The simplest firing loop is a poller. Every second it reads timers whose time has passed, and for each one makes a conditional write on the job row: set status to QUEUED only if it is still SCHEDULED at the version the timer names. If the write succeeds, the poller pushes a small envelope — job ID, version, attempt, idempotency key — onto a ready queue.

A ready queue is a work queue with per-message leases, which Amazon SQS calls a visibility timeout: a received message stays in the queue, hidden from other consumers, until the worker deletes it; if it is not deleted in time, it becomes visible again. SQS defaults to 30 seconds, lets a consumer extend it while working, and caps the total at 12 hours. Keep one queue per tenant queue and priority — ATF keeps one per task type and priority — so one team’s backlog never sits in front of another’s.

There are two kinds of worker:

• HTTP callbacks run on a shared executor fleet that claims jobs, calls the tenant’s endpoint with a timeout and reports the result — “POST to this URL at this time”.
• Tasks are claimed by the tenant’s own worker pool, which runs any code for up to an hour. Dynein and ATF both give each service its own workers, so a bad deploy stalls only that team.

A failure — an error, a timeout or a lost lease — moves the job to RETRYING and writes a new timer at now plus a backoff. After the last allowed attempt the job is DEAD and lands in the queue’s dead-letter list for a human to inspect and re-drive. Long tasks follow the long-running jobs pattern: claim, heartbeat, report.

Tenants should be able to define recurring jobs

Do not expand a cron schedule into rows for the next month. An every-minute schedule would create 43,200 rows a month, and every edit, pause or time-zone rule change would have to rewrite them. Instead, store the schedule and keep exactly one next occurrence as an ordinary job with its timer. When that occurrence fires, the scheduler computes the following one from the cron expression, in the schedule’s time zone, and writes it in the same step.

Three rules keep this robust:

• Compute from the scheduled time, not the actual one. If Tuesday’s run fires 3 seconds late, Wednesday’s is still 02:00:00, so schedules never drift.
• Use a deterministic occurrence ID — the schedule ID plus the occurrence’s UTC instant. If two nodes both compute Wednesday’s run, the second insert is a no-op.
• Write the next occurrence before running this one. A run that fails, hangs or is cancelled never breaks the chain.

Store the zone name, Asia/Kolkata, never the offset +05:30: India has no daylight saving, but customers in New York or London do, and their offsets change twice a year.

Tenants should be able to see status and history

Every job moves through a small state machine, and the job row always holds its current state.

Each transition also emits an event — job, attempt, old and new state, timestamps, worker, error — to a log such as Kafka. A consumer writes the events to an append-only history store (30 days hot, then object storage), which serves GET /v1/jobs/{id}/attempts, and another feeds metrics. History never sits on the firing path. At 1B executions a day and about 500 bytes an event, it grows by 500 GB a day, 15 TB for the 30 hot days.

The metrics that matter:

• Lateness: actual start minus scheduled fire time, per tenant and timing class. This is the SLO.
• Backlog: due-but-unfired timers per partition, and queue depth and oldest-message age per ready queue.
• Dead jobs: DLQ size and growth per queue.
• End-to-end canaries: rolling out its new job queue, Slack enqueued a heartbeat canary on every Kafka partition every minute and alerted on its timing. Do the same with one canary job per timer partition per minute (see observability).

You now have a working scheduler: durable, and fine for one shard at moderate traffic. It breaks at the poller’s query, the single poller, retries in bulk and the top of the hour, and cron hides traps of its own.

Deep dives

How do we find due jobs without scanning everything?

With 11.6k fires a second on average and 100k at the peak, “what is due now?” is asked constantly against 5 billion waiting jobs. The answer must cost the same whether the store holds a thousand jobs or five billion.

Bad: one cron daemon, or a scan every tick. Classic cron wakes once a minute and checks every entry in its tables: one machine, one-minute resolution, no history, and nothing fires while that machine is down. Moving jobs into a table and running SELECT … WHERE fire_at <= now() every second without a suitable index is worse: each tick reads billions of rows to find a few thousand (see index scan vs full table scan).

Good: an indexed timer table, sharded, polled with SKIP LOCKED. Index live timers by fire time, shard the table by job ID (see sharding a relational database), and let several pollers per shard take batches without blocking each other:

-- Every 200 ms, each poller on its own shard
BEGIN;
SELECT job_id, version
  FROM timers
 WHERE fire_at <= now()
 ORDER BY fire_at
 LIMIT 500
 FOR UPDATE SKIP LOCKED;   -- rows another poller has locked are skipped, not waited for
-- for each row: conditional update of the job (SCHEDULED → QUEUED at that version),
-- push the winners to their ready queues, move the timer to now() + 5 min as a safety net
COMMIT;

PostgreSQL’s documentation describes SKIP LOCKED in exactly these terms: an inconsistent view, unsuitable for general work, but useful when many consumers read a queue-like table. You gain transactions and a design any team can run. You pay in writes and polling: several random writes and index churn per fire, dead rows for vacuum, queries even when nothing is due, and lateness no better than the poll interval. Airbnb hit this wall with Quartz on MySQL: each enqueue became 4 SELECTs and 3 INSERTs, and one r4.8xlarge database topped out at about 1,000 operations a second — 500 enqueues and 500 dequeues. At 100k fires a second you would need hundreds.

Great: time-bucketed timers plus a near-term tier. Two observations change the shape. A job due next March needs no attention until next March. And in its last few minutes, a job should be in memory, not on disk.

1. Durable timers in buckets. Timers are stored by (partition, minute bucket), sorted by fire time inside each bucket. The partition is a hash of the job ID, so even six million jobs for the same second spread evenly over 1,024 partitions. Dynein used the same trick in DynamoDB — a random ID as the partition key, the scheduled time as the sort key — so load stayed even and hot partitions never formed.
2. A loader on the node that owns a partition reads each minute bucket once, about five minutes ahead, and adds its timers to a Redis sorted set for that partition: ZADD due:{417} 1793849652000 job_7Qk2:1 (the score is the fire time in milliseconds). One sequential read of about 68 KB replaces hundreds of index probes.
3. A dispatcher on the same node asks the sorted set for members due now, every 100 ms. For each, it makes the conditional write — SCHEDULED to QUEUED at the timer’s version, the only disk access on the firing path (4) — and pushes the job onto its ready queue (5), removing the entry only after it is safely queued.

Two details complete it. A job due inside the loaded horizon — “retry this webhook in 2 minutes” — is written to the store and also ZADDed by the API; adding the same member twice is harmless, so it still fires once. And a past bucket holds only stale timers, so it expires as a unit, as in time-partitioned tables.

Why minutes? Second buckets mean 60 times as many reads of about 11 timers each; hour buckets hold about 41k timers and force the loader to read an hour ahead. And because Redis holds only the horizon, losing it loses nothing: the loader reloads from the store. Contrast Slack’s outage, when its Redis-based job queue hit its memory limit, stopped accepting jobs and — since dequeuing also needed a little free memory — could not drain them either.

Inside one process: timing wheels. The owner could instead keep its next few minutes in its own memory as a timing wheel, the structure Varghese and Lauck described in 1987: an array of slots, one per tick, each holding a list of timers, and a pointer that advances every tick. Starting, stopping and expiring a timer are all constant-time. For longer ranges, they stack wheels of different granularity, like the hands of a clock; their example covers 100 days at one-second resolution with 100 + 24 + 60 + 60 = 244 slots instead of 8.64 million.

For a year at one-second resolution: 365 × 86,400 ≈ 31.5 million slots flat, against 365 + 24 + 60 + 60 = 509 with four wheels. You gain firing with no network hop. You pay because the state lives in one process: a new owner must rebuild it after a failover — trivial at about 3,400 timers per partition — and every API insert must be routed to the owner. Redis sorted sets are the simpler shared choice here; wheels shine inside servers that track millions of connection timeouts.

How do we fire each job once but run it at least once?

Separate two promises. Triggering — moving a job from SCHEDULED to QUEUED — can happen exactly once per occurrence, because it is one conditional write on one row: set QUEUED where the job is still SCHEDULED at version v. If two dispatchers race on the same job, both send the write and exactly one sees “1 row updated”. Dynein does exactly this: a DynamoDB conditional update sets the job to “Acquired”, and only the winner dispatches it to SQS.

Execution cannot be exactly-once end to end. A worker charges a card and crashes before it reports success. From outside, “never ran” and “ran but didn’t say” look identical. The scheduler must either run it again (at-least-once, maybe twice) or never again (at-most-once, maybe zero times). For a payment reminder, a duplicate SMS is far cheaper than a missed one, so choose at-least-once and make the handler idempotent (see delivery guarantees).

Bad: fire and forget. The worker takes the job off the queue and deletes it immediately. If it crashes, the job is gone. Airbnb lists exactly this as a reason to leave Resque: it was an at-most-once system, so jobs were not guaranteed to be delivered.

Good: leases. The worker claims a job; the queue hides it for a lease period and the worker deletes it only when done. If the worker dies, the lease runs out and another worker takes attempt 2. Long jobs extend the lease as they go. You pay with a tuning problem: too short a lease and a slow but healthy worker loses its job to a second worker; too long and a crash delays the retry. And there is a subtle trap: count the attempt when the job is claimed, not when a failure is reported. A job that crashes its worker every time never reports anything, and would retry forever.

Great: leases, heartbeats, fencing and idempotency keys together.

• Heartbeats with self-fencing. Lease 45 s, heartbeat every 10 s, and a worker abandons its attempt after three failed heartbeats in a row. Dropbox’s ATF uses this rule, with a heartbeat timeout longer than those three, so a cut-off worker stops before its job is handed to someone else (see heartbeats).
• Fencing by attempt. Every claim returns a lease token holding the attempt number, which heartbeats and results must quote. The store accepts results only for the current attempt, so attempt 1 waking from a long garbage-collection pause cannot overwrite attempt 2 — the fencing-token idea from the distributed lock design:

UPDATE jobs
   SET status = 'SUCCEEDED', finished_at = now()
 WHERE job_id = 'job_7Qk2' AND attempt = 2 AND status = 'RUNNING';
-- 0 rows updated: a newer attempt owns the job, so this report is ignored

• One idempotency key per run. The handler, or the API it calls, records the key with its effect in one transaction and returns the stored result on a repeat, turning at-least-once execution into effectively-once effects (see idempotent consumers and the inbox pattern).
• A safety-net timer. Before its conditional write, the dispatcher writes a recheck timer a few minutes out. If the job is still QUEUED with no claim, or RUNNING past its timeout, when it fires — the queue lost the message, or a dispatcher died between write and push — the job is pushed again. ATF does the same with per-state deadlines for enqueue, claim and heartbeat.

How do we scale the scheduler without firing jobs twice?

Bad: one leader, or one global lock. A single elected leader that fires everything is simple, but 100k fires a second pass through one process, and every failover leaves a gap as long as the leader’s lease. The opposite — every node polls everything and a lock in the shared database decides who fires — is Quartz’s clustering: the first node to lock a trigger fires it. Its documentation warns that this cluster-wide lock degrades beyond about three nodes, and suggests splitting thousands of short jobs across separate schedulers instead.

Good: deterministic partition assignment. Split timers into 1,024 partitions and let each node compute its share from the list of live nodes. Dynein’s scheduler pods watch the Kubernetes ReplicaSet they belong to and derive their partitions from the pod names, so an autoscaler can add or remove pods freely. You gain: no coordinator. You pay during changes: nodes briefly disagree about membership, so two may load the same partition (wasted work, caught by the conditional write) or none may (late jobs until their views converge).

Great: partition leases with a coordinator. Keep ownership in ZooKeeper or etcd:

• Each node holds a session lease — 10 s TTL, renewed every 3 s — and stops dispatching by itself if it cannot renew for 5 s, before anyone else may take over.
• One elected coordinator writes the assignment map (partition → node, with a version). It fires nothing, so it is never the bottleneck.
• To move a partition, the old owner stops and releases it; the new owner claims it and reloads its five-minute window. The partition pauses for a second or two, and two live owners never overlap.
• When a node dies, its partitions move once its lease expires, so their jobs fire up to 10–15 s late — the price of never overlapping. Drain nodes gracefully on deploys, and keep partitions small, so a crash delays 3% of timers rather than a third.

Slack’s job queue uses the same pattern: each relay process holds a Consul lock on one Kafka topic, so every topic has exactly one relay.

Why 1,024 partitions for 32 nodes? Many more partitions than nodes means rebalancing moves small pieces, just as partitions do in a Kafka consumer group (see design a distributed message queue). You can grow to about 128 nodes before the pieces get coarse.

How do we retry failures without making them worse?

First, classify. Timeouts, 5xx answers, 429s and lost leases are worth retrying; a validation error or “invoice already paid” is not. ATF lets the handler mark a task succeeded, fatally failed or retriably failed, and warns that signalling this wrongly causes infinite retries.

Bad: retry at once, forever. A bank API goes down for a minute at 10:00 and 300,000 reminder jobs fail. Immediate retries hammer it at the worst moment, and “forever” turns every bug into poison jobs — ATF reported tasks stuck in infinite retry loops as a common source of manual work.

Good: exponential backoff with a cap and a limit. Wait 10 s, then 20, then 40, capped at 10 minutes, for at most 6 attempts. You pay for the determinism: all 300,000 jobs come back together, at 10, 30 and 70 seconds, as three spikes three times the platform’s peak dispatch rate.

Great: full jitter, plus limits that catch poison jobs.

• Full jitter: wait a random time between zero and min(cap, base × 2^(attempt − 1)). AWS’s simulations show backoff alone still leaves clusters of calls, while full jitter spreads them to a roughly constant rate and, with 100 contending clients, cut the number of calls by more than half. Honour Retry-After when the target sends one.
• Timeouts separate from leases. Each job has an execution timeout — 30 s by default for callbacks, up to an hour for tasks. The lease detects a dead worker; the timeout, a runaway one.
• A dead-letter list. After the last attempt — counted at claim — mark the job DEAD with its last error, alert its owner, and offer a re-drive once the bug is fixed.
• A breaker per target. If most calls to one endpoint fail, pause dispatch to it with a circuit breaker rather than burn attempts against it. Without these, retries become a retry storm.

Cancelling a running job. A cancel bumps the version and sets CANCEL_REQUESTED: no new attempts, no retries. A running task learns of it from its next heartbeat reply, stops and reports CANCELLED. An HTTP callback in flight cannot be recalled; if it succeeds, the job ends SUCCEEDED and the cancel was too late — hence DELETE answers 202 for a running job. Kubernetes draws the same line: suspending a CronJob does not affect Jobs already started.

How do we survive the top of the hour and stay fair between tenants?

At 09:00:00 IST, 6M jobs fall due in the same second. In one line at 100k/s the last starts a minute late, and a seat-hold release due at 09:00:00 may be near the back of that line.

Bad: fire everything as fast as possible. Dispatchers flood the ready queues, the store takes a write spike, and workers and the tenants’ downstream systems — a notification system, SMS gateways, bank APIs — get 6M requests at once. Precise jobs are exactly as late as the pile is big.

Good: preload and cap. The near-term tier holds the 09:00 buckets by 08:55, so firing runs at memory speed, through a global budget of 100k/s — about 98/s per partition, borrowing when a neighbour is idle — sized to what the store, queues and workers absorb. Nothing is lost or melted, but it is still first come, first served.

Great: lanes, windows and weighted fairness.

1. A precise lane, drained first. Each partition keeps precise and flexible timers in separate sorted sets, and the dispatcher empties the due precise set before touching the flexible one. 50k precise jobs at 100k/s take half a second.
2. Windows for flexible jobs. The API spreads each flexible job across its window when it is created, using a hash of the job ID. 5.95M reminders with a ten-minute window become 5.95M ÷ 600 s ≈ 9.9k/s.
3. Weighted fair queuing between tenants in the flexible lane. Each tenant has a sub-queue served by deficit round robin: every round, a tenant earns credit in proportion to its weight and spends it on due jobs; unused share flows to the others (see noisy neighbours and fairness).
4. Quotas at both ends. At create time, cap a tenant’s jobs per minute bucket, returning 429 or widening the window beyond it. At run time, cap each tenant’s concurrency on the shared executors. ATF gives each task type its own queues, workers and scheduling quota for the same reason.
5. Nudge people off :00. When someone creates a cron schedule, suggest a random minute — 17 2 * * * rather than 0 2 * * * — unless the hour really matters.

How do we get cron right across time zones, daylight saving and downtime?

Parse strictly and show the result. Cron has five fields — minute, hour, day of month, month and day of week — with lists, ranges, steps such as */15, and macros such as @daily. Validate on create and show the next five run times; that catches most mistakes. Know the classic quirk: if both day fields are restricted, the job runs when either matches, so 30 4 1,15 * 5 runs on the 1st and 15th and every Friday, as crontab(5) documents.

Compute in the schedule’s zone, fire in UTC. India stays on UTC+05:30 all year, but a tenant’s New York customers do not. On Sunday 8 March 2026, New York clocks jump from 02:00 to 03:00, so 30 2 * * * has no 02:30 that day. On Sunday 1 November 2026 they fall back from 02:00 to 01:00, so a job at 01:30 matches twice.

Pick rules and publish them: a run in a skipped hour fires at the first valid instant after the gap, and a run in a repeated hour fires once. That is how Vixie cron behaves — jobs in a skipped interval run soon after the change, and jobs in a repeated interval are not re-run. Compute elapsed-time schedules (“every 6 hours”) in UTC, so they never stretch to 7. And since governments change daylight-saving rules at short notice, recompute affected schedules whenever the time-zone database is updated.

Missed runs and overlaps. After an outage, or when the last run is still going at the next tick, you need a rule. Kubernetes’ CronJob has the canonical knobs: startingDeadlineSeconds skips a run that cannot start within that many seconds of its time (unset means no deadline), and concurrencyPolicy is Allow (the default), Forbid (skip the new run while the old one is going) or Replace (stop the old run and start the new one). Its controller refuses to catch up when it counts more than 100 missed schedules, and because it checks every 10 seconds, a deadline under 10 seconds may mean the job never runs.

Clock skew. A dispatcher running 2 seconds fast fires everything 2 seconds early; Quartz insists clustered nodes keep clocks within a second of each other. Run time sync, alert on offset, and stop a dispatcher whose offset passes about 250 ms (see clocks, ordering and conflict resolution).

The final design

Put the pieces together. Only steps 4 to 6 decide how late a job starts, and they touch memory, one conditional write and a queue.

Follow the 09:00 payment reminder through it:

1. Schedule. The payments service calls the API with an idempotency key; the API checks the tenant’s quota for that minute and spreads 09:00 IST to 03:34:12Z inside the window.
2. Persist. The job row and timer reach a quorum of replicas before the 201. Jobs due within five minutes also go into the near-term set; jobs due now go straight to a ready queue.
3. Load. The node that owns partition 417 through its etcd lease reads the 03:34 bucket at about 03:29 into Redis.
4. Fire. At 03:34:12 the dispatcher pops the timer, writes a recheck timer and flips the job to QUEUED at version 1 with a conditional write; a cron occurrence would also write its successor here.
5. Queue. The job enters the tenant’s flexible ready queue, behind any due precise jobs.
6. Claim. A callback executor claims it with a 45-second lease as attempt 1.
7. Execute. It calls the reminders endpoint with the idempotency key, heartbeating if the call runs long.
8. Report. It acks through the API: SUCCEEDED, or RETRYING with a jittered timer, or DEAD after the last attempt. Every transition flows to the history store and the lateness metrics.

Check it against the requirements. Precise jobs fire from memory every 100 ms in their own lane, so p99 lateness stays under a second even at 09:00. Forty store nodes take 58k writes/s on average and 370k in the peak minute. A job is durable before the 201 and guarded afterwards by timers, leases and rechecks; it fires once per occurrence and runs at least once with a stable key. No node is special: partitions move within seconds of a crash, and Redis and the queues can be rebuilt from the store.

What interviewers look for by level

Follow-up questions

What if jobs depend on each other, like a DAG?

Then you want a workflow engine, not a bigger scheduler. A graph needs state per step, fan-out and fan-in, per-step retries and durable progress across hours — what Temporal, or Airflow for data pipelines, provides. The scheduler fires “start tonight’s settlement workflow at 02:00”; the engine runs the graph. Jobs that schedule their successors work for two or three steps, but Dropbox noted such chaining in ATF had no framework-level visibility or control (see multi-step workflows).

What if a job must run exactly once, such as a payout?

No scheduler can promise that: a worker can always die between doing the work and reporting it. Put the guarantee where the money moves. The payout service stores the job’s idempotency key in the same transaction as the ledger entry, returns the stored result on a repeat, passes the key to the bank where it can, and reconciles daily. The scheduler keeps the key stable and fences stale attempts (see design a payment system).

What if one tenant schedules 50 million jobs for the same second?

Treat it as admission control, not scaling: 50M ÷ 100k/s is over eight minutes of the whole platform’s dispatch budget. Enforce the per-tenant cap per minute bucket, offer a bulk API that takes a window and spreads the jobs itself, and buffer creates in an inbound queue so the store absorbs them steadily, as Dynein does with an inbound queue that acts as a write buffer for spikes. Fair queuing protects everyone else at run time, and a tenant regularly this large gets its own cell.

What if a whole region fails?

Home each tenant in one region and replicate its jobs asynchronously to a standby. On failover, the standby takes the failed region’s partitions under a new assignment version and reloads timers from its replica. You pay twice: jobs created in the last seconds may be missing (the replication lag), and jobs fired just before the failure may fire again — idempotency keys absorb those. Tenants that cannot lose one job need synchronous cross-region writes, at tens of milliseconds per create (see multi-region and active-active).

What if a job needs a GPU or a particular machine?

That is the other kind of scheduler. Ours decides when; a cluster scheduler such as Kubernetes or Borg decides where, packing work onto machines by CPU, memory, accelerators and affinity. Make it a target type: when the job fires, a worker submits a Kubernetes Job or a GPU-queue request and tracks it, heartbeating on its behalf. Placement and preemption stay with the cluster scheduler (see design a GPU job scheduler).

How to say it in the interview

“I’ll design a shared job scheduler: about a billion one-off, delayed and cron jobs a day, up to a hundred thousand falling due in one second at the top of the hour, under a second of lateness for precise jobs, and no lost jobs.

The core split is between deciding when and doing the work. Tenants call a scheduler API, which writes each job and a small timer to a sharded wide-column store. Timers are bucketed by partition — a hash of the job ID, 1,024 of them — and by minute, so a bucket holds about 680 timers, and about six thousand at 09:00.

Scheduler nodes own partitions through leases in etcd. Each loads the next five minutes of its buckets into Redis sorted sets, and a dispatcher fires what is due every 100 milliseconds with a conditional write, SCHEDULED to QUEUED at this version. So a job fires once even if two nodes overlap, and a cancelled or moved job simply loses that write. Then it goes onto a ready queue per tenant and lane.

Workers claim with a 45-second lease, heartbeat and ack. If a worker dies, the lease expires and the job runs again, so execution is at-least-once: every attempt carries the same idempotency key, and an attempt number fences stale results. Retries use exponential backoff with full jitter, counted at claim, and exhausted jobs go to a dead-letter queue.

At the top of the hour, precise jobs have their own lane, flexible jobs are spread across a declared window, and weighted fair queuing plus quotas share the rest between tenants.

Cron schedules keep an IANA time zone, compute the next run as each one fires, and follow explicit rules for daylight-saving gaps, missed runs and overlaps.”

Concepts used in this design

• Message queues — ready queues with per-message leases.
• Delivery guarantees: at-least-once, exactly-once and DLQs — why execution is at-least-once and where dead jobs go.
• Effectively-once: idempotent consumers and the inbox pattern — turning duplicate runs into no-ops.
• Idempotency — stable keys on create and on every attempt.
• Timeouts, retries and backoff with jitter — the retry policy.
• Retry storm — what synchronised retries do to a recovering service.
• Heartbeats — keeping leases alive and detecting dead workers.
• Consensus and leader election — the coordinator that assigns partitions.
• ZooKeeper and etcd — leases and watches for partition ownership.
• Redis — sorted sets for the near-term tier.
• Cassandra and DynamoDB — a wide-column store for jobs and timers.
• Database sharding — spreading jobs and timers by hash.
• Table partitioning — time buckets that expire whole.
• Back-of-envelope estimation — Little’s law and bucket sizes.
• Noisy neighbours and fairness — weighted fair queuing between tenants.
• Clocks, ordering and conflict resolution — clock skew, and why fencing beats timing.

References and further reading

1. Dynein: Building a Distributed Delayed Job Queueing System (Airbnb Engineering)medium.comDelayed jobs in DynamoDB keyed by a random partition and the scheduled time, a conditional update before each dispatch to SQS, and why Quartz on MySQL topped out.
2. Asynchronous Task Scheduling at Dropboxdropbox.techATF’s store consumer, one queue per task type and priority, heartbeats, per-state timeouts that re-enqueue stuck tasks, and a 5-second p95 delivery target.
3. Scaling Slack’s Job Queueslack.engineering1.4 billion jobs a day at peaks of 33,000 a second, a Redis memory outage, and durable Kafka placed in front of Redis with one relay per topic.
4. Kubernetes documentation: CronJobkubernetes.ioCron syntax, time zones, startingDeadlineSeconds, the Allow, Forbid and Replace concurrency policies, and the 100-missed-runs rule.
5. Amazon SQS visibility timeoutdocs.aws.amazon.comLeases on received messages: a 30-second default, a 12-hour maximum, extension with ChangeMessageVisibility, and redelivery after a crash.
6. Quartz Scheduler: configuring clustering with the JDBC JobStorequartz-scheduler.orgNodes share one database and race to lock each trigger; the cluster-wide lock degrades beyond about three nodes; clocks must agree within a second.
7. Hashed and Hierarchical Timing Wheels (Varghese and Lauck, SOSP 1987)cs.columbia.eduConstant-time timer start, stop and tick; hierarchical wheels cover 100 days at one-second resolution with 244 slots instead of 8.64 million.
8. PostgreSQL documentation: SELECT, the locking clausepostgresql.orgFOR UPDATE SKIP LOCKED for queue-like tables read by many consumers, and why the view it gives is deliberately inconsistent.
9. Exponential Backoff and Jitter (AWS Architecture Blog)aws.amazon.comSimulations showing that backoff alone still clusters retries and that full jitter spreads them and cuts total work.
10. cron(8), Debian manual pagemanpages.debian.orgVixie cron wakes once a minute; jobs in a skipped daylight-saving hour run after the jump, and jobs in a repeated hour are not run twice.