Design a GPU job scheduler

A GPU job scheduler decides which team’s work runs on which accelerators, when, and where in the network. Picture the central ML platform of a large Indian tech group: 4,096 GPUs — 3,072 H100s and an older pool of 1,024 A100s — shared by about 40 teams. On one ordinary Tuesday the fraud team fine-tunes a 7B model before the festive-sale rush, search launches 500 hyper-parameter trials at once, the GenAI team is ten days into a three-week training run on 1,024 H100s, and a few hundred people have notebooks open. At on-demand cloud prices that fleet is worth about $575,000 a day, so an idle GPU is money burning — yet the 1,024-GPU job cannot start until 128 whole machines are free at the same moment, in the right corner of the network. In this problem you design the scheduler that keeps that cluster full, fair and moving.

Understand the problem

Start by saying how this differs from the cron-style job scheduler you may have designed before. That system answers “when should this run?” and hands each task to any free worker. A GPU cluster scheduler answers “who gets these scarce machines, which ones, and who has to wait?” — and every word of that question is harder.

Functional requirements

1. Users should be able to submit a job that needs anything from a slice of one GPU to 1,024 GPUs, and have it start with all its GPUs at once — never half of them.
2. Teams should get a guaranteed share of each GPU type, and be able to borrow idle GPUs from other teams, knowing that borrowed GPUs can be taken back.
3. Long jobs should survive hardware failures and preemption, restarting automatically from their latest checkpoint and losing at most one checkpoint interval of work.
4. Users should be able to see where their job is in the queue, why it is waiting, and what their team has used.

Non-functional requirements

• Scale. 512 nodes of 8 GPUs each: 3,072 H100s and 1,024 A100s. About 40 teams, 5,600 jobs a day, up to 2,000 jobs pending at the busiest hour. Plan for twice that within 18 months.
• Utilisation. At least 90% of healthy GPUs allocated whenever jobs are waiting that could use them. An idle GPU next to a non-empty queue is a bug.
• Waiting time. A team asking for GPUs inside its guaranteed quota gets them within 15 minutes, which is the time it takes borrowers to checkpoint and leave. Jobs of 8 GPUs or fewer start in under 5 minutes at p50. A 1,024-GPU job starts within 24 hours.
• Lost work. At most one checkpoint interval plus about 10 minutes of restart per failure, with no human in the loop.
• Placement quality. A 256-GPU job runs inside one network block; a 1,024-GPU job spans no more than four.
• Control plane. 99.9% available for submissions and decisions. Running jobs keep running if the scheduler is down. A scheduling pass over 2,000 pending jobs finishes in under 10 seconds.
• Isolation. One team’s job can never read another team’s GPU memory, and a crash in one notebook cannot kill another.

Out of scope

• Online inference serving. The serving fleet has its own design in Design an LLM inference service; here it only lends and borrows GPUs.
• Time-triggered pipelines (“retrain nightly at 02:00”). A workflow engine submits jobs to us; we decide where and when they run.
• The training code itself — parallelism strategy, optimiser, data loading. We assume a job can save and load a checkpoint and exits cleanly on SIGTERM.
• Experiment tracking, model registries, data pipelines and buying the hardware.

The set-up

Plan the approach

Say the shape early: “Admission decides whether and when a job runs; placement decides where. Quotas sit in front of admission; checkpoints sit under everything.” Satisfy each functional requirement with a simple design, then spend the deep dives on the problems that make GPU scheduling hard: gangs, topology, fairness, preemption, failures, sharing one GPU, and scaling the scheduler itself.

First, the hardware, because it shapes every decision.

Inside a node, eight GPUs talk over NVLink through an NVSwitch, by far the fastest path. Each GPU also has its own 400 Gb/s InfiniBand NIC — 3,200 Gb/s per node, the same aggregate the cloud’s 8 × H100 instance advertises. In a rail-optimised layout, NIC i of every node in a block plugs into leaf switch i, so same-numbered GPUs on different nodes are one switch apart. Blocks join through a spine that we assume is 2:1 oversubscribed. That is a common cost choice: the network behind Llama 3 gave pods of 3,072 GPUs full bisection bandwidth but ran 1:7 above them, and Meta made its training scheduler topology-aware to avoid crossing pods.

Core entities

• Team and quota. Per GPU type (a flavour: h100, a100, h100-mig-1g): a guaranteed nominal quota, borrowing and lending limits, a fair-share weight, and the cohort of teams it shares with.
• Job (workload). One or more pod sets (“N pods × 8 GPUs of flavour F”), a topology constraint, a priority class, a time limit, whether it is preemptible, and a checkpoint policy.
• Node. Flavour, topology labels (block, leaf, rail), allocatable GPUs and health state (ready, suspect, draining, repair).
• Allocation. Job → exact nodes, admitted under nominal quota or as a borrower, and when.
• Checkpoint. Job, training step, object-storage URI, size, time written, verified or not.
• Usage record. Team, flavour, GPU-seconds, borrowed or not — the raw material for chargeback.

API

Users talk to a small REST API through a CLI, an SDK or a notebook. A submission declares what the job needs, never which machines.

POST /v1/jobs
Content-Type: application/json
Idempotency-Key: 7d0c2a51-6f7e-4f0e-9a51-1b2a3c4d5e6f

{
  "name": "risk-7b-ft-festive",
  "team": "fraud-ml",
  "priority_class": "standard",
  "flavour": "h100",
  "pods": 4,
  "gpus_per_pod": 8,
  "topology": { "required": "block" },
  "image": "registry.internal/fraud/ft:2026.10.2",
  "command": ["torchrun", "train.py", "--config", "ft.yaml"],
  "time_limit": "12h",
  "preemptible": true,
  "checkpoint": {
    "uri": "s3://ckpt/fraud-ml/risk-7b-ft-festive/",
    "interval": "auto",
    "grace_period": "300s"
  },
  "max_restarts": 20
}

The reply comes straight back; the job waits in its team’s queue.

{
  "job_id": "job_8c41",
  "state": "PENDING",
  "queue": "fraud-ml",
  "position": 3,
  "reason": "QuotaFull",
  "estimated_start": "2026-10-27T14:20:00+05:30"
}

Later, GET /v1/jobs/job_8c41 shows where it runs and how it is doing. A 7B model at 14 bytes per parameter makes a 98 GB checkpoint; the arithmetic is below.

{
  "job_id": "job_8c41",
  "state": "RUNNING",
  "admitted_as": "nominal",
  "placement": { "blocks": ["b07"], "nodes": 4 },
  "restarts": 1,
  "last_checkpoint": { "step": 18400, "at": "2026-10-27T15:02:11+05:30", "bytes": 98000000000 },
  "gpu_hours_used": 212.5
}

The rest of the surface:

GET  /v1/teams/fraud-ml/quota        nominal, borrowed, lent and fair-share value per flavour
GET  /v1/queues/fraud-ml?state=PENDING
POST /v1/jobs/job_8c41:cancel
POST /v1/jobs/job_8c41:checkpoint    ask the job to save now, e.g. before node maintenance
POST /v1/sweeps                      N trials in one request; each trial becomes its own small job

There is also a job contract, which matters as much as the API: on SIGTERM, finish the current step, checkpoint within the grace period and exit 0; write checkpoints atomically (data first, then a small manifest naming the step); on start, resume from the newest manifest. Jobs that cannot keep this contract run as non-preemptible, and pay for it.

Back-of-envelope estimates

These numbers decide what the scheduler should optimise. (For the method, see back-of-envelope estimation.)

Capacity and cost. 4,096 GPUs × 24 hours = 98,304 GPU-hours a day. Price them at the public on-demand rate for the equivalent cloud machine. On 27 October 2026, AWS listed a p5.48xlarge (eight H100s) at $55.04 an hour in US East — $6.88 per H100-hour — and a p4d.24xlarge (eight A100s) at $21.96, or $2.74 per A100-hour:

• H100s: 3,072 × 24 = 73,728 GPU-hours × $6.88 ≈ $507,000 a day
• A100s: 1,024 × 24 = 24,576 GPU-hours × $2.74 ≈ $67,000 a day
• Total: ≈ $575,000 a day, ≈ $210 million a year

Owned or reserved capacity costs less, but on-demand is the honest yardstick: it is what a team would pay to go elsewhere.

What 1% is worth. One percent of utilisation is 41 GPUs, about five nodes, idle all year: ≈ $2.1 million a year. That number justifies most of the complexity below — backfill, borrowing and defragmentation each win back a few percent.

Demand. A day’s work looks roughly like this:

Demand averages 93% of capacity and arrives in office hours, so queues form every day. The scheduler is never short of work; it is short of GPUs in the right shape at the right time.

Arrivals and decisions. 5,600 jobs ÷ 86,400 s ≈ 0.065 jobs a second. The peak is a sweep: 500 trials in a minute is about 8 a second. Re-examining 2,000 pending jobs every 10 seconds is 200 evaluations a second against a few hundred nodes held in memory. The Borg paper reports cells with more than 10,000 task arrivals a minute — about 167 a second, twenty times our peak. The hard part is decision quality, not rate.

Checkpoint size. Mixed-precision training with Adam keeps, per parameter, 2 bytes of bf16 weights, 4 bytes of fp32 master weights and two fp32 Adam moments of 4 bytes each: 14 bytes to save. (Add 2 bytes of bf16 gradients and you get the familiar 16 bytes per parameter of GPU memory, but gradients are not saved.) A 70B model: 70 × 10⁹ × 14 B = 980 GB, call it 1 TB. The 7B fine-tune: 98 GB.

Checkpoint write time. Assume the 1,024-GPU job gets 20 GB/s of write bandwidth: 980 GB ÷ 20 GB/s = 49 s, call it 50 s of idle GPUs if the save is synchronous — about 1 GB per GPU. With asynchronous checkpointing (copy shards to host memory, resume, upload in the background) the pause shrinks to a few seconds. Storage must absorb the bursts: Llama 3’s storage sustained 2 TB/s and peaked at 7 TB/s, and checkpoint writes still saturated it briefly.

Failures. Llama 3’s report gives real numbers. In a 54-day snapshot of pre-training on up to 16,384 H100s there were 466 job interruptions: 47 planned (firmware upgrades, configuration changes) and 419 unexpected, about 78% of them confirmed or suspected hardware, with GPU problems alone 58.7%. Assume all 16,384 GPUs were in the job, which flatters the hardware:

• 54 days × 24 h × 16,384 GPUs = 21.2 million GPU-hours
• 21.2M ÷ 419 ≈ one interruption per 50,700 GPU-hours, about 5.8 years per GPU
• A 1,024-GPU job: 50,700 ÷ 1,024 ≈ 49.5 hours between failures — about two days
• The whole 4,096-GPU cluster: 4,096 × 24 ÷ 50,700 ≈ two interruptions a day

A synchronous training job is as fragile as its weakest GPU: one failure stops all 1,024. And the bigger the job, the more often it dies.

Checkpoint interval. Checkpoint too rarely and every failure throws away hours; too often and you spend all day saving. The Young–Daly approximation gives the sweet spot: interval τ ≈ √(2 × C × M), where C is the checkpoint cost and M the mean time between failures. With C = 50 s and M = 49.5 h = 178,200 s:

τ ≈ √(2 × 50 × 178,200) = √17,820,000 ≈ 4,220 s, about 70 minutes

The waste at that interval has three parts: time spent checkpointing, C ÷ τ = 50 ÷ 4,220 ≈ 1.2%; work redone after a failure, on average half an interval, τ ÷ 2M ≈ 1.2%; and a 10-minute restart per failure, 600 ÷ 178,200 ≈ 0.3%. Total ≈ 2.7%. Checkpointing every 6 hours instead wastes 0.2% + 6.1% + 0.3% ≈ 6.6%. On a 30-day run that costs 1,024 × $6.88 × 720 h ≈ $5.07 million on demand, the difference of 3.9 points is about $200,000. Make checkpoints asynchronous (C ≈ 5 s) and the optimum drops to √(2 × 5 × 178,200) ≈ 1,335 s, about 22 minutes, with total waste near 1.1%.

High-level design

Build one requirement at a time. By the end of this section the cluster works; the deep dives make it efficient and fair.

Users should start jobs with all their GPUs at once

The core path is short. The Job API validates a submission and writes it to the job store — a strongly consistent store such as etcd — and to the team’s pending queue. The scheduler loops: take pending jobs in order, decide whether each can be admitted now, and if so place it on specific nodes. A node agent on every machine launches pods, and a device plugin advertises the node’s healthy GPUs — in Kubernetes, the extended resource nvidia.com/gpu, counted in whole devices.

The key word is gang. A data-parallel job with 128 pods cannot take a single step until all 128 are running and connected; 127 of them is the same as zero. So the scheduler admits a job only when it can reserve every GPU it needs, and only then creates its pods. Kueue keeps a job suspended until quota is reserved for the whole workload; Volcano’s PodGroup has a minMember count below which no pod is scheduled. The first deep dive shows how this can still go wrong.

Teams should get a guaranteed share and borrow idle GPUs

A single queue in arrival order lets whichever team submits first take the cluster. Add a quota manager. Each team gets a queue (Kueue calls it a ClusterQueue) with a nominal quota per flavour: fraud-ml is guaranteed 256 H100s, search 512, genai-core 1,536, and so on until the guarantees add up to the cluster. Teams that share hardware form a cohort, inside which a team may borrow others’ unused quota up to its borrowing limit, while each team caps how much it lends.

# One team's queue in the h100-shared cohort (Kueue-style, abridged)
kind: ClusterQueue
metadata: { name: fraud-ml }
spec:
  cohortName: h100-shared
  resourceGroups:
    - coveredResources: ["nvidia.com/gpu"]
      flavors:
        - name: h100
          resources:
            - name: nvidia.com/gpu
              nominalQuota: 256      # guaranteed
              borrowingLimit: 512    # may run up to 768 when others are idle
              lendingLimit: 256      # lends everything it is not using
  preemption:
    reclaimWithinCohort: Any         # take back lent GPUs from borrowers
    withinClusterQueue: LowerPriority

Admission now has three outcomes. A job that fits the team’s unused nominal quota is admitted and protected. One that fits only the cohort’s idle quota, within the team’s borrowing limit, is admitted as a borrower and marked preemptible. Anything else waits, with a recorded reason. When a lender wants its GPUs back, borrowers are preempted — checkpoint, exit, requeue — until the owner fits. How to share the borrowable pool fairly is a deep dive.

Long jobs should survive failures and preemption

With a failure every two days for a 1,024-GPU job, restart must be automatic. Add a checkpoint store — object storage with a fast tier in front — for sharded checkpoints; a health monitor that turns node heartbeats and GPU error signals into “this node is unfit”; and restart logic in the scheduler that re-admits a broken gang, on a hot spare if possible, from the latest verified checkpoint.

Preemption uses the same machinery: a preempted job was asked to stop, saved its state and went back into the queue. This is the long-running jobs pattern — durable progress plus idempotent resume — for jobs that cost $7,000 an hour.

Users should see their place in the queue and their usage

Otherwise engineers waiting for 256 GPUs ask in chat every ten minutes. Every scheduler decision — admitted, borrowed, preempted, waiting for quota or topology — becomes an event. A queue view answers “where am I and why?”, and a usage ledger turns GPU-seconds into per-team reports.

Reason codes matter. “Waiting for quota” sends the engineer to their manager; “waiting for topology: needs one whole free block” tells them a preferred constraint would start today; “waiting behind a reservation” tells them when. The ledger splits GPU-seconds by team, flavour, and nominal versus borrowed — the basis for metering and billing.

You now have a working scheduler. The deep dives are about what breaks at 4,096 GPUs and 40 competing teams.

Deep dives

How do we start a 1,024-GPU job without deadlocking the cluster?

Bad: place pods one at a time. A general-purpose scheduler, such as the default Kubernetes one, binds each pod as soon as it finds room. Now imagine eight free nodes and two jobs that need six each. Pods from both are created together, so job A gets four nodes and job B gets four. Neither can start a training step; both hold their nodes forever, waiting for two more. Sixty-four GPUs, worth about $440 an hour, do nothing until a human notices.

Good: all-or-nothing admission. Treat the job as one unit: reserve all 48 GPUs for A in one transaction against the scheduler’s in-memory view, create A’s pods, and leave B queued holding nothing. Partial allocations cannot exist. You pay in two ways.

First, quota is not capacity. A queueing layer that admits on quota arithmetic can admit two jobs that fit the quota but not the machines, for example while some nodes drain. Add a readiness timeout: if a gang’s pods are not all running within, say, 10 minutes, evict the whole job and requeue it. Kueue’s all-or-nothing mode does exactly this, optionally admitting gangs strictly one at a time.

Second, big jobs starve. A 1,024-GPU job needs four whole blocks free at the same instant, but small jobs keep grabbing nodes as they free up, so that instant never comes. Strict FIFO fixes starvation by starting nothing behind the big job — and leaves blocks idle for hours as they drain.

Great: a reservation for the head job, and backfill around it. HPC schedulers such as Slurm have long done this. When the highest-priority job does not fit, compute when it will fit — from running jobs’ time limits — and reserve those nodes from then. Then backfill: Slurm starts a lower-priority job only if it does not delay the expected start of any higher-priority job.

Without backfill, blocks 1–3 sit idle for 3 + 1.5 + 1 = 5.5 block-hours while J waits: 5.5 × 256 = 1,408 GPU-hours, about $9,700, every time a big job queues. Backfill recovers 4 of those hours here. You pay three costs:

• Time limits must be honest. Slurm’s guide says backfill needs reasonably accurate limits. Enforce them — at its limit a backfilled job is preempted to checkpoint — and people learn to ask for 3 hours, not 72.
• The planning window must cover the longest job. Slurm looks one day ahead by default (bf_window) and advises a window at least as long as the longest time limit, to prevent starvation. Two-week pre-training runs are better made preemptible and resumable than planned around.
• Reservations idle capacity on purpose. The last block before J starts may sit empty for minutes. That is the price of a promise.

One GPU-specific refinement: if the reservation is far off, preempt borrowers on the target blocks instead. They checkpoint and leave within minutes, and the big job starts now instead of in four hours.

How do we place a job so its GPUs talk fast?

Synchronous training ends every step with collective communication — an all-reduce of gradients, or the all-gather and reduce-scatter of sharded data parallelism — so a step is only as fast as its slowest link. From fast to slow: NVLink inside a node, the rail leaf switch inside a block, the oversubscribed spine between blocks. Placement decides which links a job uses.

Bad: any free GPUs. A flat pool turns a 32-GPU job into 32 single GPUs on 32 nodes, or spreads a 256-GPU job over four blocks. Every collective crosses the spine, competing with everyone else’s traffic; the job slows down and slows its neighbours.

Good: whole nodes, same block preferred. Allocate multi-node jobs in whole nodes, so tensor parallelism stays on NVLink, and prefer one block, falling back to the next level up. You gain most of the speed most of the time — but “preferred” quietly becomes “scattered” exactly when the cluster is busiest.

Great: topology as a constraint, plus a fight against fragmentation. Model the network as a tree — node, leaf group, block, spine — and let jobs say required: block (never split) or preferred: block. Kueue’s topology-aware scheduling, beta since v0.14, does this with node labels per level and per-job annotations, computing free capacity per domain from healthy nodes. Then choose the block carefully:

• Best fit, not first fit. Put a 32-node job in the block with the fewest free nodes that still fits, leaving emptier blocks whole for the next big job.
• Keep small jobs together. Route single-node jobs, sweeps and evaluations to designated “small-job” blocks. A 500-trial sweep spread over twelve blocks is the fastest way to lock out big jobs.
• Defragment at checkpoint boundaries. When free nodes are scattered, checkpoint a few small preemptible jobs and restart them in the small-job block. Gandiva (OSDI 2018) used migration like this to give incoming jobs better locality.
• Bound the wait for compactness. Slurm lets a job wait for its desired number of leaf switches up to max_switch_wait (300 seconds by default) before accepting a worse placement. Required for pre-training; preferred with a short wait for fine-tuning.

Put numbers on it. If each of the twelve H100 blocks has just four nodes busy with small jobs, 12 × 28 = 336 nodes — 2,688 GPUs — are free, yet no job that requires a whole block can start. Track “largest free shape” next to “free GPUs”, and alert when they diverge.

Two more duties. Replace in place: when a node dies, find a replacement inside the same block; Kueue’s topology-aware scheduling swaps in a replacement for a single failed node without moving the rest of the job, and evicts the job if several fail at once. Not every job cares: Tiresias (NSDI 2019) found some models barely affected by placement, so relax constraints for jobs that profile as insensitive.

How do we share the cluster fairly between 40 teams?

Bad: static partitions. Give each team its own nodes. Isolation is perfect — until fraud-ml’s 256 GPUs sit idle over a long weekend while genai-core queues 600 GPUs of work next door. With 40 teams someone is always idle and someone always waiting, at $2.1 million per percentage point.

Good: one pool with priorities. Order one shared queue by priority, then arrival. Utilisation is high, but the team that submits most, or labels everything “high”, takes the cluster. Priorities without quotas become an arms race.

Better: guaranteed quota with borrowing and lending. The cohort model from the high-level design: nominal quota is a promise, idle quota is lent, and the owner reclaims it by preemption. Kueue’s details are worth copying: workloads that fit their own nominal quota are admitted before ones that must borrow, and when reclaiming, its classic preemption picks victims from borrowing queues first, then the lowest priority, then the most recently admitted.

You gain utilisation and guarantees. You pay with preemptions, and the model leaves a question open: when three teams want the same idle GPUs, who gets them?

Great: fair sharing of the borrowable pool, on dominant shares, with memory. Three refinements finish the job.

• Share the idle pool by weight. Kueue’s fair sharing gives each team’s queue a share value summarising its borrowing, weighted by fairSharing.weight. Admission prefers the lowest share; preemption takes from the highest.
• Remember the past. Fairness over a minute is not fairness over a quarter. Slurm’s multifactor priority compares promised share with consumed usage, decaying past usage over a configurable half-life, and adds job age so nothing waits forever. Order each queue the same way.
• Count the dominant resource, not just GPUs. An evaluation job that needs 1 GPU but 96 of a node’s 192 vCPUs (OCR pre-processing, say) uses half the node’s CPU; two per node leave six GPUs stranded. Count GPUs only and a team running 100 of them looks like 100 ÷ 3,072 = 3.3% of the H100 pool, while it blocks 50 nodes. Dominant Resource Fairness (Ghodsi et al., NSDI 2011) measures each user by their largest share of any resource — here CPUs, 9,600 of the pool’s 384 × 192 = 73,728 vCPUs, 13% — and equalises that. The paper shows it keeps the properties you want: sharing beats a static slice, lying about demand does not help, and nobody envies another’s allocation.

Finally, priority bands. Borg uses non-overlapping bands — monitoring, production, batch, best effort — and forbids production tasks from preempting one another, to stop preemption cascades. Use four: critical (inference-fleet reclaim, incidents), guaranteed, borrowed, and scavenger (idle GPUs only, killed without ceremony). Jobs in one band never preempt each other.

How do we preempt without throwing away hours of work?

Preemption is how quotas are honoured, big jobs start and the inference fleet gets its GPUs back. Done badly, it destroys the work it was meant to protect.

Bad: kill the job. SIGKILL frees the GPUs instantly, and the victim loses everything since its last checkpoint — on average half an interval, 35 minutes for our 70-minute policy. On 256 GPUs that is about 150 GPU-hours, roughly $1,000, per preemption.

Good: SIGTERM, a grace period, and the job contract. Send SIGTERM and wait a grace period sized to the checkpoint cost — 300 seconds for a 50-second save. The job finishes its step, checkpoints, exits, and goes back to the front of its queue. Lost work drops to one training step. The preemptor waits for the grace period, and you depend on every job keeping the contract.

Great: preemption as a planned, priced operation.

• Choose victims by cost, not just priority. Among eligible borrowers, prefer small jobs that checkpointed recently, so the fewest GPU-hours are at risk. Heartbeats tell the scheduler each job’s last checkpoint.
• Guarantee a minimum run. A job admitted less than, say, 30 minutes ago can only be preempted by the critical band; otherwise two teams thrash the same nodes, starting up and checkpointing in turn.
• Shorten the reclaim path. Ask for an early checkpoint (:checkpoint) when a reclaim looks likely, so the preemption itself costs only the final step. That keeps the 15-minute quota promise even for slow savers.
• Order short work by attained service. You rarely know how long a sweep trial will run. Tiresias (NSDI 2019) orders such jobs by attained service — GPUs × time used so far — so new and small jobs go first. Because preempting training is expensive, it discretises priority into a few queues so jobs are demoted rarely (with two queues, a 1-GPU-hour threshold covered more than 60% of jobs in their traces), and promotes jobs that wait too long. It cut average job completion time by up to 5.5× against a YARN-based production scheduler.

Gandiva (OSDI 2018) went further for small jobs: suspend at a mini-batch boundary, when GPU memory use is lowest (up to 77 times below the peak in their measurements), and time-slice GPUs among jobs. Suspend plus resume took under 100 ms for typical image models and up to a second for large translation models, so a one-minute slice cost 2% or less; with migration as well, utilisation rose 26% on a 180-GPU cluster. The price is integration with the training framework, which is why most platforms use checkpoint-based preemption and keep time-slicing for small interactive work.

How do we keep a 1,024-GPU job alive when hardware fails every two days?

A 1,024-GPU job expects an interruption about every 49.5 hours; the cluster, about two a day. Llama 3’s root-cause table maps what breaks: faulty GPUs and GPU HBM3 memory lead, then software bugs, network switches and cables, and unplanned host maintenance. In those 54 days Meta needed significant manual intervention only three times.

Bad: page a human. The job dies at 03:00; someone wakes up, finds the bad node and resubmits. Recovery takes hours at about $7,000 an hour.

Good: automatic restart from the last checkpoint. The scheduler sees the gang break (a pod exits, a heartbeat stops, the collective library times out), marks the suspect node, and re-admits the job with a free node from the same block. You gain sleep. You pay when the cluster is full: no free node in the block means waiting, or a bad placement.

Great: a health pipeline, hot spares and straggler detection.

• Hot spares. Keep one healthy node per block out of the training pool: 16 nodes, about 3% of the cluster. Restart never waits for capacity, and the replacement sits under the same leaf switches. Spares run scavenger work that can be killed instantly. If repair takes about two days, two failures a day keep about four nodes out at any moment (Little’s law), so 16 is comfortable.
• Health checks at three levels. Passive: the node agent watches driver errors (NVIDIA’s Xid events), ECC memory errors, thermal throttling and NIC link flaps, and marks the node suspect. Active: idle nodes run short burn-in tests — a matrix multiply per GPU, an all-reduce across node and rail — before rejoining the pool. Job-level: the collective library’s watchdog turns a hang into a clean failure. See health checks done right.
• Drain, don’t yank. A suspect node finishes a healthy job, takes no new work and goes to repair. Planned work such as firmware upgrades uses the same flow, one block at a time.
• Restart fast. Budget about 10 minutes: a minute or two to detect, one to pick a spare, two for pods to start from cached images, about 50 seconds to read 980 GB at 20 GB/s, and a few to rebuild communicators and warm up. Each minute saved is worth $117 on this job.
• Catch stragglers. Llama 3’s team noted that one slow straggler can slow thousands of GPUs, and built tooling to find it. Have every rank report its step time; a node consistently, say, 20% slower than the median is treated as failed and swapped at the next checkpoint.

How do we share one GPU between notebooks and small jobs?

Kubernetes device plugins advertise GPUs as whole numbers that cannot be overcommitted or shared between containers. That suits training and wastes money on notebooks, which run a cell for ten seconds and then wait minutes for their human.

Bad: a whole GPU per notebook. Simple and isolated — and 300 concurrent notebooks park 300 H100s, about $2,000 an hour, to hold open Python sessions.

Good: time-slicing. Advertise each GPU as, say, four replicas and let the driver switch contexts. Any job fits with no setup, but there is no memory isolation — one notebook allocating 70 GB starves the rest — and no fault isolation. Fine for one trusted team; wrong across 40.

Good for one team’s batch work: MPS. NVIDIA’s Multi-Process Service runs kernels from several processes at once, for better throughput than taking turns. Its documentation warns that a fatal fault in one client may bring down others sharing the GPU, so keep it inside one team.

Great: MIG for anything shared across teams. Multi-Instance GPU splits a GPU into up to seven instances, each with its own compute and its own path through the memory system — L2 cache, memory controllers, DRAM — so one tenant cannot see or slow another. An H100 80 GB offers 1g.10gb (seven per GPU), 2g.20gb (three), 3g.40gb (two), 4g.40gb (one) and the whole 7g.80gb.

So the design sets aside 8 H100 nodes (64 GPUs) as a MIG pool of 1g.10gb slices: 64 × 7 = 448 notebook slots for about 300 concurrent notebooks, using 64 of the 3,072 H100s. The slices are their own flavour (h100-mig-1g) with their own quotas. You pay three costs: shapes are fixed, so a notebook needing 30 GB asks for a bigger profile; re-slicing needs an empty GPU, so the profile mix changes by draining a node at a time, not per request; and idle notebooks still hold slices, so reap sessions idle for more than an hour or two.

How does the scheduler scale, and what happens when it is down?

At 200 evaluations a second over 512 nodes, one well-written scheduler process copes from memory. The real questions are availability, and what fails when the scheduler does.

Bad: one process doing everything. One loop accepts API calls, talks to node agents, schedules and writes state. A 500-trial sweep blocks submissions, a slow pass delays heartbeats until nodes look dead, and a crash loses whatever was in memory.

Good: separate the pieces, replicate the state. Keep job and node state in a strongly consistent store (etcd or ZooKeeper, or the Kubernetes API server on etcd). Run stateless API replicas, and an active and a standby scheduler chosen by a lease (leader election). Node agents send heartbeats and keep running their pods if the control plane disappears — as Borg’s Borglets keep tasks running even if every Borgmaster replica fails; a Borg master failover typically takes about 10 seconds. An outage then means “no new jobs for a minute”, not “the 1,024-GPU run dies”.

Great: an optimistic scheduler on a snapshot, and cells. Borg’s scheduler runs as a separate process on a cached copy of cell state and submits assignments that the master rejects if they conflict with newer state — optimistic concurrency, so a slow pass never blocks writes. It scores identical tasks once per equivalence class (a 500-trial sweep is one placement problem) and caches scores until something changes. And Borg grows by adding cells — a median of about 10,000 machines each, some taking more than 10,000 task arrivals a minute — rather than one ever-bigger scheduler. Do the same when you add a second cluster (see the follow-ups and cell-based architecture).

Finally, guard against split brain: if two schedulers both believe they lead, both may bind the same nodes. Fence every bind with the lease’s version (a fencing token) so the store rejects a stale leader.

The final design

Put the pieces together. The control plane decides; the nodes run; checkpoints and the usage ledger remember.

Follow a fine-tuning job from submission to bill:

1. Submit. A fraud-ml engineer submits a 32-GPU job from the CLI. The Job API checks identity, team, flavour and image.
2. Record. The spec lands in the job store and the fraud-ml queue; the queue view already shows its position and reason.
3. Admit. Within 10 seconds the scheduler asks the quota manager whether the job fits fraud-ml’s nominal quota, or the cohort’s idle quota as a borrower. If a 1,024-GPU run holds a reservation, the job may still start as backfill if its time limit ends first.
4. Place and bind. The scheduler picks the best-fitting block with four free nodes and binds all four pods together; the device plugin hands each pod its eight GPUs.
5. Checkpoint. The job saves its 98 GB checkpoint to object storage on the computed interval, writing the manifest last.
6. Recover. If a GPU fails, the health monitor cordons the node and the scheduler restarts the gang on the block’s hot spare from the last manifest — about 10 minutes, no human. Stragglers are swapped at the next checkpoint.
7. Share. At night the inference fleet lends idle GPUs to the cohort and reclaims them, via preempt-to-checkpoint, before the morning ramp; teams reclaiming quota by day use the same flow.
8. Account. Every admission, preemption and exit becomes an event; the usage ledger turns them into monthly chargeback.

Check it against the requirements: gangs start whole and big jobs get reservations; teams have guarantees and reclaim lent GPUs within about 15 minutes; a 1,024-GPU job loses about 2.7% of its time to failures and checkpoints (1.1% with asynchronous saves); topology is a constraint and fragmentation a metric; notebooks get isolated MIG slices; and the scheduler can fail over without killing a running job.

What interviewers look for by level

Follow-up questions

What if we add a new GPU generation next year?

Make each GPU type its own flavour with its own quotas, so “1 GPU” never means two things. Let jobs list acceptable flavours in order — Kueue tries a ClusterQueue’s flavours in the listed order and can be told whether to keep looking when one would need borrowing or preemption — so a fine-tune takes new GPUs if free, else H100s. Charge in normalised GPU-hours, weighted by measured throughput on a reference job, so landing on older hardware is not a penalty. Never let one gang span generations: the slowest GPU sets everyone’s pace.

What if the inference fleet lends us its GPUs at night?

The inference fleet is sized for its evening peak, so much of it idles at 02:00. Add its GPUs to the cohort as a lending queue, with lendingLimit set to what it can safely give, and admit only preemptible, checkpointing jobs onto them — evaluations and fine-tunes, never pre-training. Its reclaims run in the critical band. Reclaim on a forecast, not a page: return GPUs 30–60 minutes before the morning ramp, because a large model replica takes minutes to load. You gain night-time capacity for free; you pay with jobs interrupted every morning.

What if we need to span several clusters or regions?

Keep one scheduler per cluster — a cell with its own control plane and failure domain — plus a thin global dispatcher that picks a cluster per job and leaves placement local. It weighs quota per cluster, free capacity of the right shape, and data gravity: run where the dataset and checkpoints are, since moving a 1 TB checkpoint between regions costs time and egress. Never split a gang across clusters; collectives over a wide-area link would crawl. Replicate critical runs’ checkpoints to a second region.

What if we burst into spot or preemptible cloud capacity?

Treat spot GPUs as another flavour with a much shorter time between interruptions, and admit only checkpointing jobs. Recompute Young–Daly for it: if each spot node is reclaimed about every 6 hours, a 4-node job’s MTBF is 21,600 ÷ 4 = 5,400 s, so with a 10-second checkpoint it should save every √(2 × 10 × 5,400) ≈ 330 seconds. Spot capacity comes in odd shapes, so keep spot jobs small and big gangs on owned capacity. You gain cheap GPUs; you pay in interruptions and checkpoint traffic.

What if every team must pay for what it uses?

The usage ledger already holds GPU-seconds by team, flavour and nominal versus borrowed. Charge for allocation, not utilisation — 64 GPUs held at 20% busy still block 64 GPUs — but publish per-job utilisation so teams can fix their own waste. Price guaranteed quota as a reservation, paid whether used or not, and borrowed capacity at a discount, because it can be taken away. Price the MIG pool per slice-hour. See cost-aware design and metering and usage-based billing.

How to say it in the interview

“I’ll design the scheduler for a shared 4,096-GPU cluster — 3,072 H100s and 1,024 A100s — used by about 40 teams for everything from notebooks to a 1,024-GPU training run. At on-demand prices that is about 575 thousand dollars a day, so each point of utilisation is worth about 2 million dollars a year.

The design separates admission from placement. Jobs come through an API into per-team queues. A quota manager gives each team a guaranteed quota per GPU type and lets it borrow idle quota from its cohort, as preemptible capacity, with fair sharing weighted by team and dominant-resource accounting when jobs are CPU-heavy.

Admission is all-or-nothing, because a training gang is useless until every pod runs; partial allocation deadlocks. The job at the head of the line gets a reservation, and short jobs with honest time limits backfill the gap. Placement is topology-aware: whole nodes, inside one block where possible, best fit, small jobs kept together so the cluster doesn’t fragment.

Every long job checkpoints. A 70B model with Adam is about 14 bytes per parameter, roughly a terabyte. Using Llama 3’s failure rate, a 1,024-GPU job fails about every two days, and Young–Daly says checkpoint about every 70 minutes for a 50-second save, which wastes about 2.7%. Preemption is SIGTERM, save, exit and requeue at the front. Failures restart automatically on a hot spare; health checks, burn-in and straggler detection catch bad nodes.

Notebooks share MIG slices for real isolation. The scheduler runs as leader plus standby on a consistent store, and nodes keep training if it goes down. Every decision is an event, so engineers see why they’re waiting and teams get a bill.”

Concepts used in this design

• Design a job scheduler — the cron-style sibling, and where time-triggered submissions come from.
• Design an LLM inference service — the serving fleet that lends and reclaims GPUs.
• Design an LLM gateway and Design ChatGPT — the products these models end up serving.
• LLM inference serving — why serving replicas take minutes to warm up.
• Long-running jobs — checkpoints, idempotent resume and progress reporting.
• Noisy neighbours and fairness and Noisy neighbour — quotas, fair share and isolation between teams.
• Metering and usage-based billing and Cost-aware design — turning GPU-seconds into chargeback.
• Blob storage — where checkpoints live.
• ZooKeeper and etcd, Consensus and leader election and Heartbeats — keeping the control plane consistent and the nodes honest.
• Health checks done right — passive, active and job-level checks.
• Autoscaling done right — the inference fleet’s side of lending capacity.
• Cell-based architecture — one scheduler per cluster, and a dispatcher in front.

References and further reading

1. The Llama 3 Herd of Models (Llama Team, Meta, 2024), Section 3.3arxiv.org466 interruptions (419 unexpected, about 78% hardware) in a 54-day snapshot on up to 16K H100s, pods of 3,072 GPUs with full bisection and 1:7 above, stragglers, and checkpoint bursts on storage.
2. Amazon EC2 On-Demand Pricingaws.amazon.comAs read on 2026-10-27 for US East (N. Virginia), Linux: p5.48xlarge (8 × H100, 192 vCPUs, 2,048 GiB, 3,200 Gb/s) at $55.04/hour and p4d.24xlarge (8 × A100) at $21.96/hour.
3. Kueue documentation: Cluster Queue (v0.20)kueue.sigs.k8s.ioNominal quota, borrowing and lending limits, cohorts, resource flavours, queueing strategies and preemption policies; the latest release was v0.20.3 when read.
4. Kueue documentation: Topology-Aware Schedulingkueue.sigs.k8s.ioRack and block labels, required versus preferred topology per pod set, capacity per domain, and replacing a single failed node without moving the rest of the job.
5. Slurm Scheduling Configuration Guideslurm.schedmd.comBackfill that never delays a higher-priority job’s expected start, the need for accurate time limits, bf_window, and max_switch_wait for leaf-switch placement.
6. NVIDIA Multi-Instance GPU User Guidedocs.nvidia.comUp to seven isolated GPU instances with dedicated memory paths, and the H100 80 GB profiles from 1g.10gb (seven per GPU) to 7g.80gb.
7. Dominant Resource Fairness: Fair Allocation of Multiple Resource Types (Ghodsi et al., NSDI 2011)usenix.orgFairness on each user’s dominant share; sharing incentive, strategy-proofness, envy-freeness and Pareto efficiency.
8. Tiresias: A GPU Cluster Manager for Distributed Deep Learning (Gu et al., NSDI 2019)usenix.orgTwo-dimensional attained service (GPUs × time), discretised priority queues to limit preemptions, starvation promotion, and up to 5.5× lower average job completion time.
9. Gandiva: Introspective Cluster Scheduling for Deep Learning (Xiao et al., OSDI 2018)usenix.orgSuspend at mini-batch boundaries, time-slicing, migration for defragmentation, and 26% higher utilisation on a 180-GPU cluster.
10. Large-scale cluster management at Google with Borg (Verma et al., EuroSys 2015)research.googleCells of about 10,000 machines, more than 10,000 task arrivals per minute, priority bands, a separate optimistic scheduler, and tasks that keep running when the master is down.