$2.1M
a year for each 1% of idle GPUs
At on-demand prices (27 Oct 2026) the 4,096 GPUs are worth ≈ $575,000 a day. One percent is 41 GPUs, idle all year.
Eight free nodes, two six-node jobs
Bad: One pod at a time
Each job grabs four of the six nodes it needs. Both wait forever.
Good: All or nothing
Admit one whole job. The other waits, holding nothing.
The big job gets a reservation. Short jobs fill the gap.
Running jobs’ time limits say when J fits, so reserve those blocks from then. Backfill a job only if it delays nobody ahead of it.
2,688 GPUs are free, yet a whole-block job waits.
Each of twelve blocks has four nodes busy with small jobs. Place by best fit, keep small jobs together, and track the largest free shape.
fraud-ml’s 256 guaranteed GPUs sit idle over a long weekend while genai-core queues 600 GPUs of work.
The trick
Lend idle quota as preemptible. When fraud-ml submits, borrowers get SIGTERM, checkpoint and requeue at the front.
70 min
checkpoint interval for a 1,024-GPU run
It fails about every 49.5 hours; assume a 50 s save. Young–Daly’s √(2 × C × M) balances the two, wasting 2.7%, versus 6.6% saving every 6 hours.
Restart without a human
- Keep one hot spare node in every block
- Check health passively, actively and inside the job
- Drain suspect nodes instead of yanking them
- Swap stragglers, say 20% slower than the median
Remember
Admission decides when; placement decides where. Admit whole gangs, pack them under few switches, lend idle quota preemptibly, checkpoint everything.