Design an online judge (LeetCode)

Every evening in placement season, hundreds of thousands of students open a coding-practice site, pick a problem, write a solution in C++, Java or Python, and press Submit. A few seconds later they see a verdict: Accepted, Wrong Answer, Time Limit Exceeded. It looks like a form post. Behind it, you compile and run code written by strangers — any of which may be a fork bomb, a port scanner or an attempt to read the hidden tests — and you must time it so fairly that the same code gets the same verdict on a quiet Tuesday and on contest day. Then, on Sunday at 8:00, 80,000 people start a weekly contest in the same minute and the load jumps fourfold in three minutes: a tatkal rush in which every request is a small batch job. This problem designs a generic coding practice and contest platform in the style of LeetCode, Codeforces or HackerRank. Nothing here describes any one company’s internals.

Understand the problem

An online judge is a batch-job system wearing a website. The website part — browse problems, read a statement, look at your history — is ordinary and read-heavy. The judge is not: every Submit is a short job that runs hostile code against secret data and must return a verdict that is both fast and fair. Say this split out loud early, because it tells the interviewer where you will spend your time.

Functional requirements

1. Users should be able to browse problems and submit code. Filter about 3,000 problems by topic and difficulty, read the statement, and submit a solution in any of about 18 languages.
2. Users should get a verdict against hidden tests. Accepted, Wrong Answer, Time Limit Exceeded, Memory Limit Exceeded, Runtime Error or Compilation Error, with the running time and memory used — and see their past submissions.
3. Users should be able to run code on custom input (the Run button) and compare their output with the expected output, without it counting as a submission.
4. Users should be able to compete in timed contests with a live leaderboard, ranked ICPC-style by problems solved and then by penalty time.

Non-functional requirements

• Scale. 20M registered users, 3M daily active. About 4M Submits and 6M Runs a day — 10M judge jobs — with an evening peak of 3× the average. Placement season (August to December) can double that.
• Contest spikes. A weekly 90-minute contest with 80,000 participants. Submissions for the first problem pile into minutes 3–10, and there is a second rush in the last 10 minutes.
• Verdict latency. For problems with time limits up to 2 s: p50 under 4 s and p99 under 15 s from click to verdict, including during contests. A Run: p50 under 3 s.
• Isolation. Submitted code cannot reach the network, the host, the expected outputs, other users’ code, or the timing of other runs.
• Fairness. The same code on the same tests gets the same verdict. Measured CPU time varies by no more than a few percent between runs, and time limits per language are published.
• Durability and availability. An acknowledged submission is never lost; if a judge crashes, it is judged again. The contest submit path targets 99.99% during contest windows; browsing may degrade first.
• Leaderboard freshness. A verdict appears on the contest leaderboard within 5 s, and the board stays correct after rejudges.

Out of scope

• Plagiarism detection — covered briefly as a follow-up.
• Ratings after contests, discussion forums and editorials.
• Proctored hiring tests with webcams and browser lockdown.
• IDE features such as autocomplete and debugging.
• The authoring tool for problem setters, beyond uploading and checking a problem package.

The set-up

Plan the approach

Draw two halves joined by a queue: a read-heavy website, and an asynchronous judging pipeline. Satisfy the four requirements simply, then spend the deep dives where judges succeed or fail: the sandbox, fair timing, contest spikes, test data, verdict delivery and the leaderboard.

Core entities

• Problem version. Statement, time and memory limits, checker, scoring style and a test manifest. Versions are immutable: fixing one test creates version 8, and every submission records which version judged it.
• Test bundle. Input and expected-output files, each named by its SHA-256 hash, plus an optional checker or interactor program.
• Language. A sandbox image with a pinned compiler or runtime version, compile flags and a time multiplier.
• Submission. User, problem version, language, source hash, mode (submit or run), optional contest, status, verdict, slowest test time, peak memory, first failing test and a judge attempt number.
• Contest and standing. Start, end and freeze times, scoring rule and penalty; per participant, problems solved, penalty minutes and the time of the last accepted solution.

A problem version, written as configuration:

problem: shortest-bus-route
version: 7
time_limit_ms: 1000              # base limit, for C++
memory_limit_mb: 256
language_multiplier: { cpp: 1.0, java: 2.0, python3: 5.0 }
checker: { type: float, abs_or_rel_error: 1e-6 }
tests: sha256:4be1c09e...        # manifest: 60 (input, output) hash pairs
scoring: icpc                    # stop at the first failing test

API

Submit. The client sends an idempotency key it generated when the user clicked, so a double-click or a retry over a flaky connection creates one submission, not two.

POST /v1/problems/shortest-bus-route/submissions
Idempotency-Key: 6f1c2a9e-0b7d-4c55-9a51-2d3e8f0a7b14
Content-Type: application/json

{
  "language": "cpp17",
  "source": "#include <bits/stdc++.h>\n...",
  "contest_id": "weekly-412"
}

{
  "submission_id": "s_8Kq2Wm",
  "status": "QUEUED",
  "events_url": "/v1/submissions/s_8Kq2Wm/events"
}

The response is 202 Accepted: the work has been accepted, not done. Read the result with a plain GET, or subscribe to events_url for live progress:

GET /v1/submissions/s_8Kq2Wm

{
  "status": "JUDGED",
  "verdict": "WRONG_ANSWER",
  "failed_test": 14,
  "tests_total": 60,
  "max_time_ms": 212,
  "peak_memory_kb": 18432,
  "problem_version": 7
}

In a contest, failed_test is only a number. The hidden input never leaves the judge; in practice mode you may choose to show the input of a small failing test.

Run on custom input is a separate resource, because it has different limits, priority and retention:

POST /v1/problems/shortest-bus-route/runs
Content-Type: application/json

{ "language": "python3", "source": "...", "stdin": "3 2\n1 2\n2 3\n" }

The rest is ordinary:

GET  /v1/problems?tag=graphs&difficulty=medium&cursor=...   list, with solved flags
GET  /v1/submissions/s_8Kq2Wm/events                        SSE: state changes and progress
GET  /v1/contests/weekly-412/standings?page=1               cached page of 50 rows
GET  /v1/contests/weekly-412/standings/me                   my row and rank
POST /v1/admin/problems/shortest-bus-route/versions         setters upload a package
POST /v1/admin/contests/weekly-412/rejudge                  problem, filters, preview=true

Back-of-envelope estimates

Size the judge in core-seconds, not requests per second. (The method is in back-of-envelope estimation.)

Normal days. 10M jobs ÷ 86,400 s ≈ 116 jobs/s on average, and about 350 jobs/s at the evening peak.

Contest. Assume each of the 80,000 participants submits 6 times and clicks Run 10 times. That is 480,000 Submits and 800,000 Runs in 5,400 s: 89 Submits/s and 148 Runs/s on average. The busiest minutes run at about 4× the contest average: ≈ 360 Submits/s and ≈ 600 Runs/s.

Cost per job. A Submit compiles (about 1 s for C++ or Java, less for Python) and runs up to 60 tests. Many submissions fail early and stop, so on average about 40 tests run, each costing about 50 ms including sandbox setup: 1 s + 40 × 50 ms = ≈ 3 core-seconds per Submit. A Run compiles and executes two or three small inputs: ≈ 1.5 core-seconds.

Cores at the contest peak. Contest work: 360 × 3 + 600 × 1.5 = 1,080 + 900 ≈ 2,000 busy cores. Assume, conservatively, that practice traffic is at its evening peak at the same time: 350 jobs/s at an average of 2.1 core-seconds (40% Submits at 3, 60% Runs at 1.5) ≈ 700 more. Total ≈ 2,700 busy cores. Keep utilisation near 70% so queues stay short: 2,700 ÷ 0.7 ≈ 3,900 judge cores, about 240 hosts with 16 usable physical cores each. A normal evening needs 700 ÷ 0.7 ≈ 1,000 cores, about 63 hosts. A contest needs about 4× the normal peak, for about two hours a week.

Storage. Source code: 4M Submits × 2 KB = 8 GB a day, about 3 TB a year; metadata rows add about 2 GB a day. Runs are deleted after 24 hours. Test data: 3,000 problems × 40 MB on average ≈ 120 GB, with a few problems at 1–2 GB; keeping old versions, plan for 400 GB of object storage. A contest’s four problems total about 1 GB.

Prefetch. Copying 1 GB of contest data to 240 hosts is 240 GB. At about 1 Gbit/s per host, each copy takes about 8 s — trivial 30 minutes before the start, but a stampede if all 240 hosts miss their caches in the first second of the contest.

Leaderboard. At most one update per contest verdict: ≤ 360 writes/s. Reads: 80,000 participants refreshing every 10 s is 8,000 reads/s, nearly all for page 1 or their own row.

High-level design

Build one requirement at a time. By the end of this section the judge works for one region at modest traffic.

Users browse problems and submit code

Browsing is a read problem. A statement only changes when a new problem version is published, so render it to HTML once and serve it from a CDN. The problem list is tiny — 3,000 rows — so the problem service keeps it in memory and filters there. The “solved” tick next to each problem is per user: a 3,000-bit bitmap is 375 bytes, so even 20M users need only 7.5 GB, which fits in a cache.

Submitting is the only write. The submission service checks that the language is enabled, the source is under 64 KB and the contest (if any) is open. It stores the row and the source, then returns 202.

Users get a verdict against hidden tests

Never run submitted code in the API process: one while (fork()); would take the API down with it. Treat each submission as a long-running job. The submission service inserts the row with status QUEUED and puts the submission ID on a judge queue. A judge agent on a separate, dedicated host pulls one job when it has a free core, so a busy fleet simply pulls more slowly — that is natural backpressure. For each job the agent:

1. Fetches the source and makes sure the problem’s test files are in its local cache.
2. Compiles inside a sandbox, with its own limits. A compile error ends the job.
3. Runs the binary once per test, inside a sandbox with time, memory and output limits.
4. Hands the output to a checker, which compares it with the expected output.
5. Stops at the first test that is not accepted, and writes the verdict, the slowest time and the peak memory.

Stopping early is allowed by contest rules: the ICPC contest-system requirements say a judge needs to run only as many test files as it takes to find the first one not accepted. The order of checks matters too. Following the same document, a compile failure is a Compilation Error; exceeding memory or crashing before the time limit is a Runtime Error; running past the limit is Time Limit Exceeded; too much output, or output the checker rejects, is Wrong Answer. Practice platforms usually report Memory Limit Exceeded separately, because it is more useful to a learner than a generic crash.

Why a queue rather than calling a judge directly? It absorbs bursts the fleet cannot serve yet, lets any judge take any job, and turns a crashed judge into a redelivered message instead of a lost submission. More on the trade-offs in message queues. For now the client polls GET /v1/submissions/s_8Kq2Wm every second; a deep dive replaces that with push.

Users run code on custom input

A Run is the same pipeline with a different job: the user’s own input instead of hidden tests, and the output returned instead of a verdict. To show an expected output next to theirs, run the problem’s reference solution on the same input. Two details make that safe and cheap. First, run the problem’s input validator before anything else: reference solutions assume the constraints in the statement, and an input with n = 10⁹ where the limit is 10⁵ could make the reference itself time out. Second, cache the reference output by problem version and the SHA-256 of the input; most Runs use the sample input, so most lookups hit.

Runs never count toward statistics or contest penalties, are kept for 24 hours, and ride a lower-priority lane than Submits.

Users compete in timed contests with a live leaderboard

A contest service holds the schedule, registrations and which problems belong to which contest. Statements are rendered in advance but published only at the start time. Scoring follows the ICPC rules as written in the contest-system requirements: contest time is counted in whole minutes, rounded down; a solved problem costs the minutes until its first accepted submission plus a fixed penalty for each earlier rejected attempt (20 minutes is the usual value, and DMOJ’s default); compile errors cost nothing; and teams rank by problems solved (more is better), then total penalty (less is better), then the time of their last accepted solution (earlier is better).

A scorer consumes verdict events for contest submissions, rebuilds the affected participant’s row from the database, and writes it into a Redis sorted set — one per contest — whose score encodes all three sort keys in a single number. A leaderboard API reads pages and ranks from Redis, and a short-lived page cache absorbs the 8,000 reads a second.

You now have a working judge: it is correct, it isolates the API from user code, and it ranks contests. What it does not yet do is survive hostile code, measure time fairly, or hold up when 80,000 people press Submit in the same three minutes. That is what the deep dives are for.

Deep dives

How do we run untrusted code safely?

List what a hostile submission will try, because each item needs its own defence: a fork bomb; allocating all memory; printing gigabytes; filling the disk; opening a socket to leak the tests or attack your internal network; reading /etc, the expected outputs or someone else’s code; smuggling data between two runs through anything the kernel does not separate; and exploiting a kernel bug to escape altogether. The compiler is attack surface too — #include "/dev/urandom" or a template that expands forever — so compilation runs in the sandbox, with its own time and memory limits.

Bad: run it on the API server, or as a child process with a timeout. It is simple, and nothing else about it is good. A fork bomb takes down the machine, open("/etc/passwd") works, sockets work, and the process can read every file the server can.

Good: a locked-down container per submission. No network, a read-only root filesystem, all capabilities dropped, a non-root user, the runtime’s default seccomp profile, and cgroup limits on memory and process count. You gain standard tooling and decent isolation. You pay with a shared kernel: every system call goes straight to the host kernel, so one kernel bug reachable through the allowed calls is a host compromise — and a judge host holds the hidden tests of every cached problem plus other people’s code in flight. Container runtimes are also tuned for compatibility with ordinary software, not for code written to attack them.

Great: two layers — a kernel boundary per judge slot, and a contest-grade jail inside it.

• Outer layer: a microVM or a user-space kernel. A Firecracker microVM puts hardware virtualisation (KVM) between the submission and the host, and its virtual machine monitor exposes very few devices. Its published specification promises at most 125 ms from the start call to the guest’s init, at most 5 MiB of monitor memory for a one-vCPU, 128 MiB guest, and guest compute above 95% of bare metal. A fresh microVM per submission costs up to 125 ms of boot against roughly 3 s of judging — about 4%. gVisor takes a different route: its Sentry, an application kernel written in Go, answers the program’s system calls itself instead of passing them to the host, and a separate Gofer process mediates file access. The price is a higher cost per system call and imperfect compatibility; gVisor’s own documentation warns that system-call-heavy workloads can perform poorly.
• Inner layer: a judge sandbox. Inside the guest, run each compile and each test under isolate, the sandbox from the IOI community that CMS and Judge0 build on, or under nsjail. By default isolate gives the process a new network namespace containing only loopback, a mount namespace with read-only bind mounts, a /proc that shows only its own processes, a per-box user ID and one process unless you allow more. It caps file size and output, puts memory and process limits in a cgroup v2 group, and blocks the few system calls that could leak data between sandboxes: keyrings, VM sockets, file locks and io_uring. nsjail combines the same namespaces with seccomp-bpf policies written in its Kafel language.
• Around both: a boring, disposable host. Judge hosts live in their own subnet with no route to internal services. The agent, outside the VM, holds the only credentials — read access to one queue and one bucket prefix — and hosts are rebuilt daily.

The recommendation: a fresh microVM per submission as the outer wall, isolate inside it for every compile and test, on a host that holds nothing worth stealing. gVisor is a reasonable outer wall where you cannot get nested virtualisation, as long as you account for its system-call cost when you set time limits. You pay for this with operational complexity — guest kernel images to maintain, a microVM to boot per submission — and you gain a design in which a single bug in any one layer does not hand over the host.

How do we make time limits fair and repeatable?

A judge has two clocks. CPU time counts only the moments the program is actually running on a processor, summed over all its threads. Wall time is the clock on the wall, from start to exit. You need both, because each catches what the other misses.

isolate’s manual gives the standard advice: make CPU time the main limit, and set the wall limit much higher as a guard against programs that sleep or wait for input that never comes. Its --extra-time option lets a program run slightly past the limit, so you can report “1.08 s” instead of a bare TLE. Judge0’s defaults follow the same shape — 5 s of CPU and 10 s of wall time. A good rule: wall limit = 2 × CPU limit + 1 s.

CPU time is fairer than wall time, but it is not immune to noise. The same instructions take longer when another process thrashes the shared cache, when the core’s SMT sibling is busy, when the frequency drops, or when the scheduler moves the program to another core.

Bad: shared cloud vCPUs, several runs per core. It is cheap and keeps utilisation high. Judge0, for example, defaults to two workers per CPU, which suits a general code-execution API. But a contestant whose solution needs 0.9 s against a 1 s limit gets Accepted on one submit and TLE on the next — the most demoralising bug a judge can have.

Good: one run per vCPU on dedicated hosts. No other customers share the machine, so the worst noise is gone. Two runs on SMT siblings of one physical core still slow each other down.

Great: one run per physical core, configured for repeatability. isolate’s manual lists the settings, and they are worth repeating in an interview: pin each sandbox to one core so the scheduler cannot migrate it; use the performance frequency governor and turn turbo off; on CPUs with mixed core types, use only one type; turn off address-space randomisation, transparent huge pages and swap. Leave the SMT sibling idle or disable SMT; nsjail can also turn on the kernel’s core scheduling, which stops untrusted code from running on an SMT sibling alongside other work. Keep each contest on one machine class, and reserve two cores per host for the agent and the operating system. The cost is real: you buy roughly twice the vCPUs for the same number of judge slots. What you gain is a verdict that does not depend on luck.

Measure with the kernel, not with a stopwatch. Put each run in its own cgroup v2 group. Its cpu.stat file reports usage_usec, user_usec and system_usec for every process and thread in the group, so a program cannot hide work in a child process. memory.max sets the hard memory limit and memory.peak reports the high-water mark for the MLE check. pids.max caps processes and threads; the kernel documentation notes that a fork bomb usually runs out of process slots before it runs out of memory. isolate wraps all this and writes a small meta file per run with time, time-wall, max-rss, cg-mem and a status (TO for timeout, SG for a fatal signal, RE for a non-zero exit). Reporting memory this way needs Linux 5.19 or newer.

Four policies finish the job:

• Limits are per test. TLE means some test exceeded the limit; the reported runtime is the slowest test, not the sum.
• Re-run near-misses. If a test lands within 10% over the limit, run it up to twice more and keep the minimum. Noise only ever adds time, so the minimum is the best estimate of the program’s real cost. Averaging, as Judge0’s optional repeated runs do, mixes the noise back in. You pay a little CPU on borderline verdicts.
• Publish language multipliers. Interpreted and JIT-compiled languages are slower and pay start-up costs. HackerRank, for example, publishes per-language limits of 2 s for C and C++, 4 s for Java and 10 s for Python 3 — multipliers of 2× and 5×. Use a table like that as the default, let problem setters override it after timing reference solutions in each language, and print the effective limit on the statement. Do not try to subtract JVM start-up: measure it into the multiplier.
• Calibrate hardware. Before a new machine class judges anything, run a benchmark suite of reference solutions on it and compare. Rejudges run on the same class as the original, and statistics such as “faster than 80% of submissions” only compare runs from the same class.

How do we absorb a contest-start spike?

The estimates say a contest needs about 2,700 busy cores, four times a normal evening, and most of that arrives within three minutes of the start. Think of the 10:00 tatkal window: everyone knows when it opens, and everyone arrives at once.

Bad: one FIFO queue and autoscaling on queue depth. It is simple and cheap on quiet days. But a contest submission waits behind thousands of practice Runs, and the autoscaler only reacts once the backlog already exists. New hosts then need minutes to boot, pull 18 language images and fetch test data. With about 1,000 cores missing for six minutes, the backlog is roughly 1,000 × 360 s = 360,000 core-seconds — more than 90 s of work for the full 3,900-core fleet — piled up exactly when every participant is watching their verdict.

Good: priority lanes and separate pools. Split the queue into lanes: P0 contest Submits, P1 contest Runs, P2 practice Submits, P3 practice Runs, P4 rejudges and other batch work. Then split the fleet, so that one kind of work cannot take another’s cores — the bulkhead pattern. A reserved contest pool takes only P0 and P1. A shared pool serves P2 and P3 and lends idle cores to P0. A batch pool runs rejudges. The ICPC requirements make two rules explicit, and both matter here: no submission may wait while a judge sits idle, and no single team may starve the others. So add per-user caps — one Submit and two Runs in flight per user; extra clicks wait behind that user’s own work. You pay for idle reserved capacity and for tuning the lane weights.

Great: lanes and pools, plus capacity that arrives before the crowd. Contests are on the calendar, so you do not have to guess. Thirty minutes before the start, a scheduler sizes the contest pool from registrations and the last few weeks’ submissions per participant, boots the hosts, warms the language images, prefetches the contest’s test data and sends one canary submission per language through every host. Capacity is released after the contest ends. This is autoscaling driven by a forecast rather than by a metric, with the reactive autoscaler kept as a backstop for surprises.

Then cut the work itself:

• Stop at the first failing test. It is what made the average Submit cost 40 tests instead of 60.
• Cache compiled binaries by the SHA-256 of source, language, compiler version and flags. A user who clicks Run and then Submit on the same code compiles once, saving about a third of the Submit’s cost.
• Reuse verdicts for identical resubmissions. Same source hash, language and problem version means the same verdict, because timing is repeatable. In a contest, still record the attempt — it still costs penalty minutes.
• Spread slow tests across cores. A correct but slow solution that needs 1.5 s on each of 60 tests takes 90 s on one core. Spreading the tests over four cores of the same host returns the verdict in about 23 s for the same core-seconds; when one test fails, cancel the others.
• Degrade honestly. If the contest lanes wait longer than 10 s, admission returns “busy, try again in 30 s” for practice Runs, shows an estimated wait on practice Submits, and limits contest Runs to the sample tests. Contest Submits are never shed. This is load shedding with a priority order you chose in advance.

Pick a queue that acknowledges each message. Judge jobs take seconds and vary a lot, so you want per-message acknowledgements and a visibility timeout: a worker that dies mid-job simply lets the message reappear. An SQS-, RabbitMQ- or Redis Streams-style queue fits. A log with ordered partitions, such as Kafka, is awkward here, because one 30-second job can hold up everything behind it in its partition unless you build per-message acknowledgement on top. Either way, the database stays the source of truth: a sweeper re-enqueues rows that have waited more than a minute past their deadline without a verdict, and a submission that has crashed three judges becomes “internal error” with an alert — the ICPC requirements call this a Judging Error — instead of a poison message that loops forever.

How do test cases get to the workers fast and safely?

Test data is the platform’s crown jewels: leak a contest’s tests and the contest is void. It is also bulky — about 40 MB per problem on average, 250 MB for a typical contest problem, and 1–2 GB for a few.

Bad: store tests in the database, or bake them into the worker image. Database blobs bloat backups and replicas. A baked image means a full rebuild and redeploy to fix one wrong output, and every host carries every problem.

Good: object storage, downloaded per submission. Versioned and cheap, but 360 Submits a second × 250 MB is 90 GB/s of downloads. No.

Great: versioned, content-addressed bundles cached on each host. When a setter uploads a package, a package check runs the input validator on every input, requires every reference solution to pass and every known-wrong solution to fail, and times the reference solutions in each language to suggest limits. Each file is then stored in object storage under its SHA-256 hash, and the problem version gets a manifest — the list of (input hash, output hash) pairs — in the database. The judge agent reads the manifest, looks each hash up in a least-recently-used cache on local NVMe, downloads only what is missing, and verifies every download against its hash before use (a checksum you get for free). Fixing one test of 60 uploads one new file; every other hash is unchanged and stays cached. Contest bundles are prefetched 30 minutes before the start, so no host misses at 8:00:00.

Inside the sandbox, only the current test’s input is visible — bind-mounted read-only or piped to standard input. Expected outputs stay outside. Judge hosts get read-only credentials scoped to the bundle prefix, and contest bundles are encrypted at rest until the start.

Checkers decide what “correct” means. The default compares token by token, ignoring differences in whitespace — DMOJ’s standard checker does exactly this. Floating-point answers need a tolerance; DMOJ’s floats checker accepts an absolute or relative error of 10⁻⁶ by default. Problems with many valid answers (“print any shortest path”) need a special checker written by the setter, which runs sandboxed as the warning above explains.

Interactive problems replace the expected output with a program. The judge starts an interactor and the solution in separate sandboxes and connects them with two pipes: the interactor asks, the solution answers, and the interactor decides the verdict. CPU time is measured for the solution only, and the wall limit catches the classic deadlock where both sides wait to read — usually because the solution forgot to flush its output.

How does the user get the verdict?

Model the submission as a state machine with one exit. Every path — success, compile error, crashed judge — ends in a stored verdict, and a rejudge is simply a new attempt that starts from QUEUED.

Write states, not progress, to the database. A submission changes state three or four times; “running test 17 of 60” changes 60 times. At 360 Submits/s × 40 tests that is 14,400 progress updates a second, so progress goes only over pub/sub, throttled to two updates per second per submission. State changes are conditional writes keyed by the attempt number, so a judge that was presumed dead and comes back late cannot overwrite a newer result:

UPDATE submissions
SET    status = 'JUDGED', verdict = 'AC', max_time_ms = 212, peak_memory_kb = 18432
WHERE  id = 's_8Kq2Wm' AND judge_attempt = 2 AND status <> 'JUDGED';

Now, how does the browser hear about it?

Bad: hold the HTTP request open until the verdict. Requests that last 15 seconds tie up API threads, die on every mobile network blip, and tempt clients to retry — which creates duplicate submissions.

Good: poll with backoff. At the contest peak about 2,600 jobs are in flight (360 Submits/s × about 4 s, plus 600 Runs/s × about 2 s); polling each once a second is about 2,600 cheap reads a second, served from a cache. Simple and robust. The cost is up to a second of extra latency and almost no live progress.

Great: push over SSE, keep polling as the fallback. The contest page already needs a live channel for announcements and clarifications. Server-sent events are one-way, run over plain HTTP and reconnect by themselves, sending the last event ID they saw; on reconnect the push gateway first sends the current state from the cache, so nothing is missed. 80,000 open connections are spread over a handful of push gateway nodes, each subscribed to per-user channels. You pay for operating long-lived connections — load-balancer idle timeouts, draining during deploys — and you gain instant verdicts and a progress bar. More patterns in real-time updates.

Make submit idempotent. The Idempotency-Key from the API section is stored with a unique index on (user, key). A double-click or a retried request returns the same submission ID — and, in a contest, costs the same single attempt. See idempotency.

How do we keep the contest leaderboard live and correct?

The rules come first. ICPC-style ranking, as above: solved (descending), penalty minutes (ascending), last accepted time (ascending). IOI-style contests instead give partial points per subtask; in DMOJ’s IOI format a problem’s score is the best result on each subtask across all of a participant’s submissions. Hide the difference behind one function — a participant’s verdict history in, a sort key out — and everything below works for both.

Bad: rank with SQL on every page view. ORDER BY solved DESC, penalty ASC over 80,000 rows, 8,000 times a second, while the same table takes verdict writes. The database melts at exactly the wrong moment.

Good: a Redis sorted set updated incrementally. On each accepted verdict, add one solved problem and its penalty with ZINCRBY. Reads are fast, but increments are fragile: a replayed event counts twice, an out-of-order event applies the wrong penalty, and a rejudge needs an exact inverse operation.

Great: rebuild the row and overwrite it. For each verdict event, the scorer reads that participant’s contest submissions from the database — a few dozen rows at most — runs the scoring function, and writes the result with ZADD. The write is idempotent and order-insensitive, so replays and rejudges are safe by construction. The cost is one small indexed read per verdict, at most 360 a second. Encode all three sort keys in one number, so that a higher score always means a better rank:

score = solved × 10^9 − penalty_minutes × 10^4 − last_accepted_seconds

3 solved, 87 penalty minutes, last AC at 61:20 (3,680 s):
3,000,000,000 − 870,000 − 3,680 = 2,999,126,320

Check the ranges, or the fields bleed into each other. Even with 50 rejected attempts on each of four problems, penalty is at most 4 × (90 + 20 × 50) = 4,360 minutes, so penalty × 10⁴ stays below 10⁹. The last accepted time is at most 5,400 s, below 10⁴. The largest score, about 4 × 10⁹, is far below 2⁵³, and Redis stores sorted-set scores as 64-bit doubles that hold every integer up to 2⁵³ exactly.

Reading the board:

ZADD   lb:weekly-412 2999126320 u_4821              # scorer, O(log N)
ZRANGE lb:weekly-412 0 49 REV WITHSCORES            # page 1
ZCOUNT lb:weekly-412 (2999126320 +inf               # my rank = this + 1

Equal scores are real ties, and Redis would order them alphabetically by member, which is not a ranking. So compute rank as one plus the number of strictly higher scores: tied participants share a rank. Eighty thousand members take a few megabytes, so one key per contest is plenty; many concurrent contests spread across Redis primaries by contest ID. Page 1 and other popular pages are rendered every two seconds into a cache, which turns 8,000 reads a second into a trickle. For a global, all-time board with millions of players, see design a live leaderboard.

Freeze. ICPC-style contests freeze the public scoreboard near the end. The contest-system requirements say the freeze must happen automatically at the configured time and that submissions after it show as pending on the frozen board. Keep two keys: the live set the scorer always updates, and a public copy taken at the freeze time. Participants still see their own verdicts; everyone else sees “pending” until the reveal.

Rejudges. Because the board is derived from verdicts, a rejudge is just new verdicts flowing through the same scorer. The same requirements ask for a preview of rejudge results before they are applied — show which verdicts and ranks would change, then commit.

The final design

The submit path is a short write and a queue; everything slow happens on judge hosts; everything users watch is pushed or cached.

Follow one contest submission through the system:

1. Submit. The browser posts code with an idempotency key. The gateway authenticates the user and applies a per-user rate limit.
2. Store and enqueue. The submission service inserts the row as QUEUED — a duplicate key returns the existing row — and admission puts the ID on lane P0.
3. Pull. A judge agent in the contest pool takes the job when one of its pinned cores is free.
4. Judge. In a fresh microVM, isolate compiles the code, then runs each test with CPU, wall, memory, process and output limits, reading the input from the host’s hash-checked cache. A sandboxed checker compares outputs; judging stops at the first failure.
5. Record. The agent writes the verdict with a conditional update keyed by the attempt number, then publishes state changes and progress as verdict events.
6. Notify. The push gateway forwards the events to the browser over SSE; a reconnecting or SSE-less client polls the cached status instead.
7. Rank. The scorer rebuilds the participant’s row from the database and overwrites their score in the contest’s Redis sorted set; leaderboard pages are re-rendered every two seconds.
8. Prepare. Thirty minutes before each contest, the scheduler scales the contest pool to the forecast, warms images and prefetches the contest’s test data.

Check it against the requirements. A typical Submit is about 1 s of compile plus about 2 s of tests on a free, pinned core: under 4 s at p50, with the contest pool sized so queueing stays near zero. Hostile code faces five layers, and the expected outputs are never inside them. The same code gets the same verdict because each run owns a quiet physical core. Contest load meets capacity that was ready 30 minutes earlier, and practice traffic is shed first. Acknowledged submissions live in the database before the queue sees them, and the leaderboard is a pure function of stored verdicts, refreshed within seconds.

What interviewers look for by level

Follow-up questions

What if a problem has a 2 GB test set, or needs a GPU?

Pin it to a dedicated pool of hosts that keep its data permanently on local NVMe, and mount the files read-only instead of copying them. Where the tests come from a generator, store the generator and its seeds, generate once per host and cache by hash. GPU problems are a different product: a separate pool with its own scheduler, time measured in GPU-seconds, and far fewer concurrent runs — see design a GPU job scheduler.

What if we add a new language, such as Rust?

Build a sandbox image with a pinned compiler version and flags, and tune the sandbox for it (Rust’s compiler starts several threads, so the compile step needs a higher process limit than the run step). Calibrate its multiplier by timing reference solutions on every problem, shadow-judge a sample of past submissions, then enable it behind a flag — practice first, contests later.

What if someone submits a fork bomb, opens a socket or reads /proc?

The fork bomb hits pids.max; new forks fail and the run ends as a Runtime Error. The socket has no network to reach: the namespace holds only loopback. /proc shows only the sandbox’s own processes, and the tests are not in its filesystem. The ICPC requirements allow a Security Violation verdict for prohibited operations; log it, and send repeat offenders for review.

What if a test-data bug forces a rejudge of 40,000 submissions mid-contest?

Publish the fixed tests as a new problem version, then rejudge on the batch pool: 40,000 × 3 core-seconds is 120,000 core-seconds, or four minutes on 500 spare cores, while live submissions keep their reserved pool. Preview which verdicts and ranks change, apply them, let the scorer rebuild the board, and announce it — extending the contest if the bug cost people time.

What if participants are spread across the world?

Give each contest a home region that holds its submissions, queue and leaderboard. Source code and verdicts are tiny, so a user 200 ms away loses almost nothing on a 3-second judge. Serve statements and leaderboard pages from the CDN everywhere. If the home region’s fleet is short, overflow judge capacity can come from another region with the contest’s test data prefetched — see multi-region and active-active.

What if two contestants submit the same code?

Exact copies are easy to spot by source hash. Renamed variables and reordered functions need token or syntax-tree fingerprints, the idea behind tools such as MOSS. Run it after the contest, per problem, comparing every pair of accepted solutions, and send the most similar pairs to human review; it is offline, so it never touches the judging path.

How to say it in the interview

“I’ll design a coding practice and contest platform: about 10 million judge jobs a day, and weekly contests where 80,000 people start at once. The two hard parts are running untrusted code safely and measuring time fairly while load jumps fourfold in three minutes.

The web tier never runs code. Submit stores the row and the source, puts the ID on a priority queue and returns 202. Judge agents on dedicated hosts pull jobs. Each submission gets a fresh microVM as the kernel boundary, and inside it isolate runs the compile and each test with namespaces, no network, seccomp, and cgroup limits on memory, processes and output. Expected outputs never enter the sandbox; a separately sandboxed checker compares them. We stop at the first failing test.

For fairness, each run owns one pinned physical core with SMT and turbo off. I limit CPU time and wall time, read CPU from cgroup accounting, re-run near-misses and keep the minimum, and publish per-language multipliers. The price is hardware — one run per physical core roughly doubles the vCPUs per judge slot — and I think a judge that flips verdicts is worse.

Sizing: a Submit is about 3 core-seconds and a Run about 1.5, so a contest peak needs about 2,700 busy cores — four times a normal evening. Contests are scheduled, so I scale the contest pool 30 minutes early, warm images and prefetch test data. Separate lanes and pools keep practice Runs and rejudges from ever taking a contest core, and under pressure practice Runs are shed first.

Test data is versioned and content-addressed in object storage, cached on local NVMe and hash-checked. Verdicts are a state machine in Postgres, with progress pushed over SSE and polling as a fallback; submit is idempotent.

The leaderboard is one Redis sorted set per contest with a composite score — solved, then penalty, then last accepted time — rebuilt from stored verdicts on every event, so replays, freezes and rejudges stay correct.”

Concepts used in this design

• Long-running jobs — every submission is an asynchronous job with a status to poll or push.
• Message queues — per-message acknowledgements and redelivery for judge jobs.
• Backpressure and load shedding — pull-based workers and shedding practice Runs first.
• Bulkheads — separate pools for contests, practice and rejudges.
• Autoscaling done right — scheduled pre-scaling with reactive scaling as a backstop.
• Multi-tenancy and isolation — the layered sandbox between strangers’ code.
• Noisy neighbour — why shared cores make time limits unfair.
• Blob storage — content-addressed test bundles.
• Checksums — verifying every cached test file.
• Content delivery network — statements and leaderboard pages at the edge.
• Real-time connections — SSE for verdicts and progress.
• Idempotency — one submission per click, one penalty per attempt.
• Redis — sorted sets for the contest leaderboard.
• Design a live leaderboard — the same structure at global scale.
• Back-of-envelope estimation — sizing in core-seconds.

References and further reading

1. isolate(1) manual page — the IOI sandboxucw.czCPU time versus wall-time limits and extra time, the meta file (time, time-wall, max-rss, cg-mem), default network namespace with only loopback, cgroup v2 mode, blocked cross-sandbox syscalls, and the reproducibility checklist (pin one core, no turbo, no ASLR, no swap).
2. Judge0 configuration file (judge0.conf)github.comAn open-source code execution system built on isolate: default 5 s CPU and 10 s wall limits, 128 MB memory, 60 processes, 1 MB file size, network off, two workers per CPU, a bounded queue and an option to run a program several times.
3. gVisor documentation: What is gVisor?gvisor.devThe Sentry as a user-space application kernel that does not pass system calls through, the Gofer for file access, and the price: higher per-system-call overhead and imperfect compatibility.
4. Firecracker specification (SPECIFICATION.md)github.comAt most 125 ms from InstanceStart to guest init, at most 5 MiB VMM memory overhead for a 1 vCPU, 128 MiB microVM, and guest compute above 95% of bare metal.
5. nsjail (GitHub README)github.comA process jail combining Linux namespaces, Kafel seccomp-bpf policies, rlimits and cgroup v1/v2 limits, with optional core scheduling for SMT isolation.
6. Linux kernel documentation: Control Group v2docs.kernel.orgcpu.stat usage_usec covering every process in the group, memory.max and memory.peak, and pids.max as the defence against fork bombs.
7. ICPC CCS System Requirements (2023-06)ccs-specs.icpc.ioJudgement order and penalties, stopping at the first non-accepted test file, minute-rounded penalty time and ranking order, scoreboard freeze, rejudge previews and preventing one team from starving the judges.
8. DMOJ documentation (contest formats, checkers, problem format)github.comICPC format with a 20-minute default penalty, IOI subtask scoring, the standard and floating-point checkers (default 1e-6), pretests that stop judging, and status codes such as OLE.
9. HackerRank: Execution environmentsupport.hackerrank.comPublished per-language time limits (2 s for C and C++, 4 s for Java, 10 s for Python 3) and CPU time counted across all threads.
10. Redis ZADD commandredis.ioSorted-set scores are 64-bit doubles that hold integers exactly up to 2^53, O(log N) inserts, and lexicographic ordering of equal scores.