Design YouTube

YouTube lets anyone upload a video and lets anyone else watch it minutes later, on a phone in a crowded local train or on a 4K television. Behind that simple promise sit three hard problems. Getting a 20 GB recording off a creator’s shaky connection without starting over every time the signal drops. Turning that one file into a dozen streamable versions, quickly and cheaply, more than 20 million times a day. And delivering hundreds of terabits per second to viewers whose tastes follow an extreme long tail: a few videos take most of the views, billions of others are watched a handful of times, and the platform must store and serve both. A fourth problem hides in plain sight — the view counter, which absorbs millions of events a second and still has to be honest.

Understand the problem

YouTube is two products sharing one catalogue: a publishing pipeline for creators and a playback service for viewers. They have opposite shapes. Uploads are relatively few, huge, and can tolerate minutes of delay. Views are many, each request is small (a few seconds of video), and a one-second stall is a failure. Say this early, because it justifies splitting the system along exactly that line.

Functional requirements

1. Creators should be able to upload a video — from a 15-second phone clip to a 12-hour recording — and see it become watchable within minutes.
2. Viewers should be able to watch any video smoothly on any device and network, from a patchy 3G signal to home fibre, with quality adapting as conditions change.
3. Viewers should be able to search and browse videos by title, channel and tags, and see titles, thumbnails and durations instantly.
4. Viewers should be able to see view counts and like videos, with their own likes remembered.

Non-functional requirements

• Fast start, no stalls. Time to first frame under 1 s at p50 and under 3 s at p99 on a reasonable connection; rebuffering under 0.5% of watch time.
• Highly available playback. 99.99% for watching. Uploads and processing target 99.9%: resumable sessions and retries hide short outages from creators.
• Durable uploads. Once an upload is acknowledged, the original is never lost — erasure-coded across failure domains. Renditions can be regenerated from the original, so they can be stored more cheaply.
• Quick processing. A low-resolution version playable within about a minute or two of upload for a 10-minute video, the full HD ladder within about 15 minutes. 4K may take longer; YouTube’s own help pages say a 60-minute 4K video at 30 fps can take up to 4 hours.
• Scale. More than 20 million uploads a day (YouTube’s press page), around a billion hours watched a day (our assumption), and on the order of 200 Tbps of video at the evening peak (derived below).
• Consistency chosen per data type. A creator sees their own title edit immediately (read-your-writes); public view counts may lag by minutes and be corrected later; a user’s like is idempotent and visible to them at once, while the total is eventually consistent.

Out of scope

• Recommendations and the home feed. Ranking is its own system; see design a recommendation feed.
• Live streaming. A different latency regime; covered briefly in the follow-ups.
• Comments, ads and monetisation. Separate services that hang off the video ID.
• Copyright matching internals. We give it a slot in the processing pipeline, but not a design.
• The short-video swipe feed. Its ranking and prefetching problems are covered in design TikTok.
• Search ranking. We index metadata; relevance tuning belongs to how search engines work.

The set-up

Plan the approach

Build the four requirements one at a time with the simplest thing that works, then spend most of the interview in the deep dives. State the shape up front: “an upload path into object storage, an asynchronous processing pipeline, a delivery path through caches, and small metadata and counting services beside them.” The deep dives follow the life of a video: getting the bytes in, encoding them, choosing what to keep, streaming, caching, and counting.

Core entities

• Video. ID, channel, title, description, tags, duration, visibility and a status that moves uploading → processing → ready (or blocked, failed).
• Upload session. Which video it belongs to, total size, part size, which parts have arrived with their checksums, and an expiry time.
• Original. The creator’s file, exactly as uploaded, in object storage. The source of truth for every rendition.
• Rendition. One encoded version — codec, resolution, bitrate — stored as fixed-length segments plus a playlist. A manifest lists all renditions of a video.
• View event and like. A playback start or progress ping from a player; a (user, video) like.

The metadata that must be transactional fits in two small tables:

CREATE TABLE videos (
  video_id     BIGINT PRIMARY KEY,          -- also the shard key
  channel_id   BIGINT NOT NULL,
  title        VARCHAR(200) NOT NULL,
  description  TEXT,
  status       ENUM('uploading','processing','ready','blocked','failed') NOT NULL,
  visibility   ENUM('public','unlisted','private') NOT NULL,
  duration_ms  INT,
  created_at   TIMESTAMP NOT NULL
);

CREATE TABLE renditions (
  video_id      BIGINT NOT NULL,
  rung          VARCHAR(8)  NOT NULL,       -- '360p', '1080p', 'audio'
  codec         VARCHAR(8)  NOT NULL,       -- 'h264', 'vp9', 'av1', 'aac'
  bitrate_kbps  INT NOT NULL,
  storage_tier  ENUM('hot','cold') NOT NULL,
  playlist_path VARCHAR(255) NOT NULL,
  PRIMARY KEY (video_id, rung, codec)
);

API

Start an upload. The client sends metadata and the file’s size and hash; the server creates the video row and an upload session, and hands back one pre-signed URL per part.

POST /v1/uploads
Authorization: Bearer <token>
Content-Type: application/json

{
  "title": "IPL final: last over from every angle",
  "visibility": "public",
  "file": { "size_bytes": 2000000000, "sha256": "9f2c…", "content_type": "video/mp4" }
}

{
  "upload_id": "u9",
  "video_id": "v123",
  "part_size_bytes": 16777216,
  "part_count": 120,
  "part_url_template": "https://upload.example.com/u9/parts/{n}?exp=1793700000&sig=…",
  "expires_at": "2026-11-03T10:00:00Z"
}

Upload, check and complete. Each part is a PUT of 16 MiB to its URL and returns a checksum. After a crash, the client asks which parts arrived. When all are in, it commits the list.

PUT  https://upload.example.com/u9/parts/5?exp=…&sig=…     body: bytes of part 5
     → 200  { "crc32c": "a1b2c3d4" }

GET  /v1/uploads/u9
     → 200  { "received": [1, 2, 3, 4, 6, 7], "missing": [5, 8, …], "expires_at": "…" }

POST /v1/uploads/u9:complete     { "parts": [ { "n": 1, "crc32c": "…" }, … ] }
     → 201  { "video_id": "v123", "status": "processing" }

Watch. The watch page returns metadata plus signed manifest URLs that point at a specific cache. The player then talks only to the CDN.

GET /v1/videos/v123
→ 200
{
  "title": "IPL final: last over from every angle",
  "duration_s": 600,
  "status": "ready",
  "views": 1284000,
  "likes": 51200,
  "manifests": {
    "hls":  "https://edge-bom-07.cdn.example.com/v123/master.m3u8?exp=…&sig=…",
    "dash": "https://edge-bom-07.cdn.example.com/v123/manifest.mpd?exp=…&sig=…"
  }
}

GET https://edge-bom-07.cdn.example.com/v123/720p/seg-00042.m4s     (one 4 s segment)

Count, like, search.

POST   /v1/playback-events        batch of { event_id, video_id, session_id, type: start | progress, position_s }
PUT    /v1/videos/v123/like       → 204   (idempotent: liking twice is still one like)
DELETE /v1/videos/v123/like       → 204
GET    /v1/search?q=ipl+final&lang=hi&cursor=…

Back-of-envelope estimates

Show the arithmetic; three of these numbers decide the architecture. (Method: back-of-envelope estimation.)

Uploads. YouTube’s press page says more than 20 million videos are uploaded on an average day. 20M ÷ 86,400 ≈ 231 uploads/s; with evening peaks and event spikes, plan for 3×, about 700 new uploads/s.

Hours of new video. Assume the average upload is 3 minutes — short clips pull it down, long streams pull it up. 20M × 3 min = 60M minutes = 1M hours of new video a day, about 700 hours every minute.

Originals and ingest. Assume source files average 8 Mbps. One hour: 8 Mb/s × 3,600 s = 28,800 Mb = 3.6 GB. A day: 1M h × 3.6 GB = 3.6 PB of originals. Ingest: 3.6 PB ÷ 86,400 s ≈ 42 GB/s ≈ 330 Gbps on average, roughly 1 Tbps at peak.

Renditions. A six-rung H.264 ladder — 144p at 0.15 Mbps, 240p at 0.35, 360p at 0.7, 480p at 1.3, 720p at 2.5 and 1080p at 5 — sums to 10 Mbps. One hour of every rung: 10 × 3,600 ÷ 8 = 4,500 MB = 4.5 GB, so 4.5 PB a day. With originals: 8.1 PB a day × 365 ≈ 3 EB a year before redundancy, about 4.4 EB of raw disk with erasure coding at 1.5× overhead.

Encoding compute. Assume a fast H.264 ladder costs about 2 core-hours per hour of source. 1M h × 2 = 2M core-hours a day ÷ 24 = ≈ 83,000 cores busy around the clock, about 250,000 with peak and retry headroom. A modern codec at roughly 10× the compute, applied to everything, would need about 830,000 cores on average.

Watching. Assume viewers watch 1B hours a day. 1B ÷ 24 = 41.7M concurrent streams on average. Most viewing is on phones at 480p–720p, so assume 2.5 Mbps on average: 41.7M × 2.5 Mbps ≈ 104 Tbps, about 200 Tbps at the evening peak.

Segment requests. With 4-second segments, each stream fetches one video segment every 4 s: 41.7M ÷ 4 ≈ 10.4M segment requests/s, plus audio.

Views and metadata reads. Assume a view lasts 6 minutes on average: 1B h × 60 ÷ 6 = 10B views a day ≈ 116k/s, about 350k/s at peak. Each view loads the watch page and browsing triples that: about 1M metadata reads/s at peak. Metadata volume is small: 7.3B new videos a year × about 2 KB ≈ 15 TB a year.

Read-to-write ratio. 10B views against 20M uploads is 500 : 1.

High-level design

Build one requirement at a time. By the end of this section the design works end to end; the deep dives then make it fast, cheap and robust.

Creators should be able to upload a video and see it become watchable

Never push gigabytes through your API servers. An upload service creates the video row and a session, and the bytes land in object storage as the original. When the object is complete, storage emits an event onto a queue; transcoder workers pick it up, write segments and manifests for each rendition, and mark the video ready.

Three choices matter already. The video ID exists from the first request, so the creator can edit the title while the file is still uploading. The original is kept forever: every rendition can be rebuilt from it when encoders improve. And the queue decouples a bursty upload rate — every cricket fan posting highlights at 11 pm after a final — from a worker pool sized for the average.

Viewers should be able to watch any video smoothly

A video is not served as one file. Each rendition is cut into short segments (4 seconds here), and a manifest lists the renditions and their segments — an HLS master playlist or a DASH MPD. The player downloads the manifest, then fetches segments one by one, choosing the rendition for each from how fast the last ones arrived. That is adaptive bitrate streaming: a viewer whose train enters a tunnel drops from 720p to 360p instead of staring at a spinner.

1. The player asks the watch API for the video. The API reads the metadata DB (through a cache, added later) and returns the title, counts and signed manifest URLs that expire in a few hours.
2. The player fetches the manifest and then segments from a CDN edge, picking a rendition for each segment.
3. On a cache miss, the edge fetches the segment from the origin — the renditions store — and keeps a copy for the next viewer. See content delivery network.

Segments are immutable — v123/720p/seg-00042.m4s never changes — so caches can keep them for as long as they like. Only manifests and metadata ever need short lifetimes.

Viewers should be able to search and browse videos

Watch pages, channel pages and search results all read video metadata, around 1M times a second at peak. The metadata DB is the source of truth for lookups by ID; a search index answers free-text queries over titles, tags and channel names.

The metadata DB. The data is relational and small — about 15 TB of new rows a year — but the read rate is not. YouTube walked this road itself: its MySQL split into a primary for writes and a replica for reads, then needed more replicas, then sharding when writes outgrew one primary, with shard routing baked into application code. Vitess was built at YouTube in 2010 to move that routing into a proxy, so applications speak plain MySQL and the proxy finds the shard. Copy the end state: shard by video_id, which spreads load evenly and makes the hottest query, “get video v123”, a single-shard lookup. Channel pages ask for “all videos of channel C” across shards, so keep a small lookup table from channel_id to video IDs. More in sharding a relational database.

The cache. A read-through cache keyed by video ID, with a TTL of about a minute and explicit invalidation on edits, absorbs over 99% of reads: 1M reads/s at peak becomes about 10k/s on the database.

The search index. Change data capture streams every committed change into an indexer that updates an inverted index such as Elasticsearch. Search lags the database by seconds, which nobody notices; private and blocked videos are filtered at both index time and query time.

Viewers should be able to see view counts and like videos

A view is an event, not a database update. The player sends a ping when playback starts (and progress pings as it plays). Collectors validate and batch them onto an event log; a counter aggregator turns them into per-video totals every few seconds. A like is different: it is state that belongs to one user, so it is a row keyed by (user, video), and PUT is idempotent.

The likes service emits a +1 or −1 only when the row actually changes, so double-taps and retries never inflate the total. The viewer’s own like state is read from their row, so they see it instantly even though the public total trails by a few seconds.

You now have a complete YouTube: uploads land, get encoded, stream through a CDN, show up in search and get counted. Each piece breaks at real scale, which is what the deep dives fix.

Deep dives

How do we get a 20 GB file in over an unreliable connection?

Take a creator in Patna exporting a 20 GB 4K recording of a wedding and uploading it over a 10 Mbps uplink. That is 20,000 MB × 8 ÷ 10 Mbps = 16,000 s, about 4.4 hours. Home broadband drops, the laptop sleeps, the phone switches towers. YouTube’s limit is 256 GB or 12 hours of video, so plan for worse.

Bad: one big POST. Fine for a 20 MB clip. For a 4.4-hour transfer, any interruption restarts from byte zero: if the connection drops on average once an hour, the chance of 4.4 hours without a drop is e^−4.4 ≈ 1.2%, so about 1 upload in 80 succeeds. Your servers also hold connections open for hours, and every deploy kills the uploads in flight.

Good: a resumable session, one stream. This is YouTube’s public upload protocol. The first request returns a session URI and the client PUTs the file to it. After a break, an empty PUT with Content-Range: bytes */TOTAL returns 308 Resume Incomplete with a Range header saying how many bytes arrived, and the client continues from the next byte. Its rules are worth copying: chunks are multiples of 256 KB, bytes must be contiguous, sessions expire (a 404 means start again), and clients back off exponentially on 500, 502, 503 and 504. You gain resumability almost for free. You pay with a single TCP stream, which cannot fill a long, fat link — from Chennai to a data centre 200 ms away, window size and RTT cap it, not bandwidth — and an upload service that tracks byte offsets.

Great: parallel parts, straight to object storage. The session hands out one pre-signed URL per 16 MiB part. The client sends several parts at once, straight to object storage — no API server touches the bytes — and keeps the checksum each PUT returns. To resume, it asks which parts exist and sends only the missing ones. complete then commits the list atomically after checking sizes and checksums.

The numbers: 16 MiB is 134 Mb, so on a 10 Mbps uplink one part takes about 13 s — the most an interruption can destroy. The 20 GB wedding video is about 1,200 parts. Several parts in flight fill long-distance links that one stream cannot, and per-part checksums catch corruption early. What you pay: more requests (YouTube’s guide discourages chunking unless you need it, for that reason), a client that tracks part state, and orphaned parts from abandoned sessions, which a lifecycle rule deletes after expiry — say 7 days. On a 5 Mbps mobile uplink the radio is the bottleneck, so the client drops to one or two parts in flight. More in large file uploads.

Two refinements. Deduplication: the client sends the file’s SHA-256 up front, so an identical file uploaded twice by mistake skips the transfer. Start before the end: the pipeline can begin encoding parts as they arrive — but many phone apps write the MP4 index at the end of the file, so nothing decodes until the last part lands; recording fragmented MP4 removes that wait.

How do we turn one upload into a dozen renditions, fast and reliably?

A 10-minute 1080p upload needs six H.264 rungs plus audio, a thumbnail set, a policy check and a copyright check, all before it is fully ready. Across 1M hours of new video a day, the encoders are the most expensive compute in the system, and creators judge you on how fast “processing” finishes.

Bad: one job per video, whole file, one machine. A worker downloads the original and encodes each rung in turn. Easy to reason about. But time grows with length — a 12-hour stream ties up a machine for many hours — a crash at 90% throws away all of it, and a few huge jobs can starve the queue while thousands of 30-second clips wait behind them.

Good: one job per rendition. Six workers encode the six rungs in parallel, and 360p can publish before 1080p finishes. Better, but each job still processes the whole file, so a long video is still slow and a failure still restarts a whole rung.

Great: a DAG of small, idempotent tasks. Split the video into chunks and encode every chunk of every rung as a separate task.

How each stage works:

• Probe reads codec, resolution, frame rate, rotation, HDR and duration. Corrupt or unsupported files fail here, in seconds, with a message the creator can act on.
• Split cuts 40-second chunks at keyframes, so each chunk decodes on its own.
• Encode runs every (chunk, rung) pair as one task: 600 s ÷ 40 s = 15 chunks × 6 rungs = 90 tasks. Every rung puts a keyframe every 4 seconds at identical timestamps, so segment boundaries line up across renditions and the player can switch at any segment — the HLS spec requires matching timestamps and the same target duration across variant streams for this reason. Audio is encoded once, separately.
• Package joins chunks per rung, cuts 4 s fragmented-MP4 segments, and writes an HLS playlist and a DASH manifest over the same files.
• Side branches build thumbnails and the scrubbing storyboard and run policy and copyright checks; publishing waits for the checks policy says must pass first.

With 90 tasks on 90 workers, a 10-minute video takes roughly the time of one task. The 1080p rung is about half of the ladder’s 2 core-hours per hour of video, so a 40 s chunk is 40 core-seconds — about 10 s on four threads. With queueing and packaging, the ladder is ready in a minute or two and 360p sooner. YouTube’s help pages describe the same visible behaviour: low quality first, higher qualities later.

Fault tolerance. Make every task idempotent: it writes to a deterministic path such as work/v123/720p/c03.mp4, so running it twice overwrites the same object. A durable workflow engine stores the DAG’s state; workers lease tasks and heartbeat, and a lease that lapses for 60 s means the task reruns elsewhere. After three failures on the same input, mark the chunk poisoned and fail the video with a precise reason (“corrupt frames at 12:40”) instead of looping. For stragglers, once 98% of tasks are done, duplicate the slowest few and keep whichever finishes first. More in long-running jobs.

Priority. Not all work is equally urgent, so give the scheduler lanes:

1. First-playable lane: 360p and audio for every new video. Small, fast, always first.
2. Standard lane: the rest of the H.264 ladder.
3. Background lane: efficient-codec re-encodes for videos that became popular, backfills when the encoder improves. Runs on spare or preemptible capacity.

Inside each lane, schedule fairly per channel, so a news channel bulk-uploading 2,000 archive clips cannot starve a student who uploaded one, and cap concurrency per video: a 10-hour stream is 900 chunks × 6 rungs = 5,400 tasks, and limiting it to 300 at a time stops it swallowing the pool. When everyone uploads highlights after the IPL final, the first-playable lane stays fast and only HD completion slows.

Which renditions does each video earn, and where do they live?

The estimates said 8.1 PB of new originals and renditions a day, about 3 EB a year. The catch is the long tail: most uploads are watched a handful of times, a few take most of the views, and you cannot tell which is which at upload time.

Bad: everything for everyone, on fast disks forever. Encode every upload in H.264 plus VP9 or AV1 and keep every rung hot. Best quality on every device — but the modern-codec ladder alone needs about 830,000 more cores, storage grows by another 3.2 GB per hour of video, and most of it serves no one.

Good: one H.264 ladder for all, kept hot. H.264 plays nearly everywhere, encodes fast and keeps the pipeline simple. What you pay: the popular videos that make up most of the 200 Tbps ship around 30% more bytes than a modern codec would need (our assumed saving), and billions of cold videos sit on expensive disks.

Great: let each video earn its renditions. Treat a video’s renditions as a lifecycle driven by its views.

• Never upscale. The ladder stops at the source resolution: a 720p phone upload gets no 1080p rung.
• Earn the efficient codec. Re-encode in VP9 or AV1 once a video’s views cross a threshold, in the background lane. Work out the threshold. Assume the modern codec saves 30% at equal quality, delivery costs $0.005 per GB, and compute costs $0.02 per core-hour (all illustrative). A 6-minute view at 2.5 Mbps is 2.5 × 360 ÷ 8 = 112.5 MB, so the saving is about 34 MB, or $0.00017 per view. The extra encode for a 10-minute video is about 10 × (2 core-hours × 10 ÷ 60) ≈ 3.3 core-hours, or $0.067. Break-even: 0.067 ÷ 0.00017 ≈ 400 views. Most uploads never get there; the ones that do repay the encode many times over.
• Move originals to cold storage as soon as processing finishes. They are needed only for re-encodes, so erasure-coded, slower, cheaper disks are fine — but they must be durable, because they are the one thing you cannot regenerate.
• Shrink videos nobody watches. After, say, six months without a view, delete everything except 360p and the original. A 10-minute video’s H.264 ladder is 4.5 GB ÷ 6 = 750 MB; its 360p rung is 0.7 Mbps × 600 s ÷ 8 = 52.5 MB, so each archived video frees about 700 MB of hot storage. If someone does come back — a 2014 Diwali vlog shared again in a family group — they start at 360p immediately while the DAG rebuilds the higher rungs in a minute or two.

Hardware moves the break-even. YouTube built its own transcoding chip, the Video (trans)Coding Unit, and reported 20–33× better compute efficiency than its previous optimised software system. If the extra encode gets 20× cheaper, the threshold falls from about 400 views to about 20, and a far larger share of the catalogue gets the efficient codec. See cost-aware design.

How does the player keep playing when the network changes?

The player, not the server, decides quality. It fetches a manifest, then one segment at a time, and before each request it picks a rung. An HLS master playlist for our ladder looks like this (three of the six rungs shown):

#EXTM3U
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-STREAM-INF:BANDWIDTH=1000000,AVERAGE-BANDWIDTH=830000,RESOLUTION=640x360,CODECS="avc1.4d401e,mp4a.40.2"
360p/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=3400000,AVERAGE-BANDWIDTH=2630000,RESOLUTION=1280x720,CODECS="avc1.4d401f,mp4a.40.2"
720p/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=6800000,AVERAGE-BANDWIDTH=5130000,RESOLUTION=1920x1080,CODECS="avc1.640028,mp4a.40.2"
1080p/index.m3u8

BANDWIDTH is the peak segment bitrate and AVERAGE-BANDWIDTH the average, both including audio; RFC 8216 warns that wrong values cause stalls. Each index.m3u8 then lists that rung’s 4-second segments. DASH expresses the same thing in XML, often with a segment URL template instead of a list, and can even address byte ranges inside one large file.

How long should a segment be? Every segment starts with a keyframe, and the player can only switch at a boundary. Across 41.7M concurrent streams:

Pick 4 s for on-demand video: it halves the request rate of 2 s segments and still lets the player react within a few seconds.

Bad: one file at one quality. Progressive download of a single 720p MP4 is simple, and it stalls for everyone below 2.5 Mbps while wasting bandwidth for everyone on fibre.

Good: pick by measured throughput. Estimate throughput from recent segment downloads and choose the highest rung under a safety margin — the throughput rule in dash.js, the DASH Industry Forum’s reference player. It handles sustained changes, but mobile throughput is noisy: one slow segment drags the estimate down and quality flaps.

Great: combine throughput, buffer and guards. The buffer — seconds of video downloaded but not yet played — is a steadier signal than throughput. dash.js by default combines its throughput rule with BOLA, a buffer-based rule that maps buffer level to a bitrate, and switches between them based on the current buffer: throughput while the buffer is small at startup, buffer-based once it is healthy. Around them sit guards:

• Insufficient buffer. Never pick a bitrate you cannot download before the buffer runs dry: bitrate ≤ safe throughput × buffer ÷ segment length. With 5 Mbps measured, a 0.9 safety factor, 10 s of buffer and 4 s segments: 4.5 × 10 ÷ 4 = 11.25 Mbps is the ceiling.
• Abandon. If a segment is downloading too slowly to finish in time, cancel it and fetch the same segment at a lower rung, rather than stalling.
• Context caps. No 1080p into a 360-pixel-wide player, and step down if the device is dropping frames.
• Hysteresis. Require a few good segments before stepping up, so quality does not flap.

Startup. Start at a modest rung and climb. The manifest, the initialisation segment and a first 480p segment (1.3 Mbps × 4 s = 5.2 Mb, about 0.5 s on a 10 Mbps connection) from a cache 20 ms away take about 0.6 s, inside the 1 s p50 target; returning the manifest inline with the watch page shaves more. Audio is a separate small track, so it never stalls when video steps down.

How do we serve both the viral head and the enormous long tail?

At 200 Tbps the delivery network is the product. The catalogue is exabytes; what is being watched this evening is a tiny, shifting fraction of it. Caches love the second and are useless for the first.

Bad: serve from the origin data centres. No handful of sites can push 200 Tbps, transit bills would be enormous, and a viewer in Guwahati fetching from a data centre a continent away pays 150 ms or more per round trip — the 1 s start target is gone before the first byte.

Good: a pull-through CDN with plain LRU caches. Edges near viewers cache whatever is requested, so the head is served locally. Two problems remain. Long-tail requests — videos watched once in that city this month — flow through the edge and evict popular segments for content nobody will ask for again. And a viral video’s first minutes send a stampede of simultaneous misses to the origin.

Great: tiered, popularity-aware caching.

• Edge caches inside ISPs. Serve the head without crossing a peering link. Google runs exactly this for YouTube as Google Global Cache: the ISP provides rack space, power and connectivity, Google operates the servers, and Google says typically 70–90% of cacheable traffic is served from them. Assume 85%.
• Regional caches as shields. Every edge in a metro sends its misses to one regional tier, which absorbs another 80% of them. The origin sees 200 Tbps × 15% × 20% = 6 Tbps, almost all of it long tail, spread over several storage regions.
• Admission, not just eviction. Cache a segment at the edge only on its second request within a window, so a one-view video is served from the regional tier and never evicts a hit. Content that does become popular pays one extra regional fetch.
• Steering and placement. The watch API returns URLs naming a specific cache (edge-bom-07 above), chosen by a mapping service from the viewer’s network, cache health and load. Within a site, consistent hashing maps each video to one or two machines, so 40 machines hold 40 slices of the head rather than 40 copies of one slice.
• Signed, expiring URLs make takedowns and privacy changes take effect when they expire, and stop hotlinkers using your bandwidth.

When a video goes viral in an hour. A last-ball six in an IPL final, clipped and uploaded by a fan, reaches 5M views in its first hour: 5M ÷ 3,600 ≈ 1,400 new viewers a second, all asking for the same first segments within moments of each other. Two techniques keep that off the origin.

Request coalescing (collapsed forwarding): a cache with a fetch already in flight for a segment makes later requesters wait for it instead of starting their own. Do it at the edge and at the regional tier, and the origin sees at most one request per regional cache per segment, however many people press play. Velocity-triggered warming does the rest: a stream job watches views per minute, and when a video’s rate jumps — say 50× in ten minutes — it pushes the first segments of the popular rungs to edges, bumps its efficient-codec re-encode, and shortens its watch-page cache TTL. The general pattern is in hot keys, cache stampede and invalidation.

How do we count views at scale — and why is the number allowed to lag?

The view counter looks trivial and is where designs quietly fail. Our long-form estimate is about 350k views a second at peak, and YouTube’s press page says Shorts alone average over 200 billion views a day — 200B ÷ 86,400 ≈ 2.3M view events a second. The counter also has adversaries: loop scripts, click farms, refresh bots.

Bad: UPDATE videos SET views = views + 1 on every play. One line of code, until the IPL clip arrives: 1,400 updates a second contend for one row lock on the shard that also serves its watch page. Retries double-count, and every bot view counts.

Good: buffered, sharded counters. Collectors sum pings per video for 10 seconds and write one +N delta to a sharded counter or a Redis INCRBY: one write per collector every 10 seconds instead of 1,400 a second. But nothing is de-duplicated, fraud is counted, and a collector crash loses its batch — or, after a retry, applies it twice.

Great: one event log, two paths.

• Events, not increments. Each ping carries an event_id, video_id, a viewer or device ID, a session ID and a timestamp, and goes to a durable Kafka-style log. Delivery is at least once; the event_id makes duplicates harmless downstream (see delivery guarantees).
• Fast path. A stream job sums events per video per 10-second window and writes the total to the counts store. The public number moves within a minute. Partitioning the log by video_id would put a viral video on one partition, so collectors pre-aggregate per video first, and the hottest videos are keyed by video_id plus a small random salt and summed back together.
• Verified path. A slower job, hours behind, de-duplicates by viewer, video and session, drops traffic its abuse models flag, and discards low-quality playbacks. It then overwrites the video’s count with the verified total. This is the number monetisation and creator analytics use.
• Unique viewers use a HyperLogLog sketch per video: with 2¹⁴ = 16,384 six-bit registers it takes 12 KB and has a standard error of 1.04 ÷ √16,384 ≈ 0.8%, and sketches from different days merge cleanly. See count-min sketch and HyperLogLog.

YouTube’s help pages describe the same split from the outside. Since 24 August 2026 a public view counts as soon as playback starts, while earnings use separately defined “engaged views”; counts may appear slowly at first, may be temporarily slowed, frozen or changed while being verified, and low-quality playbacks — the same video streamed in several tabs or on several devices — are discarded.

The final design

Put the pieces together. Three planes share one video ID: ingest and processing at the top, delivery down the left, metadata and counting along the bottom.

Follow a video from upload to its millionth view:

1. Upload. The upload service creates the video row and a session; 16 MiB parts go straight to object storage in parallel, and complete verifies checksums and commits the original, which later moves to erasure-coded cold storage.
2. Process. The object-created event starts the DAG: probe, split at keyframes, encode chunk × rung in priority lanes, package 4 s segments with HLS and DASH manifests, thumbnails, policy and copyright checks. 360p publishes first; the row moves to ready and change data capture updates search.
3. Watch page. The watch API reads metadata and counts from the cache (over 99% hits; sharded MySQL otherwise) and returns signed manifest URLs naming the best edge cache for the viewer’s network.
4. Stream. The player fetches segments from the edge inside its ISP, choosing each rung from throughput and buffer. Misses go to the regional cache, which coalesces them; only the long tail reaches the origin.
5. Count. Pings flow through collectors into the view-events log. The fast counter updates the public number every few seconds; the verified job replaces it hours later. Likes are per-user rows whose changes feed the same counters.

Check it against the requirements. Start-up is about 0.6 s at p50 because the first segment comes from inside the viewer’s ISP. Playback survives an origin region outage because caches keep serving the head. Uploads are durable once complete returns and resumable before it. 360p is ready in about a minute, HD in a few. Only about 6 Tbps of the 200 Tbps peak reaches the origin, and the 3 EB a year is tamed by cold originals, shrunk cold videos and popularity-gated codecs. View counts are eventually consistent by design; likes are idempotent per user.

What interviewers look for by level

Follow-up questions

What if creators want to stream live?

Live inverts the pipeline: there is no file to split. The creator pushes a continuous stream to a nearby ingest point; a real-time transcoder produces every rung as video arrives and appends 2–4 s segments to a live playlist that players reload. The HLS spec tells players not to start less than three target durations from the end of a live playlist — with 4 s segments, at least 12 s behind live — so low-latency live needs shorter or partial segments. Delivery reuses the same caches; a live segment is just a very hot, short-lived object, so coalescing matters even more. When the stream ends, the recording goes through the normal DAG.

What if a video must disappear everywhere within minutes?

Set the video’s status to blocked and invalidate its watch-page cache entries, so no new viewer gets a manifest URL. Manifest URLs are signed with short expiries, so existing links die within that window; for urgent cases, also purge the v123/ prefix across the edge fleet. Cached segments are useless without a valid manifest and are deleted from origin afterwards, with the original kept under legal hold if required.

What if a whole storage region fails?

The head is unaffected: ISP and regional caches do not depend on any one origin. For the tail, keep originals erasure-coded across regions and hot renditions in at least two, so the mapping service points regional caches at the survivor. Cold videos whose renditions lived only in the lost region are rebuilt from the original elsewhere — slower for that first viewer, but nothing is lost. Rehearse it by draining a region on purpose. See disaster recovery.

What if most viewers are on slow, metered mobile data?

This is daily life for a large share of viewers in India and elsewhere. Tune the bottom rungs (144p, 240p) carefully, offer a data-saver mode that caps the rung whatever the throughput, make audio-only playback first-class, and support offline downloads over Wi-Fi, encrypted and tied to the app. Every saved megabyte is the viewer’s money, so lower the efficient-codec threshold for videos popular in those regions.

What if the catalogue were small and licensed instead of user-uploaded?

Then almost every choice flips. With thousands of titles known weeks in advance instead of 20 million surprises a day, you can spend hours of compute tuning each title’s encoding and copy popular titles into caches inside ISPs overnight instead of pulling on demand. The hard problems become pre-positioning, steering each play to the best cache, and quality for hundreds of millions of members. That is design Netflix.

How to say it in the interview

“I’ll design a video platform for user uploads: more than 20 million uploads a day, around a billion hours watched, about 200 terabits a second at peak, and a catalogue with an extreme long tail.

I split it into three paths. Upload: the client opens a resumable session and sends 16 MiB parts in parallel straight to object storage, each with a checksum, so a dropped connection costs one part. The original is kept forever on durable, cold storage, because every rendition can be rebuilt from it.

Processing: an upload-complete event starts a DAG in a workflow engine. Probe, split at keyframes into 40-second chunks, encode every chunk of every rung as an idempotent task, then package 4-second segments with HLS and DASH manifests. Keyframes are aligned across rungs so the player can switch anywhere. Priority lanes put 360p out in about a minute, and per-channel fairness stops bulk uploaders starving everyone else.

Playback: the watch API returns metadata from a cache over sharded MySQL, plus signed manifest URLs. The player fetches segments from caches inside the viewer’s ISP, backed by regional shields that coalesce misses, so only about 3% of bytes reach the origin — mostly long tail. The player picks each rung from throughput at startup and from buffer once it is healthy, with guards that abandon slow downloads.

Cost: renditions are earned. Everything gets an H.264 ladder; videos past a few hundred views get an efficient codec, which pays for itself; and videos nobody watches shrink to 360p plus the original.

Counting: views are events in a log. A fast path shows an approximate count within a minute; a verified path de-duplicates and filters fraud hours later and replaces it. That is why counters lag, and why only verified numbers pay creators.”

Concepts used in this design

• Large file uploads — resumable sessions, pre-signed parts and checksums.
• Blob storage (S3) — where originals and segments live, and how storage tiers work.
• Long-running jobs — leases, heartbeats and retries for encode tasks.
• Workflow engines (Temporal) — durable state for the processing DAG.
• Message queues — decoupling bursty uploads from the encoder pool.
• Content delivery network (CDN) — edge and regional tiers for 200 Tbps.
• Hot keys, cache stampede and invalidation — request coalescing for viral videos.
• Consistent hashing — spreading videos across the machines in a cache site.
• Sharding a relational database — the Vitess path for video metadata.
• Change data capture — keeping the search index in step with the database.
• Elasticsearch — the inverted index behind video search.
• Kafka — the durable log of view events.
• Delivery guarantees — at-least-once pings made safe with event IDs.
• Count-min sketch and HyperLogLog — unique viewers in 12 KB per video.
• Batch vs stream processing — the fast and verified counting paths.
• Cost-aware design — popularity-gated codecs and cold storage.

References and further reading

1. YouTube Official Blog: press page, “YouTube by the numbers”blog.youtubeOver 20 million videos uploaded on an average day; Shorts averaging over 200 billion daily views.
2. YouTube Data API: Resumable uploadsdevelopers.google.comSession URIs, 308 Resume Incomplete with a Range header, 256 KB chunk multiples, expiring sessions and backoff on 5xx.
3. YouTube Help: Upload videos longer than 15 minutessupport.google.comThe upload limit: 256 GB or 12 hours, whichever is less.
4. YouTube Help: Low video quality after uploadsupport.google.comVideos become playable in low quality first; a 60-minute 4K30 video can take up to 4 hours for high-resolution processing.
5. YouTube Help: How engagement metrics are countedsupport.google.comViews count when playback starts (from 24 August 2026); counts are verified over time and may be slowed, frozen or adjusted; low-quality playbacks are discarded.
6. Reimagining video infrastructure to empower YouTube (YouTube Blog, 2021)blog.youtubeWhy transcoding is costly, and the custom Video (trans)Coding Unit with 20–33× better compute efficiency.
7. Vitess documentation: Historyvitess.ioHow YouTube’s MySQL went from one primary to replicas to application-level sharding, and why Vitess moved routing into a proxy.
8. RFC 8216: HTTP Live Streamingrfc-editor.orgMaster and media playlists, BANDWIDTH and AVERAGE-BANDWIDTH, a typical 10 s target duration, and aligned timestamps across variant streams.
9. dash.js (DASH-IF reference player): ABR settings and rulesdashif.orgDefault mix of a throughput rule and the buffer-based BOLA rule, plus insufficient-buffer and abandon-request rules.
10. Interconnect Help: Introduction to Google Global Cache (GGC)support.google.comCaches inside ISP networks; typically 70–90% of cacheable traffic is served from them.