Design a URL shortener (and Pastebin)

A URL shortener turns https://shop.example/sale/diwali-2026/electronics?utm_source=whatsapp&utm_campaign=… into sho.rt/3cVrDte and sends anyone who opens the short link to the long one. Pastebin is the same idea for text: paste a stack trace or a config file, get a short link to share. Both look like a weekend project — a table with two columns — which is why interviewers like them: the interesting parts hide in the details. How do you mint billions of seven-character codes that never collide and cannot be guessed in sequence? How do you answer 140,000 redirects a second in a few milliseconds when one link shown during an IPL final suddenly gets a flood of scans? And how do you count clicks, expire content and stop phishing without slowing a redirect down?

Understand the problem

Two products share one skeleton here. A short link maps a code to a URL; a paste maps a code to a block of text. Agree on that up front, then pin down the numbers, because almost every design decision falls out of one fact: people open links far more often than they create them.

Functional requirements

1. Users should be able to shorten a long URL, optionally choosing a custom alias (sho.rt/diwali-sale) and an expiry time.
2. Users should be able to open a short link and land on the original URL.
3. Users should be able to create a text paste of up to 1 MB, with optional syntax highlighting, a visibility setting (public, unlisted or private) and an expiry, and share it through a short link.
4. Users should be able to see how many times their link or paste was opened, by day and by country.

Non-functional requirements

• Fast redirects. Inside our servers, p50 under 5 ms and p99 under 20 ms when the code is cached; p99 under 50 ms on a cache miss. Users add their own network round trip on top, so the answer should come from somewhere close to them.
• Very available reads. 99.99% for redirects, about 53 minutes of downtime a year. Short links are printed on posters and sent in SMS campaigns; when the service is down, every copy breaks at once. Creating links can live with 99.9%.
• Scale. 30 million new links a day and 3 billion redirects a day — a 100:1 read-to-write ratio — with peaks about 4× the average. Pastes are smaller in count: 2 million a day.
• Uniqueness and durability. A code maps to exactly one destination, forever. Once we return a short link we never lose it, and we never hand an expired code to someone else.
• Consistency where it is visible. The creator gets read-your-writes: the link works the instant it is returned. Edits and deletes should show everywhere within seconds, with a minute or two as the worst case; analytics may lag by a minute or two.
• Hard to enumerate. People shorten “anyone with the link can view” documents. If codes are sequential, a script can walk through all of them.

Out of scope

• Accounts and login. We assume a signed-in user or an API key arrives already authenticated; anonymous creation is allowed but rate-limited.
• Rich marketing analytics such as funnels and attribution. We count opens and break them down by day, country and referrer; anything deeper reads our raw event archive.
• QR codes and link-in-bio pages. A QR code is just the short URL drawn as a picture; it adds no backend design.
• Malware detection itself. We consume a reputable list of unsafe URLs; we do not build a crawler that classifies websites.

The set-up

Plan the approach

Say the shape out loud early: “a write path that mints codes and stores mappings, a read path that answers redirects from cache, and an analytics path off to the side.” The read path is 100 times busier and needs a higher availability target, so it gets its own service. Satisfy each requirement simply, then spend most of your time in the deep dives.

Core entities

• Link. The code (primary key), the long URL, owner, creation time, optional expiry, a kind (url or paste) and a status (active, blocked, deleted).
• Paste. A link whose kind is paste. It adds the object key of the body, its size, language, visibility and a burn-after-read flag.
• Code block. A range of 1,000 IDs reserved by one server, so it can mint codes without asking anyone.
• Click event. Code, timestamp, country, referrer and a coarse device class — one per open.
• Stats rollup. Counts per code per minute (later per day), by country and referrer.

A link stored in a key-value table, partitioned by code, is one small item:

{
  "code": "3cVrDte",
  "kind": "url",
  "long_url": "https://shop.example/sale/diwali-2026?utm_source=whatsapp",
  "owner_id": "u_51f0",
  "created_at": 1793094600,
  "expires_at": 1794767399,
  "status": "active"
}

expires_at is epoch seconds, which is what managed TTL features expect; a link that never expires simply has no such attribute.

API

Creating a short link:

POST /v1/links
Authorization: Bearer <token>
Idempotency-Key: 7d0c5a1e-1f0b-4c7a-9e57-0a2f1c3b9d44
Content-Type: application/json

{
  "long_url": "https://shop.example/sale/diwali-2026?utm_source=whatsapp",
  "alias": "diwali-sale",
  "expires_at": "2026-11-15T23:59:59+05:30"
}

HTTP/1.1 201 Created
Location: https://sho.rt/diwali-sale
Content-Type: application/json

{
  "code": "diwali-sale",
  "short_url": "https://sho.rt/diwali-sale",
  "expires_at": "2026-11-15T18:29:59Z"
}

Leave out alias and the server generates a seven-character code. If the alias is taken, the answer is 409 Conflict.

Opening a link is a plain GET; the deep dives explain the status code and the caching header:

GET /3cVrDte HTTP/1.1
Host: sho.rt

HTTP/1.1 302 Found
Location: https://shop.example/sale/diwali-2026?utm_source=whatsapp
Cache-Control: max-age=0, s-maxage=60

Creating a paste returns a page URL and a raw URL. Unlisted pastes get a longer, random code (the deep dives explain why):

POST /v1/pastes
Content-Type: application/json

{
  "content": "Traceback (most recent call last): …",
  "language": "python",
  "visibility": "unlisted",
  "expires_in": "1d"
}

{
  "code": "kq81ZcX0aT",
  "url": "https://sho.rt/p/kq81ZcX0aT",
  "raw_url": "https://raw.sho-content.example/kq81ZcX0aT",
  "expires_at": "2026-10-28T07:20:00Z"
}

The rest is ordinary management:

GET    /p/{code}                         rendered paste page (HTML)
GET    /{code} on raw.sho-content.example  raw paste text, on a separate domain
DELETE /v1/links/{code}                  owner deletes a link or paste; caches are purged
GET    /v1/links/{code}/stats?from=2026-10-01&to=2026-10-27   counts by day and country

Back-of-envelope estimates

Write the arithmetic down; three of these numbers shape the design. (For the method, see back-of-envelope estimation.)

Creates. 30M links a day: 30,000,000 ÷ 86,400 ≈ 350 creates/s on average, and 4× at peak ≈ 1.4k creates/s. Tiny.

Redirects. 100 opens per link created: 30M × 100 = 3B a day. 3,000,000,000 ÷ 86,400 ≈ 34.7k redirects/s on average; 4× at peak ≈ 140k redirects/s.

Code length. Keep links for ten years: 30M × 365 × 10 ≈ 110 billion codes. With the 62 characters 0–9, a–z, A–Z, six characters give 62⁶ ≈ 56.8 billion — not enough. Seven give 62⁷ ≈ 3.52 trillion, so after ten years the space is only 110B ÷ 3.52T ≈ 3% full. Seven characters it is.

Link storage. About 300 bytes per link once you count the URL (average around 120 bytes), owner, timestamps, item overhead and an index on owner. 30M × 300 B = 9 GB a day, × 365 ≈ 3.3 TB a year, ≈ 33 TB over ten years — about 100 TB with three replicas. Too much for one database server, unremarkable for a partitioned key-value store.

Cache. Assume 200M distinct links get at least one click on a given day, and the busiest 20% of those take about 80% of the clicks. Caching the busy fifth: 40M × 300 B ≈ 12 GB. One machine could hold that; you shard Redis for throughput and availability, not memory.

Redirect bandwidth. A 302 with headers is about 400 bytes: 140k × 400 B ≈ 56 MB/s ≈ 450 Mbit/s at peak. Not a concern.

Pastes. 2M a day ÷ 86,400 ≈ 23 creates/s, about 90/s at peak. At an average of 10 KB: 2M × 10 KB = 20 GB of text a day, ≈ 7.3 TB a year. Views run about 10 per paste: 20M a day ≈ 230/s, about 930/s at peak; a rendered page of about 30 KB makes that 930 × 30 KB ≈ 28 MB/s at peak.

Click events. One event of about 200 bytes per redirect: 3B × 200 B = 600 GB a day of raw events.

Three numbers shape what follows. The 100:1 ratio and 140k/s peak mean the read path is the whole game: it must be served from memory, close to users. The 110 billion codes fix the length at seven characters. And 20 GB of paste text a day — more than double the 9 GB of link rows — is data nobody queries by field, so it does not belong in the database.

High-level design

Build one requirement at a time. By the end of this section the design works in one region at moderate traffic; the deep dives then make it fast, safe and big.

Users should be able to shorten a long URL

The Write API receives the long URL and checks it: the scheme must be http or https (never javascript: or data:), it must be under 2,048 characters, and it must not point back at sho.rt itself, or two short links could redirect to each other forever.

For storage, the access pattern is a single-key lookup by code, with 33 TB arriving over ten years and no joins. That is the job description of a partitioned key-value store such as DynamoDB or Cassandra (see Cassandra and DynamoDB): partition by code, add a secondary index on owner_id for “my links”. A sharded PostgreSQL would also work; you would be using it as a key-value store anyway.

The Write API validates the URL, takes the next code (imagine a counter for now; the first deep dive replaces it) and writes code → URL with a conditional put — “insert only if no item with this code exists” — before returning 201 Created. If the condition ever fails, it takes another code and retries.

Users should be able to open a short link and land on the original URL

Redirects get their own Redirect service: small, stateless and run as many copies, because it carries 100 times the traffic of the Write API and needs the higher availability target. It looks the code up in a Redis cache first and falls back to the Links DB on a miss — the classic cache-aside pattern. A known code gets a 302 with the long URL in Location; an unknown code gets 404; an expired or deleted one gets 410 Gone.

Two details make this correct from day one. Write-through on create: the Write API also puts the new mapping in Redis, so the creator’s first test click is a cache hit and never races a lagging database replica. Status and expiry live in the row: the redirect checks status and expires_at before answering, so a deleted link stops working even before any cleanup job runs.

Users should be able to create a paste and share it with a short link

A paste is a link whose target is text instead of a URL, so it reuses the code generator, the Links DB and the analytics. What changes is the size of the thing behind the code: up to 1 MB instead of about 120 bytes. Real paste sites cap it for the same reason; Pastebin.com’s FAQ lists 512 KB for free users and 10 MB for paying ones.

Put the body in object storage, the metadata in the database. The Write API stores the text as an object (pastes/kq81ZcX0aT) in blob storage and writes a normal Links DB row with kind = paste, the object key, size, language and expiry. Putting 20 GB of text a day into the key-value store would more than double its growth with data nobody filters on, slow its compactions and backups, and cost more per gigabyte. The row stays about 300 bytes; the bytes go where bytes are cheap.

The Write API also copies the few fields a reader needs — expiry, language, visibility, owner — into the object’s own metadata, so rendering a paste takes one GET from object storage. The Links DB remains the source of truth for ownership, listing and takedowns; a delete removes the object as well as marking the row.

Writes. A 1 MB body is small enough to send through the Write API, which lets it enforce the size cap, reject binary junk and scan the text before anything is stored. (For much larger uploads you would hand out a pre-signed upload URL instead; see the follow-ups.) Pastes are immutable: “editing” creates a new paste with a new code. That one rule makes caching trivial.

Reads. GET /p/{code} goes to the CDN. On a miss, the Paste viewer fetches the object, checks expiry and visibility from its metadata, highlights the syntax and returns HTML. Highlighting a large file can take tens of milliseconds of CPU, but since the paste never changes, the edge keeps the page until the paste expires: a viral paste is rendered once per edge location, not once per view. Unlisted pastes are cacheable too — anyone with the link may see them — but private ones are served with Cache-Control: private, no-store and checked against the session on every request.

Raw text lives on a separate domain. The raw view is served as text/plain with X-Content-Type-Options: nosniff from raw.sho-content.example, a domain with no cookies. If someone pastes HTML and JavaScript and tricks a browser into rendering it, the script runs on a throwaway domain, not on the one holding your users’ sessions.

Users should be able to see how many times their link was opened

Adding one to a database counter on every redirect would turn the busiest read in the system into a write. Instead, the Redirect service sends a small click event (code, time, country, referrer, device class) to a Kafka topic without waiting, and answers the visitor straight away. An aggregator writes per-code, per-minute counts to a Stats DB; raw events are archived for 30 days.

You now have a working system: links and pastes are created, redirects are served from cache, and clicks are counted. Everything that makes it interesting at 140,000 redirects a second is in the deep dives.

Deep dives

How do we generate short codes that are unique, short and hard to guess?

The requirements pull in three directions: seven characters, never a duplicate, and no visible pattern. There are four common approaches.

Bad: hash the long URL and keep the first seven characters. Take an MD5 or SHA-256 of the URL, encode it in base62 and keep seven characters (about 42 bits). No coordination, and the same URL always gives the same code. The trouble is collisions, and they arrive early: by the birthday bound, once you have about 2.2 million codes there is a 50% chance two different URLs share one — under two hours after launch at 350 creates a second. So every create must check the database and, on a clash, rehash with a salt; by year ten about 1 in 32 new hashes lands on a taken code. You pay a read-before-write on every create, and “same URL, same code” no longer holds once you salt. It is rarely wanted anyway: two shopkeepers shortening the same product page each expect their own link, expiry and click counts.

Good: random codes with a conditional put. Draw seven random base62 characters from a cryptographically secure generator and insert with “only if this code does not exist”. No coordination, nothing to guess, and the retry rate equals how full the space is — about 3% in year ten. A perfectly good answer. What you pay: every create is a conditional write, retries creep up as the space fills, and a weak generator such as Math.random() makes codes predictable.

Great: a counter, handed out in blocks, then shuffled. Counters guarantee uniqueness without checking anything, but a single counter is a single point of failure and an extra hop on every create. Flickr solved the first part with ticket servers: two MySQL servers, one handing out odd IDs and the other even (auto-increment-increment = 2), each bumped with a one-row REPLACE INTO — the “dumbest possible thing that will work”, in production since 2006.

You can remove the per-create hop too. Each Write API pod reserves a block of 1,000 IDs with one atomic increment of a block counter, then hands them out from memory. With 20 pods sharing the 1.4k/s peak, each pod does about 70 creates a second and fetches a new block roughly every 14 seconds; the whole fleet makes about 1.4 block requests a second. If a pod crashes, the rest of its block is simply never used — a gap, never a duplicate. Even 1,000 restarts a day waste 1M IDs a day, about 0.1% of the code space over ten years.

Where the block counter lives matters more than how fast it is. It must never go backwards. Redis INCR is atomic and handles 64-bit values, but Redis replicates asynchronously: if the primary fails after handing out block 4,811 and before a replica heard about it, the new primary hands out 4,811 again, and two pods mint the same codes. Keep the counter in a strongly consistent store — an etcd key updated with compare-and-swap, or one row in a relational database updated in a transaction. At 1.4 calls a second, either is idle. And keep the conditional put anyway: if something upstream ever does go wrong, the database refuses the duplicate instead of overwriting a live link.

What about Snowflake-style IDs? Twitter’s Snowflake packs a 41-bit millisecond timestamp, a 10-bit machine ID and a 12-bit sequence into one 64-bit integer, with no coordination between generators, and refuses to issue IDs if the clock moves backwards. Perfect for time-ordered tweet IDs; wrong for short links. A 63-bit number needs 11 base62 characters, because 62¹⁰ ≈ 8.4 × 10¹⁷ is smaller than 2⁶³ ≈ 9.2 × 10¹⁸ — and most of those bits encode a timestamp the shortener does not need. (More in design a unique ID generator.)

Now make the counter unguessable. Encode counter values directly and consecutive links get consecutive codes: 000kb9e, 000kb9f, 000kb9g. Anyone can scrape every link you have ever made by counting upward — including all the “anyone with the link” document shares and unlisted pastes. The fix is to pass each ID through a keyed permutation of the whole code space before encoding. A permutation is a one-to-one mapping, so distinct IDs still give distinct codes; with a secret key, neighbouring IDs give unrelated codes.

A small Feistel network does the job. Split a 42-bit number into two 21-bit halves and run four rounds, each mixing one half with a keyed hash (an HMAC) of the other; the result is a bijection on 42-bit numbers. Since 2⁴² ≈ 4.4 trillion is a little larger than 62⁷ ≈ 3.52 trillion, use cycle walking: if the output is 62⁷ or more, encrypt it again — on average 2⁴² ÷ 62⁷ ≈ 1.25 passes, microseconds in all. Do not use “multiply by a big constant modulo 62⁷” instead: it looks random, but two consecutive codes reveal the constant. And treat the key as permanent: a new key is a different permutation, which could map a fresh ID onto a code that is already live.

A shuffle stops enumeration, not guessing. At 3% full, a random seven-character guess still hits some live link one time in 32. That is fine for public links, but not for unlisted pastes or shared documents. Give those ten random characters instead: ten years of pastes (2M × 3,650 ≈ 7.3 billion) fill the 62¹⁰ ≈ 8.4 × 10¹⁷ possible codes so thinly that a guess succeeds about once in 115 million tries — and rate-limit clients that collect 404s.

Seven characters also leave you a choice of alphabet. If people will type codes from a printed poster, drop look-alikes such as 0, O, I and l: 58⁷ ≈ 2.2 trillion is still twenty times the 110 billion you need.

Should a redirect be a 301 or a 302?

Interviewers use this to check whether you know HTTP. RFC 9110 defines four redirects that carry a Location header, differing on two axes:

• Permanent or temporary. 301 Moved Permanently and 308 Permanent Redirect say the resource has a new permanent URI and future requests ought to go there. 302 Found and 307 Temporary Redirect say the target lives elsewhere for now, so the client ought to keep using the short URL.
• Method changes. For historical reasons a client may turn a POST into a GET when following a 301 or 302; 307 and 308 forbid that.

Short links are opened with GET, so the method rule does not matter. Permanence does, because of caching: the RFC marks 301 and 308 heuristically cacheable — a browser or proxy may store and reuse them even if you send no caching headers. 302 and 307 are cached only if you explicitly say so.

Bad for most shorteners: a bare 301. You gain load reduction — repeat clicks from the same browser never reach you. You pay three ways: repeat clicks vanish from your analytics, an owner who edits the destination cannot reach browsers that remember the old one, and a link that starts pointing at a phishing page cannot be switched off for anyone who has it cached. Nothing in a bare 301 says when the browser should stop trusting it.

Good: a 302 (or 307) with no caching. Every click comes back, so counts are exact and edits and takedowns work immediately. You pay with every click hitting your servers — at 140k/s, that is the load we sized for.

Great: a 302 with explicit caching headers. Send Cache-Control: max-age=0, s-maxage=60. RFC 9111 says s-maxage overrides max-age for shared caches only, so the CDN may reuse the redirect for 60 seconds while browsers come back every time. You keep most of the load reduction (the next deep dive puts a number on it), the edge’s copy is never more than a minute old and can be purged at once, and since every click still passes the CDN, its request log becomes the source for analytics. If an owner really wants a permanent redirect, such as for a domain migration, send a 301 or 308 with an explicit Cache-Control: max-age, so browsers do not invent their own lifetime.

How do we serve 140,000 redirects a second in a few milliseconds?

Bad: read the database on every redirect. A partitioned key-value store can serve 140k reads a second if you pay for it, but every redirect then inherits the database’s tail latency, and hot links break it anyway. Picture the last over of an IPL final: a sponsor’s QR code is on screen, 50 million people are watching and 2% scan it within a minute. That is 1 million requests in 60 seconds — about 17k redirects a second on one code, all on one database partition with a fixed throughput ceiling.

Good: a Redis cache in front. With cache-aside and write-through on create, the 12 GB hot set fits easily and the hit ratio is about 90%. A small Redis cluster — four primaries at a comfortable 50k reads a second each, plus replicas — carries the whole peak at about 1 ms. Give every entry a TTL (an hour, or less if the link expires sooner), and cache misses too: bots probe random codes all day, and a short-lived “not found” entry keeps them off the database. But one hot code still lands on one Redis shard, and every redirect still crosses the network.

Great: three layers, each absorbing what the next would struggle with.

1. The CDN edge keeps each 302 for 60 seconds (s-maxage=60). A hot link costs each edge location one origin request a minute, however many people scan it; the 17k/s sponsor link becomes a trickle at the origin.
2. An in-process cache in every redirect pod — a plain LRU of about 100,000 entries, 30 MB of memory — answers the merely-warm links in microseconds and spreads any remaining hot-key load across all pods instead of one Redis shard. Give its entries a short TTL, around 30 seconds, so edits propagate.
3. Redis, then the database, for everything else.

What you pay for the layers is staleness, so bound it explicitly. Left alone, an edit could take 90 seconds to show everywhere: 30 seconds in a pod’s LRU, then 60 at the edge on top. So the Write API never waits: on every edit, delete or takedown it purges the path at the CDN and publishes an invalidation that every pod applies to its LRU and to Redis; the TTLs only matter if a purge is lost. Counting moves too: once the edge answers some redirects, the origin no longer sees every click, so analytics read the CDN’s request logs, which record hits and misses alike.

How do we hand out custom aliases without races?

Two shopkeepers in the same market both want sho.rt/diwali-sale for their festival offers, and both press “create” at the same moment.

Bad: check, then write. Both requests read the alias before either writes, both see it free, and both write. In DynamoDB a plain PutItem replaces an existing item with the same key, so the second write wins silently: the first shopkeeper’s printed flyers now send customers to a competitor. A lock would fix it, at the cost of a lock service on the write path.

Great: let the database decide, once. Make the write itself conditional. In DynamoDB that is a PutItem with the condition attribute_not_exists(code) — the documentation describes exactly this use — and the losing write fails with ConditionalCheckFailedException. In SQL it is a primary key plus INSERT … ON CONFLICT DO NOTHING. The loser gets 409 Conflict, ideally with suggestions such as diwali-sale-2026. This is optimistic concurrency: no locks, and check and write are one atomic step.

Four more rules keep aliases from causing trouble:

• Keep the namespaces apart. Generated codes are exactly seven case-sensitive base62 characters. Make aliases case-insensitive (store them lowercase — people type them from posters), limit them to a–z, 0–9 and hyphens, and forbid a length of exactly seven, so an alias can never equal a future generated code.
• Reserve your own routes. api, p, raw, login, admin, static, help and anything else your site serves must never become an alias.
• Watch for impersonation. Aliases such as sbi-kyc-update or upi-refund are phishing waiting to happen. Hold aliases that contain bank, wallet or government brand names for review.
• Make retries safe. If the creator’s app times out and retries POST /v1/links, the retry finds its own alias already taken and reports a confusing 409. Store the request’s Idempotency-Key with the link and treat a repeat with the same key as success; see idempotency.

How do we expire links and pastes, and clean up after them?

Expiry has three separate jobs: stop serving on time, stop caches serving on time, and eventually reclaim storage. Mixing them up is the usual mistake.

Stop serving: check on every read. The row and the paste object both carry expires_at; the redirect and the viewer compare it with the clock and answer 410 Gone once it has passed. This check is not optional, because physical deletion is lazy. DynamoDB’s TTL, for example, deletes expired items typically within a few days, and until then they still come back from reads.

Stop caches serving: cap every TTL at the time left. Every cache entry — edge, pod LRU, Redis — gets a lifetime of min(default, expires_at − now). A paste that expires at 11:00 and is viewed at 10:58 is cached for two minutes, not ten.

Reclaim storage: delete lazily, in bulk, off the hot path. Let the database’s TTL feature remove rows; DynamoDB does it without consuming write capacity, and each TTL deletion appears in the table’s change stream as a service deletion. A small consumer of that stream deletes the matching paste object from object storage. For a store without TTL, keep a secondary index on expiry date and sweep yesterday’s bucket each night — never scan 110 billion rows looking for old ones.

Never reuse a code. A QR code printed on a 2026 Diwali flyer will still be scanned in 2028. If its code had been recycled, that scan would open a stranger’s page — perhaps a malicious one. With the space only 3% full after ten years, recycling buys nothing.

Burn-after-read pastes need an atomic read. “Show this once, then delete it” must be decided by one conditional delete — delete if the status is still active — and only the request that wins the delete gets the content, sent with Cache-Control: no-store. There is a practical trap: chat apps fetch links to build previews, so the preview bot would burn the paste before the human sees it. Show a “click to reveal” page first and burn on that POST, never on the GET.

How long should a paste live if the user sets no expiry? It is a product decision with a storage bill attached; Pastebin.com’s FAQ says it keeps such pastes but may one day remove ones nobody has viewed for six months.

How do we count clicks without slowing redirects down?

Bad: increment a counter in the database on each redirect. UPDATE links SET clicks = clicks + 1 turns every read into a write, adds a database round trip to the redirect, and piles the sponsor link’s 17k clicks a second onto one row — a lock hot spot or a throttled partition, slowing redirects exactly when the most people are watching.

Good: Redis counters. INCR clicks:3cVrDte:20261027 is atomic and fast, and fired without waiting it barely touches the redirect. But it gives only totals — breakdowns by country and referrer multiply the keys — counts can vanish in a failover, and there is no record to recompute from after a bug.

Great: count from a log, aggregate in a stream. Every request already leaves a line in the CDN’s request log. Stream those logs into Kafka and let a stream processor (see Flink and stream processing) do the counting:

• Filter first. Drop link-preview bots, crawlers and health checks, and count a repeat from the same IP and user agent within 10 seconds once; a chat app’s preview fetch is not a reader.
• Aggregate twice. If Kafka is partitioned by code, the sponsor link floods one partition. Partition randomly instead, have the first stage pre-aggregate per code, minute and country on every partition, and let a second stage, partitioned by code, merge those partial counts. The hot link now contributes one partial count per partition per minute.
• Store rollups, not events. At most one row per code per minute, compacted to hourly and daily rows after a week, read through the Stats API.
• Keep the raw events for 30 days. At 600 GB a day that is about 18 TB in object storage — cheap insurance that lets you recompute any rollup after fixing a bot filter.
• Late and repeated events. Accept events up to five minutes late; a nightly batch over the raw archive corrects the rest. Kafka delivers at least once, so deduplicate by request ID or accept a tiny overcount.

Unique visitors per link — “how many different people?” — need a set per link, which is far too big at this scale. A HyperLogLog per code per day estimates it within a couple of percent in a few kilobytes. If analytics becomes the product, as it is for ad networks, this pipeline grows into ad click aggregation.

How do we stop the service being used for phishing and malware?

A shortener hides where a link goes, which is exactly what a scammer wants: “Your SBI KYC expires today, update at sho.rt/…” A paste site is a free host for leaked passwords and malware. Abuse handling shapes the write path and justifies the 302.

Check URLs when they are created — locally. Google’s Safe Browsing APIs show the two ways to consult a list of unsafe URLs. The Lookup API sends each URL to Google’s servers — simple, but Google sees every URL you check, with no guarantee on response time. The Update API keeps hashed copies of the lists in a local database and contacts the server only when a hash prefix matches. A write path that must not wait on a third party wants the second model. (Those APIs are for non-commercial use; commercial products are pointed at Google’s Web Risk API, which offers the same two modes.)

Rate-limit creation. Per IP for anonymous users, per account for signed-in ones, with a CAPTCHA when patterns look automated (see design a rate limiter). Paste sites set these limits low: Pastebin.com allows guests 10 new pastes in 24 hours, free members 20 and paying members 250.

Re-check after creation. A link that was clean on Monday can point at a phishing page on Tuesday when its domain changes hands. Re-check links whose click rate suddenly jumps, and run a nightly pass over the links clicked that day — about 200M local lookups against the hashed list, minutes of CPU.

Take down fast. A report or a failed re-check sets status = blocked, purges the code at the edge, invalidates the pod and Redis caches, and makes the redirect serve an interstitial: “This link has been disabled because it leads to a page reported as unsafe.” With a 302 and a purge, the takedown is complete within seconds, and within 90 at worst if a purge is lost; with a bare 301 it would never reach browsers that had cached the redirect.

Scan pastes for secrets. Patterns for cloud access keys, private key headers and card numbers are cheap to match at write time; flag or quarantine matches.

The final design

Put the pieces together. Reads flow through the edge and three cache layers; writes go through one API that mints codes from reserved blocks; counting happens off to the side, from logs.

1. Edge. sho.rt resolves to the nearest CDN location, which terminates TLS and answers cached 302s (for 60 seconds) and cached paste pages (until they expire) without touching the origin. That alone takes about 43% of peak redirects.
2. Redirect. On an edge miss, a redirect pod checks its in-process LRU, which absorbs warm links and spreads hot ones across all pods.
3. Cache and database. On an LRU miss the pod reads Redis, and on a Redis miss the Links DB, filling both caches with a TTL capped at the link’s expiry. It checks status and expires_at, then answers 302, 404 or 410. Only about 3% of clicks reach the database.
4. Paste views. The Paste viewer fetches the paste object, checks expiry and visibility from its metadata, highlights it and returns HTML that the edge keeps until the paste expires. Raw text is served from the separate cookieless domain.
5. Writes. The Write API validates the URL, checks the local unsafe-URL list and the caller’s rate limit, takes the next ID from its block (one etcd call per 1,000 codes), shuffles and encodes it, and writes the row with a conditional put — plus the body to object storage for a paste. It writes the mapping through to Redis and returns 201. Custom aliases take the same conditional put.
6. Analytics. The edge streams request logs into Kafka; the aggregator filters bots, pre-aggregates and writes per-minute rollups to the Stats DB.
7. Stats and management. Owners read counts through the Write API. Deletes and takedowns update the row, purge the edge and broadcast an invalidation to the pods and Redis.

Check it against the requirements. Redirects come from the edge or from memory in about a millisecond, with a database read only on the roughly 3% of misses — well inside 20 ms at p99. The read path depends only on stateless pods, a replicated cache and a replicated store; if Redis fails, about 48k reads a second fall through to the database, which is sized to absorb them, so the outage costs latency, not errors — that is how you reach 99.99%. Codes are seven characters, unique by construction, unguessable thanks to the keyed shuffle, with a conditional put as the last guard. Creators get read-your-writes through write-through caching; edits and takedowns take seconds because every change purges the edge and invalidates the caches, and even a lost purge heals within 90 seconds. Paste bodies sit in object storage, where 7.3 TB a year is cheap, and analytics run a minute behind without touching the redirect.

What interviewers look for by level

Follow-up questions

What if users want to change where a link points after sharing it?

Allow it, but treat it as a cache problem. The Write API updates the row with a conditional write on a version number, so two editors cannot overwrite each other, then purges the edge and invalidates pods and Redis. Because the redirect is a 302 with max-age=0, no browser holds the old answer. Keep an audit trail of past destinations: when a link starts pointing at a phishing page, support needs to know when it changed and who changed it. Links created as permanent 301s should not be editable at all.

What if we need to serve users on several continents?

Run the read path in every region — redirect pods, Redis and a replica of the Links DB through a multi-region table or cross-region replication — and send users to the nearest one with latency-based DNS or anycast. Replication is asynchronous, so a link created in Mumbai may be missing in Frankfurt for about a second; on a miss, Frankfurt falls back to the link’s home region. Code generation needs no cross-region coordination: give each region its own range of block numbers, so un-shuffling a code even tells you its home region. Custom aliases must be globally unique, so route alias claims to one region for the conditional put. See multi-region and active-active.

What if the same long URL is shortened twice?

Ask what the product wants. Usually each creator gets their own code, because a code carries an owner, an expiry and its own analytics; handing out someone else’s would leak data and break deletes. Deduplicate per owner instead: index a hash of (owner_id, long_url) to the code and return the existing code on a repeat. Bulk API callers who shorten the same product page thousands of times save storage, at the cost of one extra index write per create.

What if pastes could be 100 MB files instead of 1 MB of text?

Stop sending bodies through the Write API. It creates the metadata row as pending and returns a pre-signed upload URL; the client uploads straight to object storage, in parts if the file is large, then calls “complete” and the row becomes active. A lifecycle rule cleans up abandoned uploads. Highlighting 100 MB is pointless, so render the first few hundred kilobytes and offer a download. See large file uploads and, for the storage underneath, design an object store like S3.

What if enterprise customers want their own short domain?

A bank wants go.bank.example/emi. Make the key (domain, code) instead of code, so each domain has its own namespace, route by the Host header, and issue a TLS certificate at the edge for each verified domain. Branded domains also simplify abuse handling: a bank’s domain only ever links to the bank.

How to say it in the interview

“I’ll design a URL shortener with a Pastebin mode: 30 million new links a day, 3 billion redirects a day, so a 100-to-1 read ratio and about 140 thousand redirects a second at peak. Ten years of links is about 110 billion codes, and 62 to the power 7 is 3.5 trillion, so seven base62 characters are plenty.

I’ll split the write path from the read path. The Write API validates the URL, checks a locally synced unsafe-URL list, and takes an ID from a block of a thousand it reserved from an etcd counter. It shuffles the ID with a keyed Feistel permutation so codes can’t be enumerated, encodes it in base62 and stores code-to-URL in a key-value store with a conditional put. Custom aliases use the same conditional put, so two users can never both win one alias.

Redirects return a 302, not a 301, because I want to count clicks, let owners edit links and take down phishing. I add s-maxage=60, so the CDN caches the redirect for a minute while browsers don’t. Behind the edge, stateless redirect pods check an in-process LRU, then Redis, then the database; only about 3% of clicks reach it. Writes go through to the cache, so a creator’s first click always works.

Pastes reuse all of this, except the body goes to object storage and only a 300-byte metadata row goes to the database. Pastes are immutable, so the rendered, highlighted page is cached at the edge until it expires; raw text is served from a separate cookieless domain.

Expiry is enforced on every read and caps every cache TTL; cleanup is lazy, via database TTL and a sweeper, and codes are never reused. Clicks are counted from CDN logs through Kafka and a two-stage aggregator, so a redirect never waits on analytics.”

Concepts used in this design

• Back-of-envelope estimation — turning 30M links a day into 140k redirects a second and seven characters.
• Caching strategies — cache-aside reads with write-through on create.
• Content delivery network — caching redirects and paste pages at the edge.
• Hot keys, cache stampede and invalidation — the one link everyone scans at once, and purging on takedown.
• HTTP and HTTPS — status codes, Location and Cache-Control.
• Cassandra and DynamoDB — a partitioned key-value store for 110 billion small rows.
• Redis — the shared cache, and why it should not hold the block counter.
• ZooKeeper and etcd — a strongly consistent home for the code-block counter.
• Blob storage (S3) — where paste bodies live.
• Optimistic vs pessimistic locking — conditional puts for aliases and edits.
• Idempotency — safe retries of create requests.
• Kafka — the log that decouples clicks from counting.
• Flink and stream processing — per-minute rollups with two-stage aggregation.
• Count-min sketch and HyperLogLog — unique visitors in a few kilobytes.
• Multi-region and active-active — serving redirects close to every user.

References and further reading

1. RFC 9110, Section 15.4: Redirection 3xxrfc-editor.orgExact semantics of 301, 302, 307 and 308: permanent versus temporary, method changes, and which codes are heuristically cacheable.
2. RFC 9111, Section 5.2.2.10: s-maxagerfc-editor.orgs-maxage overrides max-age for shared caches only, and heuristic freshness is not used when explicit freshness is present.
3. Ticket Servers: Distributed Unique Primary Keys on the Cheap (Flickr, 2010)code.flickr.netTwo MySQL ticket servers handing out odd and even IDs with REPLACE INTO, in production since 2006.
4. Snowflake (Twitter, 2010 README)github.comThe 41-bit time, 10-bit machine and 12-bit sequence layout, coordination-free generation, and refusing to issue IDs when the clock goes backwards.
5. Amazon DynamoDB Developer Guide: conditional writesdocs.aws.amazon.comPutItem replaces an existing item by default; a condition expression makes the write fail with ConditionalCheckFailedException instead.
6. Amazon DynamoDB Developer Guide: time to live (TTL)docs.aws.amazon.comExpired items are deleted typically within a few days, without consuming write throughput, and appear in streams as service deletions.
7. Redis INCR commandredis.ioAtomic increment of a 64-bit signed counter, the building block for a simple global counter.
8. Google Safe Browsing APIs (v4) overviewdevelopers.google.comLookup API versus Update API (local hashed lists), their privacy and latency trade-offs, and the non-commercial-use restriction.
9. Pastebin.com FAQpastebin.comA real paste site's limits: 512 KB per paste for free users, daily paste caps for guests and members, visibility levels and expiry.