Design an object store like S3

An object store is the simplest storage API there is — put bytes under a name, get them back, delete them, list names by prefix — and one of the hardest systems to build at scale. Amazon S3, the system most people mean by “object storage”, held well over 100 trillion objects and served tens of millions of requests a second as of 2021, by its own CTO’s account. Three problems make it hard. Durability: losing one object in 100 billion per year is the target, on hardware where a disk dies somewhere every day. Consistency: the moment a PUT succeeds, every reader and every LIST must see it, even though metadata and bytes live in different fleets. And scale without hot spots: a trillion keys, a single index that must stay sorted for LIST, and customers who suddenly hammer one prefix or one object.

Understand the problem

An object store looks like a key-value store, so say early what makes it different: values range from zero bytes to tens of terabytes, objects are immutable once written (you replace them, you never edit them in place), and the only query besides “get this key” is “list keys with this prefix”. There are no rename, append or lock operations — this is not a file system, and file sync is a separate problem (Dropbox is built on top of something like this).

Functional requirements

1. Users should be able to create buckets and PUT, GET and DELETE objects by key, with user metadata and byte-range reads.
2. Users should be able to upload very large objects in parts, in parallel, retrying only the parts that fail.
3. Users should be able to list objects in a bucket by prefix, optionally grouping keys into “folders” with a delimiter.
4. Users should be able to turn on versioning for a bucket, so overwritten and deleted objects can be recovered.

Non-functional requirements

• Durability: 99.999999999% (eleven nines) per object per year, surviving the loss of an entire availability zone. That is S3 Standard’s stated design target, with data spread over at least three zones that sit within 100 km of each other.
• Availability: 99.99% for reads and writes in a region — about 53 minutes a year.
• Strong consistency for single keys. After a successful PUT or DELETE, every GET, HEAD and LIST reflects it. A GET that races a PUT returns the old object or the new one, never a mix. Concurrent writers to one key: last writer wins. No multi-key transactions.
• Scale. One region: 1 trillion objects, 400 PB of data, objects from 0 bytes to 48.8 TiB, and 5 million requests a second at peak, with no limit on objects per bucket.
• Latency. Time to first byte for small objects around 20 ms at p50 and under 200 ms at p99; large objects stream at many gigabits a second when read in parallel ranges.
• Multi-tenancy. One customer’s burst must not slow down anyone else, and no customer can see another’s data.
• Cost. Raw storage well under 3× the logical bytes; disks are most of the bill.

Out of scope

• Cross-region replication and storage classes such as archive tiers — both are follow-up questions.
• The permission system. We assume an IAM-style service answers “may this principal do this action on this bucket and key?” in a millisecond.
• Query features such as running SQL over objects, and static website hosting.
• File-system semantics: rename, append, locks, and syncing folders to laptops.
• Billing and metering, beyond noting that every request and byte is counted.

The set-up

Plan the approach

State the central idea first: separate “where is it and what is it” from “the bytes themselves”. A metadata plane holds a sorted index of keys; a data plane stores bytes on a huge fleet of disks; a stateless front end ties the two together; and background services repair, scrub, compact and expire. AWS describes S3 at this level the same way — a front-end fleet with a REST API, a namespace service, a storage fleet full of hard disks and a fleet for background work — in a 2023 write-up by one of its engineers. Then go deep where the difficulty is: durability, the index, consistency, deletes, heat and integrity.

Core entities

• Bucket. A globally unique name, an owner account, a home region and settings (versioning, lifecycle rules, policies).
• Object version. Bucket, key, version ID, size, ETag, checksums, content type, user metadata, creation time and a manifest — the list of byte ranges that make up the object and where each range lives.
• Extent. A large, append-only container of bytes (about 1 GB) on the storage fleet. Many small objects share one extent; a huge object spans many. Sealed extents are erasure-coded into shards.
• Multipart upload. An upload ID, the target bucket and key, and the parts received so far: part number → size, ETag, checksum and manifest.
• Partition. A contiguous range of the key index, owned by one replicated group of index servers.

One entry in the key index looks like this:

{
  "key": "acme-logs/2026/10/27/app-03.log.gz",
  "version": "v-0f9a2c71",
  "size": 198180864,
  "etag": "\"4d1c…-3\"",
  "checksum_crc64nvme": "Lq0Q8z1rWUo=",
  "content_type": "application/gzip",
  "manifest": [
    { "extent": "e-9f31c2", "offset": 0,         "length": 67108864 },
    { "extent": "e-9f31c7", "offset": 0,         "length": 67108864 },
    { "extent": "e-a00412", "offset": 524288000, "length": 63963136 }
  ]
}

The index entry is about a kilobyte; the bytes it points to can be terabytes. Keeping those two apart is the whole design.

API

The API is plain HTTP on paths of the form /{bucket}/{key}:

PUT /acme-logs/2026/10/27/app-03.log HTTP/1.1
Host: store.example
Content-Length: 5242880
x-checksum-crc64nvme: Lq0Q8z1rWUo=
If-None-Match: *

HTTP/1.1 200 OK
ETag: "9b2cf535f27731c974343645a3985328"
x-version-id: v-0f9a2c71

If-None-Match: * makes the PUT succeed only if the key does not exist yet (412 Precondition Failed otherwise); If-Match with an ETag gives compare-and-swap. Reads support ranges, which is how clients pull a large object in parallel:

GET /acme-logs/2026/10/27/app-03.log.gz HTTP/1.1
Range: bytes=0-8388607

HTTP/1.1 206 Partial Content
Content-Range: bytes 0-8388607/198180864

Multipart upload is three calls plus a cleanup call, and LIST takes a prefix, a delimiter and a continuation token:

POST   /{bucket}/{key}?uploads                      start; returns uploadId
PUT    /{bucket}/{key}?partNumber=7&uploadId=u-7f3   upload one part (any order, in parallel)
POST   /{bucket}/{key}?uploadId=u-7f3               complete: list of part numbers and ETags
DELETE /{bucket}/{key}?uploadId=u-7f3               abort and free the parts
GET    /{bucket}?prefix=2026/10/&delimiter=/&max-keys=1000&continuation-token=…
DELETE /{bucket}/{key}                              delete (or add a delete marker if versioned)

Back-of-envelope estimates

Assume one large region. (Method: back-of-envelope estimation.)

Objects and bytes. 1 trillion objects averaging 400 KB: 10¹² × 400 KB = 400 PB of logical data. The average hides a skewed distribution: most objects are a few kilobytes, most bytes sit in large objects.

Requests. 5M requests/s at peak: 4M reads (GET, HEAD) and 1M writes (PUT, DELETE, part uploads). Reads average about 100 KB — small objects are requested far more often, and big ones are read in ranges: 4M × 100 KB = 400 GB/s ≈ 3.2 Tbit/s out. Writes at the same average: 1M × 100 KB = 100 GB/s in.

Index. About 1 KB per entry: 10¹² × 1 KB = 1 PB of metadata, 3 PB with three replicas. At about 10 GB per partition that is 100,000 partitions. Every request touches the index, so 5M index operations a second ÷ 100,000 ≈ 50 per partition on average — but averages lie; a single hot prefix can want thousands a second.

Raw capacity. With erasure coding at 9 data + 6 parity shards, overhead is 15 ÷ 9 ≈ 1.67×: 400 PB × 1.67 ≈ 667 PB, or ≈ 833 PB of disks if you keep them 80% full. On 26 TB drives: 833 PB ÷ 26 TB ≈ 32,000 drives. With 3× replication it would be 1,500 PB ÷ 26 TB ≈ 58,000 drives.

Random reads. A hard disk does about 120 random reads a second, a number that has barely moved since 2006. 32,000 × 120 ≈ 3.8M random reads/s for the whole fleet — less than the 4M reads a second we need at peak, even if every read cost exactly one seek.

Failures. Assume a 1% annual failure rate: 32,000 × 1% = 320 dead drives a year, about one a day, each holding about 21 TB.

The numbers that shape the design: 1 PB of index means metadata is a distributed database of its own; 1.67× versus 3× saves about 26,000 drives; and 3.8M seeks for 4M reads means disk I/O, not capacity, is the scarce resource — so heat must be spread perfectly and hot data cached.

High-level design

Build it one requirement at a time. By the end of this section the store works correctly in one region; the deep dives make it durable, fast and cheap.

Users should be able to create buckets and PUT, GET and DELETE objects

Buckets are rare, small and global: a bucket registry maps each name to its owner, region and settings. It is a small, strongly consistent database, so two customers can never create the same name. Settings are cached everywhere, so a configuration change may take a while to reach every front end. S3 documents exactly this: bucket configuration is eventually consistent, and after enabling versioning it recommends waiting 15 minutes before writing.

Objects go through three components:

• Front ends, stateless and many: they authenticate, enforce limits, find the right index partition, and stream bytes to and from the storage fleet.
• The key index (metadata plane): a sorted map from bucket/key/version to the object’s manifest, split into partitions, each replicated across three zones with a consensus protocol such as Raft.
• Storage nodes (data plane): servers full of disks that store extents and know nothing about keys or buckets.

A PUT (numbers match the figure):

1. The client sends the bytes with a checksum. The front end authenticates it and checks the bucket’s policy.
2. It streams the bytes to storage nodes in all three zones — as erasure-coded shards for a large object, as three replicas for a small one (the first deep dive explains why).
3. The storage nodes acknowledge once the data is durable on disk.
4. The front end commits a manifest to the key index partition that owns the key. This single write is when the object starts to exist.
5. It returns 200 OK with the ETag and version ID.

A GET runs the other way: read the manifest from the index, then fetch the byte ranges it lists — in parallel, from whichever shards answer fastest — verify checksums, and stream them out. A DELETE only touches the index; the bytes are reclaimed later.

Why the strict order on PUT? Because the two halves can fail independently, and the order decides what a failure looks like.

Two details from the figure come back later. The front end acknowledges after 14 of 15 shards are durable rather than waiting for the slowest disk — the durability deep dive shows why that is still safe. And the index stamps each commit with a sequence number per key, which is what makes consistency and versioning work.

Users should be able to upload very large objects in parts

A 2 TB database backup cannot be one HTTP request: a dropped connection at 1.9 TB would mean starting again, and one stream is limited by one TCP connection. Multipart upload turns one object into many independent part uploads.

1. Initiate. The front end creates an upload record in the index partition that owns the key and returns an upload ID. Nothing is visible yet.
2. Upload parts, in any order and in parallel. Each part is stored exactly like a small object — bytes to the data plane, then the part’s manifest added to the upload record. Re-sending a part number simply replaces that part, so a failed part is retried alone.
3. Complete. The client sends the list of part numbers with their ETags. The front end checks that every part exists and matches, then writes one index entry whose manifest is the parts’ manifests concatenated in order. No bytes are copied; the commit is atomic, so readers see either no object or the whole object.

The limits matter because they shape client behaviour. S3 allows up to 10,000 parts of 5 MiB to 5 GiB each (the last part may be smaller), which caps an object at 10,000 × 5 GiB ≈ 48.8 TiB, and suggests multipart from about 100 MB. The minimum part size keeps the manifest from being a million tiny entries; the maximum part count keeps it from being unbounded. So clients choose part size from object size: a 1 TiB object needs parts of at least 1 TiB ÷ 10,000 ≈ 105 MiB.

Two loose ends become deep-dive material. Parts of an upload that is never completed or aborted are invisible to LIST yet still occupy disk, so a lifecycle rule must abort stale uploads. And the object’s checksum must be built from the parts’ checksums without re-reading terabytes.

Users should be able to list objects in a bucket by prefix

LIST is why the index is sorted. Because keys of one bucket are stored in order, “all keys under logs/2026/10/” is one contiguous range: seek to the prefix, scan forward, stop at the first key that no longer matches. A page returns up to 1,000 keys plus a continuation token — the last key returned — so the next page resumes with another seek.

The delimiter creates the illusion of folders. With delimiter=/, any key with another / after the prefix is collapsed into a common prefix: logs/2026/10/01/ stands for every object inside it. The scan does not need to visit all of them; after emitting a common prefix, it seeks straight past it.

LIST is also the main argument against hashing keys across partitions, the usual way to spread load. Hashing would scatter logs/2026/10/ over all 100,000 partitions, and every LIST would have to ask all of them. The second deep dive keeps range partitioning and deals with its downside, hot spots.

Users should be able to turn on versioning

With versioning on, the index key becomes bucket/key/version, where the version ID sorts newest first (an inverted timestamp plus the commit sequence). A PUT adds a version instead of replacing one; a GET without a version ID returns the newest; a DELETE adds a delete marker as the newest version, so plain GETs return 404 while every older version is still readable by ID. Removing the delete marker “undeletes” the object. Without versioning, a PUT overwrites the single entry and a DELETE removes it; in both cases the old bytes become garbage for the data plane to reclaim.

You now have a correct object store. Every deep dive below is about making it durable, fast and cheap at a trillion objects.

Deep dives

How do we reach eleven nines without paying for three copies?

List the failures first. With 32,000 drives, one dies about every day. Servers and racks fail less often but take many disks at once. A whole zone can go dark from a fire, flood or power fault. And some failures are silent: a disk returns wrong bytes, or software writes garbage (the integrity deep dive covers those).

Bad: replicas in one zone, or RAID inside one server. Cheap and fast, but a zone outage or a server fire loses data outright. Eleven nines is impossible if a single building holds every copy.

Good: three replicas, one per zone. Lose a whole zone and a disk elsewhere, and one copy still survives. Repair is a simple copy, and any of the three replicas can serve a read, which helps with hot data. The price is 3× raw storage — about 58,000 drives for our 400 PB — and every write crosses the network three times.

Great: erasure coding across zones. A Reed–Solomon code splits data into k data shards and computes m parity shards; any k of the k + m rebuild the data. S3’s engineers describe using exactly this — k “identity” shards plus m parity shards — and Facebook’s f4 used Reed–Solomon(10,4) at 1.4× within a data centre, cutting its effective replication factor from 3.6 to 2.8, and then to 2.1 with XOR coding across data centres.

The zone constraint fixes the minimum cost. Spread k + m shards evenly over three zones; losing one zone removes a third of them, so you need m ≥ (k + m) ÷ 3, which means an overhead of at least 1.5×.

So 9 + 6 across three zones, five shards each, matches three replicas’ protection at 56% of the bytes: about 32,000 drives instead of 58,000. It also explains the 14-of-15 acknowledgement: even if the missing shard is in a surviving zone, losing a whole zone leaves 14 − 5 = 9, enough to rebuild. What erasure coding costs you:

• Repair traffic. Rebuilding one lost shard means reading nine others. A dead 21 TB drive costs about 9 × 21 ≈ 190 TB of reads; at one failure a day that is roughly 190 TB ÷ 86,400 s ≈ 2.2 GB/s of background traffic, forever, against about 0.24 GB/s for replicas.
• Small objects. Coding a 4 KB object into 15 slivers would turn every read into nine disk seeks. So small writes first land in an open extent replicated three ways, one copy per zone — fast and durable — and when the extent reaches about 1 GB it is sealed and erasure-coded in the background. Windows Azure Storage described this exact pattern: append-only 1 GB extents, sealed, then coded down from 3× to 1.3–1.5×. Code in large blocks so that each small object sits whole inside one data shard: a normal read is still one disk read, and only a degraded read touches nine. f4 reads BLOBs this way and, when a block is missing, reconstructs just the requested BLOB, not the whole 1 GB block.
• Large objects are striped across the nine data shards as they stream in, which is a gift: a range GET reads from nine disks in parallel.
• Tail latency. For reads of large ranges, ask ten or eleven shards and use the first nine to arrive; a slow disk no longer sets your p99.

Repair speed is the hidden durability knob. Copy a dead drive’s 21 TB onto one spare at about 200 MB/s and it takes 21 TB ÷ 200 MB/s ≈ 29 hours, during which those stripes run with less protection. But the lost shards belong to thousands of different stripes, so rebuild them in parallel across the fleet: 190 TB of reads spread over 1,000 disks at 50 MB/s each is 50 GB/s, or about an hour. A toy model shows why that matters. Take three replicas and a 1% annual failure rate per disk: with a 24-hour repair the chance of losing a given object in a year is about 4.5 × 10⁻¹¹ — roughly ten nines — and with a one-hour repair it falls to about 8 × 10⁻¹⁴. Repair time enters the formula squared. Run the same model for 9 + 6 with one-hour repair and you get thirty-odd nines.

How do we partition a trillion-key index without hot spots?

The index must answer point lookups for every GET and PUT, ordered scans for every LIST, 5M operations a second at peak, and hold 1 PB. No single database does that.

Bad: hash partitioning. Consistent hashing of keys spreads load evenly and makes sequential names harmless. But it destroys order: a LIST of logs/2026/10/ must ask all 100,000 partitions and merge their answers, so one LIST page costs 100,000 requests. And it does nothing for a single hot key.

Good: one partition per bucket. LIST stays local and the mapping is trivial. But buckets range from ten objects to tens of billions; the giant ones overflow any one server, and a busy bucket cannot borrow capacity from idle ones.

Great: key-range partitions that split, move and merge by load. Cut the sorted key space into contiguous ranges, each owned by one replicated group of index servers (three replicas across zones, kept in step by Raft). A small, strongly consistent partition map says which group owns which range; front ends cache it. A partition manager watches each range’s request rate, size and CPU and does three things, which are exactly the three operations Windows Azure Storage’s partition manager was built around: load-balance (move a busy range to a quieter server), split (cut a range that is too hot or too big at a key chosen from real traffic) and merge (join cold neighbours so the partition count stays manageable).

S3 documents what this looks like from outside. Each partitioned prefix supports at least 3,500 writes or 5,500 reads a second, there is no limit on the number of prefixes, so ten prefixes can serve 55,000 reads a second — and scaling up is gradual, with 503 Slow Down errors while it happens. That is a split in progress: the manager has noticed the load, chosen a split key, and is copying one half’s few gigabytes to another group and catching up its log before switching the map. Front ends shed the excess with 503s, and well-behaved clients back off with jitter instead of retrying harder.

Three details separate a good answer from a great one:

• Sequential keys defeat splitting. If every new key is a timestamp, every write lands at the end of the key space. Split the last partition and the hot end simply moves to the new right-hand half. The service can pre-split ahead of the write frontier; the customer can put a high-cardinality component early in the key (sensor-417/2026-10-27/…), at the cost of listing a day’s data with several prefix scans in parallel.
• Stale partition maps are normal. After a split, some front ends still route to the old owner. The old owner replies “not mine — here is map version 8,812”, the front end refreshes and retries. No central lookup sits on the request path.
• A single hot key cannot be split. If one object is read 50,000 times a second, its index entry lives in exactly one partition. That needs caching — of metadata in the front ends and of bytes in a flash tier — which is why the next deep dive matters.

How do we keep reads strongly consistent when metadata is cached?

The index on its own is strongly consistent. Each partition’s leader orders the writes to its keys and serves reads only while it holds a valid lease, so a GET or LIST that starts after a PUT was acknowledged sees it. If every request read the index, we would be done.

But 4M reads a second, heavily skewed towards popular objects, would pile onto a few partition leaders. So you cache metadata in front of the index — and a cache can serve the old manifest after a new one is committed elsewhere. That is why, until December 2020, S3 could occasionally return stale data or listings just after a write: its metadata tier used a highly resilient cache, and writes might flow through one part of the cache while reads queried another. Customers built their own fixes — Netflix’s s3mper, Hadoop’s S3Guard — tracking writes in a separate database.

Bad: a short TTL on cached entries. Staleness drops to a second, but “usually fresh” is not consistency. A data pipeline that writes a file and immediately lists the folder will still miss it now and then, at scale many times a day.

Good: skip the cache and always read the index. Correct, but it adds a round trip to every GET and concentrates hot keys on single partition leaders. S3’s team considered this and rejected it because it would cost performance.

Great: keep the cache, but check freshness against a witness. Every commit gets a per-key sequence number. A witness service is told about every write and keeps, in memory, the latest sequence number for each recently written key. On a read, the cache asks the witness one tiny question — what is the newest sequence for this key? — and serves its copy only if it matches; otherwise it invalidates the entry and reads the index. Vogels describes S3’s version of this: a component that is “a witness to writes”, acts as a read barrier, holds minimal state only in memory, and can be replaced quickly because it has no durable state to transfer.

The cost is one more in-memory lookup per read, and two correctness rules. The witness must hear about a write before the writer is acknowledged, or a read that starts after the 200 could slip past the barrier. And a lost witness must fail safe: a replacement knows nothing, so until it has rebuilt its view, caches treat every key as possibly stale and read through to the index. S3’s team went further and verified the protocol with proofs and model checking, because, as Vogels put it, something that happens once in a billion requests happens several times a day at their scale. The result shipped on 1 December 2020: GET, PUT and LIST strongly consistent for every object, in every region, at no extra cost or latency.

What strong consistency does not give you is worth stating too, and S3’s consistency model is explicit about it. Two concurrent PUTs to one key are resolved by last-writer-wins on timestamp, and there are no multi-key transactions. For coordination, use conditional requests, which the partition leader checks at commit time: If-None-Match: * creates an object only once (a simple lock or a run-once marker), and If-Match with an ETag is compare-and-swap. To publish many objects atomically, write them all first and then write one small manifest object that names them; readers trust only what the manifest lists.

How do deletes, versions and garbage collection reclaim space?

A DELETE is a cheap index write; the bytes stay where they are. They live inside shared, sealed, erasure-coded extents, and you cannot punch a hole in a coded stripe without re-encoding it. Deleting immediately would also be dangerous: a bug that deletes the wrong things should be recoverable for a while.

Space comes back in bulk. A garbage collector follows the index’s change log to keep a count of live bytes per extent. When an extent is mostly garbage — say over 60% — compaction copies its live objects into a fresh extent, then updates each moved object’s manifest with a conditional index write (“only if it still points at extent E17”), because the object may have been overwritten or deleted meanwhile. The old extent is reclaimed only after a grace period of days, long enough for in-flight reads holding old manifests to finish and for anyone to notice a GC bug. The same machinery collects orphan shards from PUTs that crashed before their commit. Windows Azure Storage works the same way: an extent no longer referenced by any stream is garbage-collected.

What you pay is write amplification: to free 600 MB of E17 you rewrite 300 MB of live data and re-encode it. Raise the threshold and you copy less but carry more dead bytes; that trade-off is tuned per storage tier.

Three kinds of garbage need explicit rules, or they grow forever:

• Old versions. With versioning on, overwritten and deleted objects keep their bytes. A lifecycle rule such as “expire noncurrent versions after 30 days” is what keeps versioned buckets from growing without limit.
• Abandoned multipart uploads. Parts are invisible to LIST but occupy disk. A lifecycle rule aborts uploads that have not completed within, say, seven days, which drops their references and hands them to GC.
• Expiring data. Rules like “delete logs/ after 90 days” run in a background lifecycle engine that walks index partitions daily, in throttled batches, through the normal delete path.

How do we place data so no disk becomes a hot spot?

The estimates already showed the squeeze: about 4M reads a second against 3.8M seeks of disk capacity. It gets worse with every hardware generation, because, as S3’s Andy Warfield puts it, capacity keeps growing while a drive still manages only about 120 random operations a second; with the 200 TB drives on industry roadmaps, a fair share of I/O falls to about one operation per second per 2 TB stored.

Bad: fill disks one after another, or keep each customer’s data on a few disks. One busy bucket saturates its disks and everyone else on them queues — and queueing at one disk becomes tail latency for every request that needs a shard from it.

Good: spread every extent randomly, with constraints — the right number of shards per zone, never two on one rack, enough free space. Random placement averages load well across tens of thousands of disks.

Great: spread widely, place by current heat, and keep hot bytes off disk.

• Spread each bucket over as many disks as possible. Warfield describes putting different objects on different sets of drives, so tens of thousands of customers have buckets spread over millions of drives. A customer occupies a sliver of each disk, so no one tenant can make a disk hot — and a sudden burst gets the seek capacity of a million disks. Scale helps here: millions of bursty, independent workloads add up to a smooth total.
• Choose by heat, not just space. For each new extent, pick two random candidate disks per slot and take the cooler one — the “power of two choices” keeps the hottest disk close to the average.
• Beware the empty new rack. Placement that prefers free space would send all new data — the hottest data — to the rack you just installed. Cap the write rate per disk, fill new disks gradually, and move cold extents onto them to balance.
• Read around trouble. Open extents have three replicas; read from the least busy one. For a coded extent, if the disk holding a data shard is slow, reconstruct from other shards — nine reads instead of one, so only as a last resort.
• Cache what is hot. A flash tier in front of the disks, holding a few percent of the bytes, absorbs a large share of reads for popular objects; public objects can sit behind a CDN.

How do we know the bytes we return are the bytes we stored?

Disks, memory, network cards and software all corrupt data occasionally. Warfield puts a drive’s error rate at about 1 in 10¹⁵ bits read and adds that, at S3’s scale, such misses happen often enough to engineer for. Our 400 GB/s of reads is 3.2 × 10¹² bits a second; over a day that is about 2.8 × 10¹⁷ bits, or roughly 280 flipped bits a day if nothing checked.

• End to end on upload. The client computes a checksum and sends it with the bytes. S3’s SDKs default to CRC64NVME and also accept CRC32, CRC32C, SHA-1, SHA-256 and others; S3 recomputes it independently. Our front end does the same while streaming, and a mismatch means 400 and no commit — corrupted bytes never become an object.
• Per shard at rest. Every shard is stored with its own checksum, checked on every read; a mismatch is treated like a missing shard and rebuilt from the others.
• Multipart without re-reading. Each part arrives with its own checksum. CRCs can be combined mathematically, so the front end can compute a whole-object CRC from the part CRCs at completion instead of re-reading 48 TiB.
• Scrubbing. Data that nobody reads can rot unnoticed until the day it is needed — and needed alongside its other shards. Scrubbers re-read every shard on a schedule. Re-reading 833 PB every two weeks is 833 PB ÷ 1.2M s ≈ 690 GB/s — more than all customer reads — but spread over 32,000 drives it is only about 22 MB/s per drive, sequential and schedulable into quiet hours.

The final design

Put it together: stateless front ends, a metadata plane of range partitions with a witness, a data plane of coded extents across three zones, and background services that keep everything healthy.

1. Front door. A client request reaches a front end through DNS and load balancers. The front end authenticates it, checks the bucket policy and per-tenant limits, and answers 503 Slow Down when a partition is mid-split.
2. Route. It finds the partition that owns the key in its cached partition map; a stale map is corrected by the partition that answers.
3. Write the bytes. On a PUT, small objects are appended to an open extent replicated across three zones; large ones stream as 9 + 6 coded shards, five per zone. The front end waits for 14 of 15, checking the client’s checksum as the bytes pass.
4. Commit. It commits the manifest to the partition’s Raft group — the moment the object exists — honouring any If-Match or If-None-Match condition, and returns 200. Multipart completion is the same single commit over many parts.
5. Read. On a GET, the front end checks its cached manifest against the witness, which the index updated before acknowledging the write; if stale, it reads the index. It then fetches ranges from the fastest shards (or the flash tier for hot objects), verifies checksums and streams the object.
6. Background. Repairers rebuild lost shards in parallel across the fleet in about an hour; scrubbers re-read everything every two weeks; GC compacts mostly-dead extents after a grace period; lifecycle expires versions and abandoned uploads; placement steers new extents to cool disks and fills new racks gradually.

Check it against the requirements. Durability: 9 + 6 coding across three zones survives a zone plus a disk, repairs in about an hour, and verifies every byte end to end — leaving correlated failures and bugs as the real risks, handled by diversity, slow rollouts and delayed deletes. Availability: every component is replicated across zones; losing a zone leaves index majorities and enough shards to keep serving. Consistency: one commit per object in a consensus-replicated index, with witness-checked caches, gives read-after-write for GET, HEAD and LIST. Scale and latency: 100,000 range partitions that split by load, about 32,000 drives at 1.67×, heat-aware placement and a flash tier keep small-object reads in tens of milliseconds. Cost: 1.67× instead of 3× saves roughly 26,000 drives.

What interviewers look for by level

Follow-up questions

What if customers need a copy in another region?

Replicate asynchronously from the index’s change log: each committed PUT or DELETE becomes an event that a replication service copies to the destination bucket, bytes first and manifest last, as in any PUT. Require versioning on both sides so that replays are idempotent by version ID and deletes travel as delete markers. Concurrent writes in two regions resolve by last-writer-wins on timestamp. Measure and alarm on replication lag, because that is the data you lose if a region disappears.

What if most data is rarely read and must be cheaper?

Add colder storage classes. Seal and re-encode cold extents with wider codes (fewer parity bytes per data byte) on denser, slower drives, or move them to an archive tier where reads need a restore step taking minutes to hours. Lifecycle rules move objects by age, and minimum storage durations stop customers from bouncing data between tiers. The index entry stays the same; only the manifest’s locations and the storage class change.

What if a customer writes billions of tiny objects?

Then the index, not the disks, is the cost. A 1 KB object needs about 1 KB of index entry replicated three times, plus a write per object. Packing objects into shared extents keeps the data plane efficient, but you must also price per request and per object, and suggest batching small records into larger objects. Facebook’s BLOB stores made the same choice, packing many photos into large volume files.

What if an entire zone goes down?

Index groups still have two of three replicas, so they keep a majority and continue to commit. Objects whose data shard sat in the lost zone are rebuilt on read from the ten surviving shards — nine reads instead of one — so the read path leans harder on caches. New writes cannot get five shards per zone into three zones, so place them as temporary 9 + 6 stripes over two zones and re-spread them when the zone returns, or write three replicas over the two remaining zones and code them later. Repair of the lost zone’s shards starts only after a timeout, because rebuilding a whole zone’s data is far more expensive than waiting out a power cut.

How to say it in the interview

“I’ll design a regional object store: about a trillion objects, 400 petabytes, 5 million requests a second at peak, eleven nines of durability and strong read-after-write consistency.

The core idea is separating metadata from data. Stateless front ends authenticate and stream the bytes to storage nodes first. Only when they are durable does the front end commit a small manifest — key, size, checksums and where the bytes live — into a sorted key index. That commit is the moment the object exists, so a crash before it just leaves invisible garbage. The index is a petabyte, so it’s split into about 100,000 key-range partitions, each replicated by Raft across three zones. I use ranges, not hashing, because LIST by prefix must be one scan; a partition manager splits hot ranges at keys chosen from live traffic.

For durability, small objects land in replicated open extents that are sealed at a gigabyte and erasure-coded nine plus six, five shards per zone. That survives a zone plus a disk at 1.67 times the bytes instead of three. Repairs run in parallel across the fleet in about an hour, and repair time matters more than the number of copies; the real risks are correlated failures and bugs.

Metadata caches are checked against a witness that knows each key’s newest version, so cached reads stay strongly consistent. Multipart upload stores parts independently and completes with one atomic manifest commit. Deletes are index writes; compaction reclaims space after a grace period. Finally, disks give only about 120 random reads a second, so I spread every bucket across many disks, place by heat, cache hot bytes on flash, and checksum everything end to end, with scrubbers re-reading all data every two weeks.”

Concepts used in this design

• Blob storage (S3) — the user-facing view of the system designed here.
• Back-of-envelope estimation — turning a trillion objects into drives, partitions and seeks.
• Data replication — replicas versus codes, and repair.
• Consensus and leader election — Raft groups behind each index partition.
• Database sharding — range partitioning, splitting and rebalancing.
• Consistent hashing — the alternative that would break LIST.
• Strong vs eventual consistency — what changed when S3 became strongly consistent.
• Caching 101 — metadata caches and the flash tier, and why caches threaten consistency.
• Checksums — end-to-end integrity and scrubbing.
• Large file uploads — multipart from the client’s side.
• Availability and the nines — durability nines versus availability nines.
• Fault tolerance — failure domains: disk, rack, zone.
• Multi-tenancy and isolation — spreading tenants so no one makes a disk hot.
• Cost-aware design — 1.67× versus 3× at hundreds of petabytes.

References and further reading

1. Building and operating a pretty big storage system called S3 (Andy Warfield, All Things Distributed, 2023)allthingsdistributed.comS3's front end, namespace, storage and background fleets; about 120 random IOPS per hard disk; heat management; Reed–Solomon shards across tens of drives; ShardStore and durability reviews.
2. Diving Deep on S3 Consistency (Werner Vogels, 2021)allthingsdistributed.comHow S3 made its metadata cache strongly consistent with per-object ordering and an in-memory witness acting as a read barrier.
3. Amazon S3 Update – Strong Read-After-Write Consistency (AWS News Blog, 1 December 2020)aws.amazon.comThe announcement: GET, PUT and LIST strongly consistent for all objects, in all regions, at no extra cost or latency.
4. Amazon S3 data consistency modeldocs.aws.amazon.comRead-after-write for PUT, DELETE and LIST; atomic single-key updates; last-writer-wins for concurrent writers; eventually consistent bucket configuration.
5. Best practices design patterns: optimizing Amazon S3 performancedocs.aws.amazon.comAt least 3,500 writes or 5,500 reads per second per partitioned prefix, gradual scaling with 503 Slow Down, and roughly 100–200 ms small-object latency.
6. Amazon S3 multipart upload limitsdocs.aws.amazon.comUp to 10,000 parts of 5 MiB to 5 GiB each, a maximum object size of 48.8 TiB, and multipart recommended from about 100 MB.
7. Data protection in Amazon S3 (durability)docs.aws.amazon.comDesigned for 99.999999999% durability and 99.99% availability, with data across at least three Availability Zones within 100 km of each other.
8. Checking object integrity in Amazon S3docs.aws.amazon.comSupported checksum algorithms (CRC64NVME by default, CRC32, CRC32C, SHA-1, SHA-256 and more) and server-side recomputation on upload.
9. f4: Facebook's Warm BLOB Storage System (Muralidhar et al., OSDI 2014)usenix.orgReed–Solomon(10,4) over 1 GB blocks, effective replication cut from 3.6 to 2.8 and 2.1, degraded reads of single BLOBs, and offline rebuilders.
10. Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency (Calder et al., SOSP 2011)sigops.orgStream, partition and front-end layers; append-only 1 GB extents that are sealed and erasure coded to 1.3–1.5×; range partitions that load-balance, split and merge.