1 PB
of metadata, just to find the bytes
Assume 1 trillion objects, 400 PB of data. Each needs about 1 KB of index, so metadata is a distributed database of its own.
Which goes first, bytes or name?
Bad: Name first
A crash leaves a manifest pointing at missing bytes.
Good: Bytes first, commit last
A crash leaves invisible orphan shards. The index commit is when the object exists.
Three copies’ protection at 1.67× the bytes, not 3×
Code each extent into 9 data + 6 parity shards, five per zone. Any 9 rebuild it, so losing zone c and a disk still leaves 9.
LIST needs keys sorted, so no hashing. But one prefix can want thousands of requests a second.
The trick
Split the hot range at a key from live traffic; move half to a quieter server.
120/s
random reads per disk, barely moved since 2006
32,000 drives give 3.8M seeks a second; peak wants 4M reads. So spread every bucket wide, place by heat and cache hot bytes on flash.
Until December 2020, S3 sometimes listed stale results after a write
Its metadata cache let writes flow through one part while reads queried another. Customers built their own fixes, like Netflix’s s3mper.
The fix: a witness to every write
- Every commit gets a per-key sequence number
- The witness hears before the writer’s 200
- Serve the cache only if its sequence matches
- A new witness knows nothing: read through
Remember
Bytes first, then one manifest commit in a sorted, range-split index: that’s when an object exists. Code 9 + 6 across three zones, repair fast, and check caches against a witness.