Blob storage (S3)

Blob storage, also called object storage, keeps arbitrary bytes (photos, videos, backups, logs, model checkpoints) under a key in a bucket, behind a small HTTP API: PUT, GET, DELETE and LIST. Amazon S3 is the reference design; Google Cloud Storage and Azure Blob Storage work the same way. In an interview it is the default home for anything over a few hundred kilobytes, provided you can say how bytes get in and out without touching your servers, what is consistent, and what the bill looks like.

Picture it first: a railway cloakroom

A big railway junction has a cloakroom. You hand over a sealed bag; the clerk writes a token such as PLT4/2026-10-27/0815 and shelves the bag unopened. The slashes mean nothing to him: his register is one list sorted by token, and if you ask for everything starting PLT4/, he reads it out a page at a time. To add a shirt you take the bag out, repack it and deposit it again under the same token. A tag on the handle says whose it is.

A signed chit, valid until 6 pm, lets a friend collect the bag without your ID. Bags left a month move to a cheaper back room; after a year they go to a warehouse outside town, and fetching one takes half a day.

Map it back: the cloakroom is a bucket in a region, the token is the key, the bag is the object, the tag is metadata, repacking is a whole-object PUT, reading the register by prefix is LIST, the chit is a presigned URL, and the back room and warehouse are storage classes reached by lifecycle rules.

Data model: buckets, keys and objects

Bucket. A named container in one region, fixed at creation. Names are 3–63 characters and by default unique across all AWS accounts (an account-regional namespace is now an option). The bucket is the unit for access policy, encryption, versioning, lifecycle rules and billing.

Key. Up to 1,024 bytes of UTF-8. The namespace is flat: photos/u42/p9.jpg is one string, not a file in two folders. LIST prefix=photos/ delimiter=/ returns matching keys and rolls deeper ones into “common prefixes”, which is how consoles draw folders. There is no rename; moving a “folder” is a copy and a delete per object.

Object. The bytes plus three kinds of metadata:

• System metadata: size, Content-Type, ETag, last-modified time, storage class, checksums.
• User metadata: x-amz-meta-* headers, at most 2 KB and fixed at upload; changing it means copying the object.
• Tags: up to 10 key-value pairs, editable, usable in lifecycle rules and policies.

No partial updates. A PUT replaces the whole object: you cannot append a log line or patch byte 400 of a video. Reads can take any slice with a ranged GET (Range: bytes=0-1048575). The exception is S3 Express One Zone, whose directory buckets accept appends, up to 10,000 appends per object.

Versioning. Per bucket. Each PUT adds a version instead of overwriting, and a plain DELETE adds a delete marker: GET returns 404 while older versions stay readable by version ID, and billed. Object Lock adds write-once-read-many retention; in compliance mode nobody, root included, can delete a locked version early.

Guarantees, precisely

Strong read-after-write, per key

Since 1 December 2020, S3 is strongly consistent for every object in every region, at no extra cost. Once a PUT (new or overwrite) succeeds, any GET, HEAD or LIST that starts afterwards sees it; once a DELETE succeeds, GET returns 404 and LIST omits the key. Before that, overwrites and listings could briefly be stale, which is why Hadoop users ran S3Guard. Prep material calling S3 “eventually consistent” is out of date.

What it still does not give you:

• No multi-key atomicity. A data file and its manifest, or a photo and its thumbnail, are separate writes; a reader can see one without the other.
• Concurrent writers: last writer wins, by timestamp, and the loser is not told. Each PUT is atomic: readers see the old or the new object, never a mix.
• Bucket configuration is eventually consistent. After enabling versioning, wait 15 minutes before writing.

How S3 built this on a cached metadata layer belongs to designing an object store like S3; the general trade-off is in strong vs eventual consistency.

Conditional writes: compare-and-set on one key

Since 2024 a write can carry a precondition:

• If-None-Match: * (August 2024): create only if the key does not exist. Racing creators: the first to finish wins, the rest get 412 Precondition Failed.
• If-Match: <ETag> (November 2024): replace only if the object is still the version you read. This is optimistic locking on one object.

Both work on PutObject, CopyObject and CompleteMultipartUpload and can be enforced by bucket policy; a concurrent DELETE can surface as 409 Conflict, so retry. S3 becomes a small coordination store: exactly one job writes a _SUCCESS marker, or a table format commits manifest 124 only if nobody else did:

PUT /tables/orders/_commits/000124.json HTTP/1.1
Host: lake-prod.s3.ap-south-1.amazonaws.com
If-None-Match: *

HTTP/1.1 412 Precondition Failed      <- another writer committed 124 first

Durability and availability

S3 Standard is designed for 99.999999999% (eleven nines) annual durability across at least three availability zones. With 10 million objects, expected loss is 10⁷ × 10⁻¹¹ = 10⁻⁴ objects a year: one object every 10,000 years. That covers failing disks and zones, not your own DELETE.

Availability is lower and varies by class: Standard is designed for 99.99% with a 99.9% monthly SLA; Standard-IA 99.9% (SLA 99%); One Zone-IA 99.5%; Express One Zone 99.95%. Both One Zone classes lose data if their zone is destroyed. See availability and the nines.

What stays asynchronous

• Event notifications (to SQS, SNS, Lambda or EventBridge) are delivered at least once, usually within seconds but sometimes after a minute or more, and not in order. PUT and DELETE events carry a sequencer that orders events for one key only.
• Replication (same- or cross-region) is asynchronous. With Replication Time Control, 99.9% of objects arrive within 15 minutes; without it there is no bound. It is a disaster recovery tool with a non-zero RPO, not a synchronous copy.

How it scales

Request rate per prefix

S3 supports at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix, with no limit on prefixes. A prefix is any leading part of the key; S3 splits the sorted key space into partitions by traffic. Ten evenly loaded prefixes give 10 × 5,500 = 55,000 reads/s.

Scaling is gradual. When a key range suddenly takes more than its partition can serve, S3 returns 503 Slow Down while it splits the range; the SDKs retry with exponential backoff, and hand-written clients must too (backoff with jitter).

The trap is a key layout that aims all new traffic at one range. Date-first keys (logs/2026-10-27/...) sort a whole day’s writes together, and tomorrow the hot spot moves to a fresh range. A hash prefix in front spreads them. A log pipeline writing 9,000 objects/s needs 9,000 ÷ 3,500 ≈ 2.6, so at least 3 partitions’ worth; with a two-hex-digit prefix (00/ to ff/) and four partitions, each carries 2,250 writes/s. Better still, batch: fewer, larger objects cut requests and cost together.

Throughput, latency and the fast tier

• Parallelism, not faster requests. Treat S3 as a fleet, not a server: many connections let one EC2 instance reach 100 Gb/s. Download big objects as parallel ranged GETs aligned to the upload’s parts (GET ?partNumber=N).
• Latency. First bytes take roughly 100–200 ms from S3 Standard. AWS suggests retrying a small request still running after 2 seconds; the retry usually takes a faster path.
• Express One Zone gives single-digit-millisecond access and hundreds of thousands of requests per second in one zone you choose, at about 5× the storage price ($0.11 vs $0.023 per GB-month). Use it for scratch data next to compute, such as training shards, not as the system of record.
• A CDN serves hot reads from edges; Transfer Acceleration speeds long-distance uploads through them.

Uploads and downloads that skip your servers

Never stream user files through your API tier: a 2 GB video ties up a server for minutes. Clients talk to S3 directly; your servers only decide who may write what, where.

Presigned URLs

A presigned URL is one S3 request (method, bucket, key, expiry) signed with your service’s credentials. Whoever holds it can make that request until it expires; it is a bearer token:

• Expiry. Up to 7 days when signed with an IAM user’s keys under SigV4; with temporary credentials it dies when they do (about 6 hours for EC2 role credentials). Use minutes. Expiry is checked when a request starts, so a download in flight is not cut off.
• Reuse. It works any number of times until expiry, and a presigned PUT overwrites an existing key, so the server chooses the key (uploads/u42/<uuid>.jpg), never the client.
• Size. A presigned PUT does not cap size; a browser POST policy can (content-length-range), or check after upload.

This is the flow the Instagram and YouTube designs use:

Writing the row before the upload is the detail interviewers listen for: every object has an owner row, and anything stuck in pending is easy to clean up. Write it after, and a crash in between leaves bytes nothing points to. Many designs also let the phone call “done” for speed and keep the event as the backstop.

Multipart upload

Above about 100 MB, and always above 5 GB (the single-PUT limit), upload in parts:

1. CreateMultipartUpload returns an upload ID.
2. UploadPart sends parts numbered 1 to 10,000, each 5 MiB to 5 GiB (the last may be smaller), in any order and in parallel. Each returns an ETag; a failed part is retried alone.
3. CompleteMultipartUpload lists the part numbers and ETags; the object appears atomically. Until then nothing is visible at the key.

The maximum object size is 50 TB (48.8 TiB = 10,000 × 5 GiB), raised from 5 TB in December 2025. The part count sets a floor: part size ≥ object size ÷ 10,000, so a 200 GB file needs parts of at least 20 MB. To resume, presign each part; after a drop the client calls ListParts and resends only what is missing. The client side is in large file uploads.

Checksums. AWS SDKs send a checksum with each upload (CRC64NVME by default; CRC32C, SHA-256 and others are supported), and S3 recomputes it and rejects a mismatch. A multipart ETag is not the object’s MD5; to deduplicate by content, store your own SHA-256 (checksums).

Storage classes, lifecycle and the bill

Every object has a storage class; cheaper storage always costs slower or pricier reads and a minimum billed stay. Prices are us-east-1 list prices, October 2026.

Lifecycle rules, filtered by prefix, tag or size, move objects down this table on a schedule and expire them at the end. Three things to know:

• Minimum stays are billed. Delete a Standard-IA object on day 10 and you pay for 30 days.
• Small objects don’t tier well. IA classes bill at least 128 KB per object, lifecycle rules skip objects under 128 KB by default (since September 2024), and each archived object carries 40 KB of extra metadata. Archive bundles, not millions of thumbnails.
• Transitions only go down. Leaving Deep Archive means a restore and a copy. If access is unpredictable, Intelligent-Tiering moves objects for you, with no retrieval fees.

Requests and egress usually beat storage

At the same prices, PUTs cost $0.005 per 1,000, GETs $0.0004 per 1,000, and internet egress $0.09/GB for the first 10 TB a month, falling to $0.05/GB above 150 TB.

• Tiny objects. A pipeline writes 1 billion 4 KB events a day as separate objects. PUTs: 1,000,000,000 ÷ 1,000 × $0.005 = $5,000 a day, while storing that day’s 4 TB costs about 4,000 GB × $0.023 ≈ $92 a month. Batched into 64 MB files, a day is 4 TB ÷ 64 MB = 62,500 PUTs: $0.31.
• Hot images. A shopping app in a Flipkart-style sale month serves 1 billion product-image views of 200 KB straight from S3: 200 TB out. GETs cost 1,000,000 × $0.0004 = $400. Egress through the tiers is 10,000 × 0.09 + 40,000 × 0.085 + 100,000 × 0.07 + 50,000 × 0.05 ≈ $13,800, against about $2,250 to store 100 TB of images. Egress is six times the storage bill. Put a CDN in front: S3-to-CloudFront transfer is free, and the edge absorbs most requests.

Uploads and same-region transfers to other AWS services are free. More in cost-aware design.

Limits and numbers

From the S3 User Guide, October 2026.

When to pick it, and when not

Pick object storage when data is large, written once and read often, reached by clients or many services, or kept cheaply for years. It is also where other systems spill big payloads: Kafka messages and DynamoDB items carry a key that points into a bucket.

Pick something else when you update in place or append often (a database or log); you need millisecond reads of small values (Redis or a database); you query by attributes (keep them in PostgreSQL and store only the key); a few-kilobyte value must commit with its row; or an old application needs a POSIX file system.

The same idea on other clouds

• Google Cloud Storage: strongly consistent object writes, deletes and listings, even in multi-region buckets; access changes take about a minute. Compose joins up to 32 objects into one, for parallel uploads and appends.
• Azure Blob Storage: account → container → blob. Block blobs reach about 190.7 TiB, append blobs suit logs, page blobs back VM disks. Tiers: hot, cool (30-day minimum), cold (90) and archive (180, hours to rehydrate).
• Cloudflare R2: S3-compatible, with no egress fees and $0.015 per GB-month storage.
• Self-hosted: MinIO or Ceph’s RADOS Gateway speak the S3 API on your disks; you own durability and repair.

Common interview usage

The data lake key is date-first, yet fine: big, batched files mean a few hundred PUTs a second, and the date lets query engines skip whole partitions.

Going deeper: failure modes

Using LIST as a database

LIST returns at most 1,000 keys per call, in key order, filtered by prefix and nothing else. Finding “user 42’s photos from last week” among 100 million objects means 100,000 calls; at an assumed 50 ms each, one after another, that is 5,000 s, about 83 minutes. Keep the index in a database keyed by owner and time; for audits, use S3 Inventory reports rather than crawling.

Hot prefixes and hot objects

Beyond a burst on one key range, a single viral object read more than about 5,000 times a second can draw 503s too. Put a CDN or cache in front of hot reads, ramp traffic gradually, and watch 5xx errors per prefix.

Bytes stored, row missing (and the reverse)

S3 and your database cannot share a transaction, so every upload is a dual write. The safe order is the presigned flow’s: pending row first, then bytes, then an idempotent update, plus a sweeper for anything left pending. Deletion runs the other way: mark the row deleted, then remove the object asynchronously. The general patterns are the transactional outbox and idempotent consumers.

Bytes you forgot you were paying for

Incomplete uploads, old versions behind delete markers and replicas in another region bill silently; with versioning on, a DELETE frees nothing. Versioned buckets need lifecycle rules that expire noncurrent versions after N days and abort stale uploads. Millions of tiny objects are the other silent cost: chatty I/O with a price tag.

Public buckets and leaked URLs

Since April 2023, new buckets block public access and disable ACLs by default, and objects are encrypted with S3-managed keys. Keep it that way: public content through a CDN that alone can read the bucket, private content through short-lived presigned URLs, since a leaked one works for anyone until it expires. See encryption and key management.

Torn ranged reads

Parallel range GETs during an overwrite can stitch together halves of two objects; Google’s documentation warns about exactly this. Pin each range to one version with If-Match and the first response’s ETag, or a version ID. Better, never overwrite: write new versions under new keys and move the pointer in the database.

Late, duplicate and missing events

Notifications arrive at least once, out of order, sometimes late, and a bad filter or failing consumer makes them look lost. Make handlers idempotent, compare sequencer values per key, and reconcile against the database on a schedule. Beware recursion: a function that writes thumbnails into the prefix that triggers it calls itself forever.

Cheat sheet

How to say it in the interview

“Media goes to S3, never through our app servers. The API writes a pending row and returns a presigned PUT URL valid for 15 minutes; big videos use multipart with presigned part URLs, so a dropped connection costs one part. An at-least-once ObjectCreated event triggers the transcoder, which marks the row ready idempotently. Keys start with a hash to spread partitions, lookups go through the database rather than LIST, and reads go through a CDN because egress would dominate the bill. Lifecycle rules move originals to Standard-IA at 30 days and Glacier at 90.”

Is S3 eventually consistent? Not since December 2020: PUTs, overwrites, DELETEs and LISTs are strongly consistent per key. Events, replication and bucket configuration are still asynchronous, and there are no transactions across keys.

How do you upload a 10 GB video from a phone? Multipart, since a single PUT stops at 5 GB. With 16 MiB parts that is 10 GiB ÷ 16 MiB = 640 parts, each with a presigned URL, a few in flight. After a drop the app asks which parts arrived, sends the rest and calls Complete; a lifecycle rule aborts abandoned uploads.

Two clients write the same key at the same time. What happens? Last writer wins; the other write vanishes without an error. Avoid it with unique keys (a UUID or content hash), or detect it with If-None-Match: * to create once and If-Match to update only an unchanged object.

Your uploads start failing with 503 Slow Down. Why? One key range is above 3,500 writes a second while S3 splits it, usually because keys start with a date or a constant. Back off with jitter, put a hash at the front of the key, and batch small objects.

Why not store the images in PostgreSQL? Fine for a few kilobytes, but large blobs bloat the table, the WAL, backups and every replica. Object storage costs a fraction per GB, needs no capacity planning and sits behind a CDN; the database keeps the key and metadata.

Where to go next

• Design an object store like S3: how S3 itself works inside, from erasure coding to the metadata layer.
• Large file uploads: chunking, resuming and verifying uploads from the client’s side.
• Content delivery network: the layer in front of almost every public bucket.
• Design YouTube, Design Instagram and Design Dropbox / Google Drive: blob storage at the centre of a design.
• Checksums and idempotency: making uploads and event handlers safe to retry.
• Disaster recovery: replication, versioning and backups as an RPO and RTO decision.
• Cost-aware design: reading request, storage and egress costs before they surprise you.

References and further reading

1. Amazon S3 User Guide: What is Amazon S3? (with the data consistency model)docs.aws.amazon.comBucket types and namespaces, keys and version IDs, strong read-after-write for PUT, DELETE and LIST, atomic single-key updates, last writer wins, and eventually consistent bucket configuration.
2. Amazon S3 Update: Strong Read-After-Write Consistency (AWS News Blog, 1 December 2020)aws.amazon.comThe date and scope of the change (all objects, all Regions, no extra cost) and the S3Guard and EMRFS workarounds it retired.
3. Best practices design patterns: optimizing Amazon S3 performancedocs.aws.amazon.comAt least 3,500 write and 5,500 read requests per second per partitioned prefix, gradual scaling with 503 Slow Down, up to 100 Gb/s per instance and 100–200 ms small-object latency.
4. Amazon S3 multipart upload limitsdocs.aws.amazon.comUp to 10,000 parts of 5 MiB to 5 GiB, a 48.8 TiB (50 TB) maximum object size, and multipart from about 100 MB.
5. Download and upload objects with presigned URLsdocs.aws.amazon.comSeven-day maximum with IAM user SigV4 credentials, shorter lives with temporary credentials, reuse until expiry, overwrites, and the signatureAge policy key.
6. How to prevent object overwrites with conditional writesdocs.aws.amazon.comIf-None-Match and If-Match on PutObject, CopyObject and CompleteMultipartUpload, and the 412, 409 and 404 outcomes under concurrency.
7. Understanding and managing Amazon S3 storage classesdocs.aws.amazon.comDesigned durability and availability, Availability Zones, minimum storage durations and minimum billable sizes for every class.
8. Amazon S3 Event Notificationsdocs.aws.amazon.comAt-least-once delivery that usually takes seconds and sometimes a minute or more, destinations, and the recursive-trigger warning.
9. Amazon S3 pricingaws.amazon.comUS East (N. Virginia) list prices published in October 2026: storage per class, PUT and GET prices, Intelligent-Tiering monitoring, internet egress tiers and free transfer to CloudFront.
10. Google Cloud Storage: consistencycloud.google.comStrong global consistency for object writes, deletes and listings, eventually consistent access changes, and torn ranged reads without a generation precondition.