Large file uploads

Large file uploads is the playbook for getting big files from a user’s device into your storage: a creator’s 4 GB wedding film, a hospital’s 2 GB MRI study, a 30 GB laptop backup. It has two halves. Moving the bytes: straight from the client to object storage, in parts, so a dropped connection costs seconds, not the whole file. Trusting the bytes: once the last part lands, you verify, scan and process the file before anyone else sees it. Weak answers stop at “use a pre-signed URL”.

Picture it first: moving house through a godown

You are moving from Kolkata to Bengaluru with 80 cartons, and your new housing society keeps incoming goods in its godown until the flat is ready.

1. Hand every carton to the society secretary, who carries them over himself. He does nothing else all day, and if he trips, he starts again.
2. He writes a gate pass instead: your truck, bay 14, until 6 pm. The truck drives straight to the godown.
3. Number the cartons and send them on three tempos. One breaks down, so you resend only its cartons.
4. The godown keeps a register of cartons received. Back from a two-day break, you send only what is missing.
5. Every carton carries a seal, checked at the gate. A carton identical to one already stored doesn’t travel again.
6. Nothing goes straight to your flat. It waits in an inspection bay, and you get an SMS when it is shelved.
7. A cousin in Delhi drops cartons at the nearest depot, and every family gets a fixed amount of space.

Map it back: the secretary is uploading through your app server; the gate pass is a pre-signed URL; tempos are multipart; the register is resumable upload; seals and duplicates are checksums and dedupe; the inspection bay is quarantine and scanning; the depot and space limit are edge entry and quotas.

Spotting the pattern

Listen for these cues:

• Files from tens of megabytes to many gigabytes: videos, raw photos, design files, backups, datasets, medical scans.
• Mobile or flaky networks: 4G on a moving train, a district hospital’s shared link, home broadband that drops.
• The upload must survive interruption: a dropped link, a closed laptop lid, a phone that kills a backgrounded app.
• You don’t want the bytes on your app servers, which are sized for small JSON requests.
• Work happens after the upload: a virus scan, transcoding, thumbnails, text extraction.

It does not apply when:

• Files are small. A 200 KB profile photo goes in one PUT, ideally still straight to storage, with no parts and no resume.
• The data is a stream: logs, metrics and live video go to a log or a media server, not an upload API.
• The uploader is your own server next to the bucket. The cloud SDK’s transfer manager already does multipart.

The running example is a creator platform for teachers and vloggers. A creator in Indore uploads a 4 GB 4K video over a 20 Mbit/s home connection that drops about every 10 minutes. One clean pass takes 4 GB × 8 = 32 Gbit ÷ 20 Mbit/s = 1,600 s ≈ 27 minutes, so the upload must survive at least two drops. The platform takes 50,000 uploads a day averaging 1 GB: 50 TB ÷ 86,400 s ≈ 579 MB/s ≈ 4.6 Gbit/s on average, about 14 Gbit/s at a 3× evening peak. A 1 GB upload at 20 Mbit/s takes 8,000 Mbit ÷ 20 = 400 s, so by Little’s law 0.58 a second × 400 s ≈ 230 uploads are in flight, about 700 at peak.

Level 0: post it through the app server

The browser sends a multipart/form-data body to POST /videos, and the app server writes it to object storage. Fine for a 200 KB avatar. For the 4 GB video it fails five ways:

• The body must live somewhere: 4 GB of RAM per upload, or a spool to local disk and a second copy. Many proxies, gateways and frameworks cap request bodies at a few megabytes by default.
• A server is held for 27 minutes. At peak that is about 700 long uploads, and 14 Gbit/s in plus 14 Gbit/s back out to storage. The app tier is now sized for bandwidth.
• Timeouts. Any request limit on the path, such as API Gateway’s 29 seconds (long-running jobs), ends the upload.
• Every drop restarts at byte 0. Each try gets 600 s × 20 Mbit/s = 12,000 Mbit = 1.5 GB before the link drops, so the file never arrives.
• Deploys kill uploads: a 27-minute request doesn’t drain.

You gain one simple endpoint. You pay with app servers that carry every byte twice. It breaks at the first file that needs longer than the link stays up: 27 minutes needed, 10 available. Say: “Bytes never pass through my API servers; the API only hands out permission.”

Level 1: a pre-signed URL, straight to storage

Split permission from transfer. The client asks the API to start an upload; the API authenticates the user, checks the declared size against quota and limits, chooses the storage key, records an upload row and returns a pre-signed URL: a storage URL signed with the app’s credentials that allows one operation (PUT) on one key until it expires. The client sends the bytes straight to S3, Google Cloud Storage or Azure Blob Storage (where it is called a SAS token).

POST /v1/uploads HTTP/1.1
Authorization: Bearer …
Content-Type: application/json

{"filename": "wedding.mp4", "size": 4000000000, "content_type": "video/mp4"}

HTTP/1.1 201 Created
Content-Type: application/json

{"upload_id": "up_81",
 "put_url": "https://raw-uploads.s3.ap-south-1.amazonaws.com/raw/t42/up_81?X-Amz-Expires=900&X-Amz-Signature=…",
 "required_headers": {"Content-Type": "video/mp4"}}

The API now handles a couple of kilobytes per upload. Lock the URL down; this is where most answers are thin:

• Short expiry. S3 allows SigV4 URLs of up to 7 days, but a pre-signed URL is a bearer token. Fifteen minutes is plenty: S3 checks expiry when a request starts, so a 27-minute PUT begun at minute 1 still finishes.
• One key, chosen by the server, such as raw/t42/up_81. Never build it from the client’s file name: S3 replaces an existing object with the same key, and ../../avatar.png is an invitation.
• Pin what you can. Sign the content type and a checksum header into the URL. For browser forms, a pre-signed POST policy can cap the size with content-length-range and pin the key prefix with starts-with.
• A private, encrypted bucket. Block public access; S3 encrypts new objects with SSE-S3 by default, and SSE-KMS gives per-tenant keys.
• CORS. A browser can’t PUT to the storage domain unless the bucket’s CORS rules allow your origin.

You gain an app tier that never touches the bytes. You pay for a signing endpoint, CORS rules and upload rows that may never complete. It breaks because it is still one request: a drop at minute 10 throws away 1.5 GB, one PUT is capped at 5 GB on S3, and one TCP stream may not fill a fast, distant link. Say: “The API checks auth and quota and returns a 15-minute pre-signed PUT for a key it chooses in a private bucket.”

Level 2: multipart, in parallel

Cut the file into parts and upload each as its own request. On S3 that is three calls:

1. Create. The API calls CreateMultipartUpload and stores the upload ID on the upload row.
2. Upload parts. The API pre-signs UploadPart URLs, each bound to the upload ID and one part number. The client PUTs parts in any order, a few at a time, recording each part’s ETag.
3. Complete. The API calls CompleteMultipartUpload with the part numbers and ETags, and S3 stitches the parts into one object. Until then nothing is visible.

S3’s limits shape the client: parts of 5 MiB to 5 GiB (the last may be smaller), at most 10,000 parts, so the largest object is 10,000 × 5 GiB = 50,000 GiB ≈ 48.8 TiB. AWS suggests multipart from about 100 MB.

Choosing a part size trades lost work against request count. For the 4 GB video, 16 MiB parts give 4,000 MB ÷ 16.8 MB ≈ 239 parts; with three in flight, a drop loses at most 3 × 16 MiB = 48 MiB, about 20 seconds of upload, instead of 1.5 GB. The 10,000-part cap sets a floor of file size ÷ 10,000: a 200 GB camera file needs at least 20 MB, so use 32 MiB (about 5,960 parts), and a 48.8 TiB object needs 48.8 TiB ÷ 10,000 ≈ 5 GiB parts, the maximum.

Parallelism helps when one connection can’t fill the pipe: a TCP stream carries at most its window per round trip, so with, say, 4 MB in flight and 150 ms to a distant region, one stream tops out near 4 MB × 8 ÷ 0.15 s ≈ 213 Mbit/s. On a 20 Mbit/s line, two or three parts in flight are plenty.

Retry only the failed part, with backoff and jitter (timeouts, retries and backoff); re-sending a part number replaces it, so retries are idempotent. Browsers also need the bucket’s CORS rules to expose the ETag header.

You gain files up to 48.8 TiB, parallel throughput, and drops that cost one part. You pay for part bookkeeping, the complete call, URL refreshes and an abort rule. It breaks when the app dies: a reloaded tab or a killed phone app forgets the upload ID and the finished parts, so the user starts again and the old parts rot. Say: “16 MiB parts, three in flight, each retried alone; complete stitches them, and a lifecycle rule aborts anything unfinished after a week.”

Level 3: resumable uploads

Resumable means the upload outlives the process that started it. The server knows how much it has; the client knows which upload it was doing, because after every part it saves the upload ID, a file fingerprint (path, size, modified time) and the finished parts or offset, in IndexedDB or SQLite. On restart it asks the server and carries on.

Picture a district hospital sending a 2 GB MRI study over 4G with a 5 Mbit/s uplink: 2 GB × 8 = 16 Gbit ÷ 5 Mbit/s = 3,200 s ≈ 53 minutes, while the tablet sleeps and the signal comes and goes. Three common ways to build it:

On S3 multipart, resuming is “tell me what you have”. Your API keeps a table of parts and ETags (or calls ListParts, 1,000 parts per page) and returns the missing part numbers with fresh URLs.

The tus protocol (version 1.0) is an open protocol for resumable uploads over HTTP, with clients for browsers and phones. The hospital’s app creates an upload, loses the link after 700 MiB, asks for the offset and continues:

POST /files HTTP/1.1
Tus-Resumable: 1.0.0
Upload-Length: 2000000000
Upload-Metadata: filename bXJpLXN0dWR5LnppcA==

HTTP/1.1 201 Created
Location: https://upload.example.com/files/up_92
Tus-Resumable: 1.0.0

HEAD /files/up_92 HTTP/1.1
Tus-Resumable: 1.0.0

HTTP/1.1 200 OK
Upload-Offset: 734003200
Upload-Length: 2000000000
Cache-Control: no-store

PATCH /files/up_92 HTTP/1.1
Tus-Resumable: 1.0.0
Upload-Offset: 734003200
Content-Type: application/offset+octet-stream
Content-Length: 1265996800

HTTP/1.1 204 No Content
Upload-Offset: 2000000000

A PATCH must carry the server’s current offset or it gets 409 Conflict, and Tus-Resumable goes on every request except OPTIONS. Extensions add an Upload-Expires deadline, per-request checksums, cancellation and concatenation of partial uploads for parallelism. Resume from the offset the server reports, not from what you think you sent.

Google Cloud Storage resumable uploads have the same shape: a start request returns a session URI, the client PUTs data to it and, after an error, asks how much was persisted. A session lasts one week and chunks must be multiples of 256 KiB. The session URI is itself the credential, it is pinned to the region that created it, and a browser must send Origin on the start request or later responses lack CORS headers.

The IETF draft “Resumable Uploads for HTTP” (draft-ietf-httpbis-resumable-upload, revision 14 in October 2026) grew out of tus and builds resume into HTTP: a 104 (Upload Resumption Supported) interim response announces an upload resource, HEAD reads Upload-Offset, PATCH appends, and Upload-Complete marks the end. It is still a draft, and deliberately sequential.

You gain uploads that survive drops, sleep, killed apps and reboots, resending at most the unacknowledged bytes. You pay for client-side state, a resume endpoint, expiring sessions and edge cases such as a file edited mid-upload. It breaks when you need to trust the bytes: nothing yet proves that what arrived is what left the device, and uploading the same 4 GB twice costs twice. Say: “The client saves the upload ID and finished parts after every part; on restart it asks the server what it has and sends only the rest.”

Level 4: integrity and dedupe

Bits flip in memory and proxies, and a file can change on disk during a long upload. Check at three grains.

Per part. Send a checksum with every part and let storage reject a mismatch. S3 accepts CRC64NVME (its default), CRC32, CRC32C, SHA-1, SHA-256 and others, recomputes the value and refuses a part that differs; tus answers a bad Upload-Checksum with 460 Checksum Mismatch. A bad part costs one 16 MiB retry.

Per file. For multipart, S3 keeps a full-object checksum (CRC64NVME, CRC32 or CRC32C, which combine across parts) or a composite checksum of the part checksums, the only option for SHA-256. Also compute a whole-file SHA-256 on the device, send it with the create call, and have the post-upload worker verify it.

Dedupe. With a content hash, the create call can say “we already have this”, and a creator’s repeat upload is linked in a fraction of a second. Go finer with content-addressed chunks: Dropbox splits files into 4 MB blocks named by their SHA-256 and uploads only the blocks the server lacks, so resume comes free and an edit sends only changed blocks.

Two cautions. The server must recompute every hash, or one bad client poisons a block for everyone who shares it. And dedupe across users leaks: if claiming a hash skips the upload, anyone can test whether a file exists on your service. Dedupe within a user or tenant.

You gain corruption caught per part, proof that the stored file is the sent file, and free repeat uploads. You pay for hashing on the device, a hash index and careful dedupe rules. It breaks when the file goes live unchecked: an executable renamed lesson.mp4, a PDF carrying malware, or a creator staring at “uploaded” with no idea what happens next. Say: “Each part carries a CRC32C that S3 checks, a worker verifies the whole-file SHA-256, and repeats within a user are deduped by hash.”

Level 5: after the upload

The last byte arriving is where the real work starts. Nothing a user sends is visible to others until it has been checked.

1. Land in quarantine: a private raw/ bucket that nothing serves from.
2. React to an event. An S3 event notification on object creation goes to a queue. Delivery is at-least-once, usually within seconds but sometimes a minute or more, so consumers must be idempotent.
3. Validate. Compare size and hash with what was declared. Find the real type from the bytes: a PDF starts with %PDF-, an MP4 has an ftyp box near the start, and a decoder such as ffprobe must open it. The client’s Content-Type and extension are claims, not facts.
4. Scan with an antivirus engine such as ClamAV, sandboxed with time and size limits so a zip bomb can’t sink the worker.
5. Promote or reject. Clean files are copied to the media/ bucket and the raw copy deleted; rejects are deleted and the user is told why. A separate bucket also avoids the loop S3 warns about, where a handler writing into its own trigger bucket triggers itself.
6. Process. Transcoding and thumbnails are long-running jobs, queued from here.
7. Tell the client. The client polls GET /v1/uploads/up_81 or listens over Server-Sent Events (real-time updates).

Every transition is a conditional write, such as UPDATE uploads SET state = 'scanning' WHERE id = 'up_81' AND state = 'uploaded', so a duplicate event changes nothing (idempotency). A sweeper expires rows stuck in uploading.

You gain nothing unsafe ever served, a visible status, and processing decoupled from the upload. You pay for two buckets, an event queue, scanners, a state machine and a sweeper. It breaks at global scale: distant creators upload slowly, one account can fill your bucket, and fees grow the bill. Say: “Uploads land in a private quarantine bucket; an S3 event starts magic-byte checks and a malware scan; only clean files are promoted and processed.”

Level 6: global scale, speed and cost

Three problems remain at this scale.

Distance. A creator in Toronto uploading to Mumbai fights a long, lossy round trip. Either enter the provider’s network nearby: S3 Transfer Acceleration accepts uploads at the nearest CloudFront edge location and carries them over an optimised path to the bucket, via a separate s3-accelerate endpoint and for an extra per-GB charge (a CDN edge, used inbound). Or upload to the nearest region; Google Cloud Storage sessions stay in the region that created them, and medical scans may have to stay in the country anyway.

Abuse and fairness. A signed URL is permission to spend your money. At create, check per-user and per-tenant quotas and reserve the declared size; cap open sessions per user and rate-limit the endpoint; sign each part URL for one part and 15 minutes; and at complete, reject a total larger than declared. Per-tenant limits are the usual noisy-neighbour problem.

Client manners. Fewer parallel parts on mobile data, a “Wi-Fi only” option, a bandwidth cap, and jittered backoff so a regional blip doesn’t become a retry storm.

Cost. Each upload now costs about 6 API calls (create, complete, a few batches of part URLs), so 50,000 × 6 = 300,000 a day ≈ 3.5 requests a second, against 14 Gbit/s at Level 0. What remains is storage: abort incomplete uploads, delete raw copies after promotion, move unwatched originals to a colder class, and watch per-GB acceleration and cross-region fees (cost-aware design).

You gain fast uploads from anywhere, limits that stop one account hurting others, and a bill that tracks real files. You pay for acceleration fees or extra regions, quota bookkeeping and a smarter client. Say: “Distant creators use Transfer Acceleration or a nearer region; create reserves quota and caps open sessions; part URLs are per part and short-lived; complete re-checks the size.”

Comparing the rungs

Which classic problems use it

Common mistakes

• Uploads through the API servers, then scaling the app tier for bandwidth.
• Long-lived or loose pre-signed URLs: a 7-day URL, or one that lets the client pick the key.
• Trusting the client: its file name as the key, its Content-Type as the type, its hash unchecked.
• One giant PUT, so a drop at 90% means starting again and 5 GB is a wall.
• Upload state only in memory, so a reloaded tab restarts from zero.
• No AbortIncompleteMultipartUpload rule: terabytes of invisible, billed parts.
• Using the multipart ETag as a content hash.
• Serving files before scanning them.

Cheat sheet

• Upload time = size × 8 ÷ uplink: 4 GB at 20 Mbit/s ≈ 27 minutes.
• Lost per drop ≈ parts in flight × part size, or only the unacknowledged tail when resumable.
• S3 part size ≥ file size ÷ 10,000, within 5 MiB to 5 GiB; largest object ≈ 48.8 TiB.
• Uploads in flight = uploads per second × average upload time (Little’s law).
• Orphaned parts ≈ abandoned bytes a day × days before the abort rule.

How to say it in the interview

“Creators upload videos of tens of gigabytes over home and mobile links, so the bytes never touch my API. Create checks auth and quota, picks a key in a private quarantine bucket, starts an S3 multipart upload and returns 15-minute pre-signed URLs per part. The client sends 16 MiB parts three at a time with a CRC32C each, retries only failed parts, and saves the upload ID and finished parts locally, so after a drop or a restart it asks what is missing and continues. After complete, an S3 event drives magic-byte checks, a malware scan, promotion and transcoding while the client watches the upload’s state. A lifecycle rule aborts unfinished uploads after 7 days.”

Why not stream the file through the API? A 4 GB body holds a server for 27 minutes, needs buffering, hits body limits and timeouts, and at peak pushes 14 Gbit/s through the app tier. With pre-signed URLs the API sees about 3.5 small requests a second.

The connection drops at 90%. What happens? At most the parts in flight are lost, 48 MiB. The client asks which parts the server has (or the tus offset) and sends the rest, with fresh URLs if the old ones expired.

How do you stop someone uploading 50 GB of junk with your URL? Each URL covers one key I chose, one part and 15 minutes; quota is reserved at create; complete rejects a total larger than declared; and lifecycle aborts unfinished parts.

How do you know it isn’t malware or a fake MP4? It stays in a private quarantine bucket until a worker checks the magic bytes, opens it with a decoder, verifies the hash and scans it. Only then is it copied to the bucket that gets served.

How do you pick the part size? At least 5 MiB and file size ÷ 10,000, but small enough that a drop loses little: 16 MiB for a 4 GB video, 32 MiB for 200 GB, 5 GiB at the 48.8 TiB maximum.

Where to go next

• Blob storage (S3): buckets, keys, storage classes and lifecycle rules.
• Design an object store like S3: multipart and checksums inside the store.
• Design Dropbox / Google Drive: content-addressed blocks and sync.
• Design YouTube: resumable uploads feeding a transcoding pipeline.
• Long-running jobs: the processing that starts when the upload ends.
• Real-time updates: pushing status to the client.
• Multi-step workflows: when upload, review and publish become one process.
• Checksums and idempotency: safe, verified retries.
• HTTP and HTTPS: the methods, headers and status codes underneath.

References and further reading

1. Amazon S3 multipart upload limitsdocs.aws.amazon.comUp to 10,000 parts of 5 MiB to 5 GiB (the last part may be smaller), a maximum object size of 48.8 TiB, multipart suggested from about 100 MB, and 1,000 parts per ListParts page.
2. Amazon S3 User Guide: Uploading objectsdocs.aws.amazon.comA single PUT holds up to 5 GB, multipart covers objects up to about 50 TB, and new objects are encrypted with SSE-S3 by default.
3. Amazon S3 User Guide: Download and upload objects with presigned URLsdocs.aws.amazon.comPresigned URLs as bearer tokens, expiry of up to 7 days with SigV4 from the SDKs, expiry checked when a request starts, and checksum headers on SigV4 uploads.
4. Amazon S3 User Guide: Checking object integrity for data uploadsdocs.aws.amazon.comSupported checksum algorithms, full-object versus composite checksums for multipart uploads, and why a multipart ETag is an MD5 of the part MD5s plus the part count.
5. Amazon S3 User Guide: Lifecycle rule to delete incomplete multipart uploadsdocs.aws.amazon.comThe AbortIncompleteMultipartUpload action with DaysAfterInitiation, recommended so you stop paying for parts of uploads that never finish.
6. Amazon S3 Event Notificationsdocs.aws.amazon.comAt-least-once delivery, usually within seconds but sometimes a minute or longer, to SQS, SNS, Lambda or EventBridge, and the loop that appears when a handler writes into the bucket that triggered it.
7. Amazon S3 Transfer Accelerationdocs.aws.amazon.comUploads enter at the nearest CloudFront edge location and travel an optimised path to the bucket, through a separate s3-accelerate endpoint and with extra transfer charges.
8. tus resumable upload protocol 1.0tus.ioCreation with POST and Upload-Length, HEAD for Upload-Offset, PATCH with application/offset+octet-stream, 409 on an offset mismatch, the Tus-Resumable header, and the checksum, expiration and concatenation extensions.
9. Google Cloud Storage: Resumable uploadscloud.google.comSession URIs that act as credentials and last one week, chunks in multiples of 256 KiB, sessions pinned to the region that created them, and the Origin header needed for browser CORS.
10. IETF draft-ietf-httpbis-resumable-upload: Resumable Uploads for HTTPdatatracker.ietf.orgThe HTTP working group’s standards-track draft (revision 14, October 2026): upload resources, 104 Upload Resumption Supported, Upload-Offset and Upload-Complete, and no parallel transfer.