Resumable & chunked uploads

A podcast hosting platform takes episodes from creators at home. Ade records a video episode of 4 GB and uploads it over a connection that sends 20 megabits a second, 2.5 MB a second. Without interruptions that takes 4,000,000,000 ÷ 2,500,000 = 1,600 seconds, about 27 minutes. But the connection drops now and then: the router restarts, the laptop sleeps, the phone hotspot switches cells.

Suppose drops arrive at random, on average once every 15 minutes. The chance of 27 minutes with no drop is e^(−1,600 ÷ 900) ≈ 17%. With a plain single-request upload, every drop throws away everything sent so far, and each retry faces the same odds. Making the upload resumable changes the shape of the curve:

Restarting grows exponentially with the transfer time: past some size, the expected time is hours and many attempts never finish. Resuming grows linearly, because a drop costs only the bytes since the last point the server confirmed. That's the whole idea; the rest of this page is the protocol, the sizing and the cleanup.

A resumable upload first creates an upload resource on the server, with its own URL. The client sends bytes to it; after a failure, the client asks the upload how many bytes arrived, and sends the rest from that offset. The open tus protocol is the clearest example:

The protocols in use differ in detail but share that shape:

The confirmed offset is a promise: it's a guarantee that those bytes won't need sending again, so the client can free them. A session URL is also a credential. Anyone holding the session URI can upload data to the bucket without further authentication, so treat it like the signed URLs in Object & blob storage.

A resumable upload can still send the file as one long request and resume wherever it broke. Splitting it into chunks or parts helps when requests have a time or size limit, when the total size isn't known at the start, or when parts should travel in parallel. In Cloud Storage a single chunk is usually best, since fewer requests means better performance; chunks, when used, must be multiples of 256 KiB, and at least 8 MiB is a sensible size.

S3's multipart upload goes further: the file is split into numbered parts that upload independently, in any order and in parallel, and a failed part is resent alone. When every part is in, a final request lists the parts and S3 assembles the object. On S3, multipart pays off from about 100 MB. The limits shape the part size:

Bigger parts mean fewer requests and less overhead; smaller parts mean less resent after a failure and less memory held per part. Parallel parts help when a single connection doesn't use the whole link, and gain nothing when the uplink itself is the limit: they just share it. The podcast app uploads up to three parts at a time and picks the part size from the file size.

Big files should bypass the application servers: the app creates the upload and hands the client a short-lived URL that goes straight to storage (Object & blob storage). The application still needs to know where each upload stands, so it keeps a record of it, separate from the bytes:

The record moves through clear states, each set by one actor:

• Initiated: the API created the upload, with the expected size and type.
• Uploading: parts or bytes are arriving; the confirmed offset or part list lives in storage, not in the client's memory.
• Uploaded: the client called complete, and storage assembled the object.
• Verified: the server checked size, checksum and real content type.
• Processed: transcoding and publishing are done (Background jobs & scheduling).

The resume handle, the upload id or session URL, must survive the browser tab. The podcast app stores it locally with the file's name, size and modification time, so that after a crash or a closed laptop it can offer "Resume upload of episode-42.mp4?" and check it's the same file before continuing (Offline-first & optimistic UI).

Resuming creates failure modes a single upload doesn't have: a chunk sent from the wrong position, a part from an older version of the file, a client that resumes after the user edited the file. Checksums catch them, at two levels:

• Per part: the client sends a checksum with each chunk or part, and the server rejects data that doesn't match. tus has an optional checksum extension for each PATCH; S3 accepts a checksum with each part.
• Whole file: a checksum of the complete file, checked after assembly, proves the parts add up to what the client has.

S3 offers two kinds for multipart uploads. A composite checksum is computed from the parts' checksums, so it depends on how the file was split. A full object checksum covers the whole file whatever the split, using CRC-based algorithms (CRC-64/NVME, CRC-32 or CRC-32C). AWS's SDKs calculate a checksum automatically when the caller doesn't supply one, and S3 fails a request whose data doesn't match it with a BadDigest error.

For the podcast app, the browser computes the full-file checksum as it reads the file, sends it with the completion request, and the upload only moves to verified if storage agrees.

Then check the content itself. A file named .mp4 can be anything; the server reads its real type and, for media, probes that it decodes, before it reaches the transcoding queue. The size declared at the start bounds the upload, so a client can't keep appending past what it asked for.

Many uploads are never finished: the user gives up, picks another file, or never comes back. Each one leaves stored bytes behind, and in S3 the parts of an incomplete multipart upload are billed until the upload is completed or stopped. They don't appear as objects in the bucket, so they are easy to forget.

The window is a trade-off: long enough that Ade can resume after a weekend away, short enough that abandoned bytes don't pile up. Per-user limits on concurrent and total pending uploads stop one account from filling storage with uploads it never finishes.

For the podcast platform, and any product that accepts large files from users' own connections: