Media Pipelines on S3: Presigned Uploads, Lifecycle Policies, and Cost Controls at Scale

Eliminate server upload bottlenecks with presigned URLs, enable massive file uploads via multipart, and automate storage archival for significant ongoing savings.

Building a media-heavy application means handling large files, video, images, documents, efficiently without bottlenecking your application servers. Whether you're processing video, managing user-generated media, or handling document storage, the traditional server-proxied approach creates multiple points of failure: memory pressure from concurrent large uploads, timeouts on big files during peak usage, no resume capability for failed uploads, rising infrastructure costs from vertical scaling, and unreliable uploads for clients with intermittent connectivity.

After migrating to direct S3 uploads with presigned URLs and intelligent lifecycle policies, we dramatically cut upload infrastructure costs, enabled seamless multi-GB uploads with resume capability via multipart, achieved noticeably faster upload speeds by removing the server proxy hop, implemented automatic archival so recent content stays hot while older content moves to cold storage, and gave clients upload mechanisms that handle network interruptions gracefully.

The problem with server-proxied uploads

A typical controller proxies uploads through the server, loading the file into memory as the bottleneck:

def upload_files(self):
    files = request.files.getlist('files')
    # File loaded into server memory - BOTTLENECK
    result = self.media_service.upload_media_files(files)

That creates memory pressure (large files mean large server RAM use and OOM crashes under concurrent uploads), a network double hop (client to server to S3, doubling bandwidth cost and latency), upload limits from framework request-size caps, timeout risk on long uploads exceeding gateway limits, and a vertical scaling trap, scaling servers for I/O rather than compute at real cost. Traditional proxied uploads also carry server egress charges, oversized compute for I/O handling, and load balancer processing fees, all of which direct uploads largely eliminate.

Presigned URLs for direct upload

Instead of client sending the file to the server which holds it in memory and forwards it to S3 (two hops, full I/O load, 2x bandwidth cost), the client requests a small presigned URL from the server, then uploads directly to S3 itself (one hop, minimal server load, 1x bandwidth cost).

Generating a presigned POST URL means checking the user's quota, building an S3 key with a UUID and date-based prefix, setting security constraints like content-length range and content-type, adding server-side encryption, and calling generate_presigned_post with an expiry, typically an hour. Filenames get sanitized against path traversal before being embedded in the key. The Flask endpoint validates the request, generates the presigned URL, and returns it along with the fields the client needs to include in its form-data POST.

On the client side, the flow is: fetch the presigned URL and fields from your server, build a FormData object with the fields first and the file last, and POST that directly to S3. The file never touches your server's memory.

Multipart upload for large files

Files under 100MB work fine with a single presigned POST. Between 100MB and 5GB, multipart is recommended for reliability, and above 5GB it's required since that's S3's per-PUT limit. AWS enforces a minimum part size of 5MB, a maximum of 5GB, up to 10,000 parts per upload, and a 5TB max object size.

Multipart gives you resumability (failed parts retry individually), parallel uploads (multiple parts upload at once), and better reliability overall (a network blip doesn't fail the whole transfer). The flow: initiate a multipart upload session server-side and get an upload ID and file key, request presigned URLs for a batch of part numbers (capped per-batch to avoid huge responses), upload each part directly to S3 with the client tracking ETags, complete the multipart upload by submitting the part list with their ETags, and abort cleanly if anything fails partway through.

On the client, calculate an optimal part size that keeps you under the 10,000-part ceiling, upload parts in controlled batches with limited concurrency, retry individual part uploads with exponential backoff on failure, track progress as parts complete, and finalize by submitting the sorted part list. A smart upload function can simply route files above a size threshold to the multipart path and everything else to the simple presigned POST path.

Lifecycle policies for cost optimization

Storage class economics differ a lot by tier. Standard costs the most but offers free, instant retrieval, good for active recent media. Standard-IA costs less with a small retrieval fee, good for warm, occasionally accessed content. Glacier Instant Retrieval saves significantly with a higher retrieval fee but still instant access, good for cold content needing instant access. Deep Archive is cheapest but has a retrieval fee and delay, good for compliance data rarely accessed.

Lifecycle policies shift older content into cheaper tiers automatically. A typical rule set transitions objects older than 30 days to Standard-IA, 90 days to Glacier Instant Retrieval, and 365 days to Deep Archive, filtered by prefix and a minimum object size (since IA and Glacier have their own minimum billable size). A separate rule handles noncurrent object versions the same way, and a critical third rule aborts incomplete multipart uploads after 7 days, since those consume storage silently, aren't visible in normal listings, and cost money if left unchecked.

Production deployment essentials

Enable bucket versioning and default server-side encryption at the bucket level. Configure CORS to allow your domain's origin with the GET, POST, and PUT methods and the headers your upload flow needs, including exposing the ETag header so multipart uploads can read it back. Use a least-privilege IAM policy scoping PutObject, PutObjectAcl, AbortMultipartUpload, and ListMultipartUploadParts to your media prefix specifically, with ListBucketMultipartUploads scoped at the bucket level. Wire up an S3 event notification to trigger post-processing, like a Lambda function transcoding video on ObjectCreated events filtered by prefix and suffix. And set a CloudWatch billing alarm on estimated S3 charges so cost surprises get caught early.

Summary

Presigned URLs eliminate server bottlenecks and cut infrastructure costs substantially. Multipart uploads enable reliable large-file transfers with resume capability, required for anything over 5GB. Lifecycle policies reduce storage costs progressively as content ages. Aborting incomplete uploads prevents silent storage leaks. Authentication and rate limiting are critical for any production deployment of these endpoints. And batch URL generation for multipart uploads avoids memory issues when handling very large files.

Implementation is a fairly clean sequence: replace proxied uploads with presigned URL generation, add authentication and rate limiting to every endpoint, implement multipart upload with batched URL generation for large files, configure lifecycle policies once, set up least-privilege IAM, wire up event notifications for post-processing, and enable cost alarms. The result is a media pipeline that scales from a handful of uploads a day to petabyte-scale storage without the server ever becoming the bottleneck.