How to Upload Large Files to AWS S3: The Definitive Guide

Understanding AWS S3 Upload Limits and Challenges

For developers and system architects, performing a file upload in AWS S3 is a fundamental task, but it comes with specific architectural constraints. Amazon Simple Storage Service (AWS S3) is the industry standard for object storage, yet standard single PUT requests are limited to a maximum upload size of 5 GB. Uploading files larger than this, or managing large datasets over unstable networks, requires more sophisticated handling to avoid timeouts and data corruption.

When an upload fails at 99%, restarting the entire process is inefficient. To ensure high availability and performance, AWS provides mechanisms for Multipart Uploads, which split data into manageable, resumable segments. In this guide, we will explore how to master the file upload in AWS S3 process using the AWS CLI, AWS SDK v3, and browser-based strategies, ensuring your architecture remains resilient and cost-effective.

What is AWS S3 Multipart Upload?

AWS S3 Multipart Upload is an API feature that allows users to upload a single object as a set of smaller parts. By breaking a large file into individual chunks, the process gains significantly improved reliability and performance.

• Improved Throughput: Parts are uploaded in parallel, maximizing network bandwidth.
• Resiliency: If a network failure occurs, you only need to re-upload the affected chunk rather than the entire file.
• Scalability: This method supports objects ranging from 5 MB up to 5 TB in total size.
• Pause and Resume: Uploads can be paused or resumed based on system resource availability.

Method 1: Uploading Large Files with AWS CLI

The AWS Command Line Interface (CLI) is the most accessible method for a large file upload in AWS S3. It automatically manages multipart logic, splitting files larger than the default threshold of 8 MB into smaller segments and executing parallel transfers.

Step 1: Install and Configure the AWS CLI

Verify your installation by running the command below. If not installed, you can obtain it via the official AWS documentation.

aws --version

Configure your environment by executing the following command and providing your credentials:

aws configure

Step 2: Execute the Upload

Use the standard aws s3 cp command. The CLI detects the file size and invokes multipart upload automatically.

aws s3 cp /path/to/local/large-file.mp4 s3://your-bucket-name/large-file.mp4

Step 3: Tuning Performance Settings

To optimize for high-bandwidth networks, adjust the CLI configuration parameters to increase throughput:

# Configure multipart threshold to 64 MB
aws configure set default.s3.multipart_threshold 64MB

# Set individual part size to 16 MB
aws configure set default.s3.multipart_chunksize 16MB

# Increase concurrent request limit to speed up transfers
aws configure set default.s3.max_concurrent_requests 20

Method 2: Programmatic Uploads via Node.js (AWS SDK v3)

Developers implementing a file upload in AWS S3 within an application should use the AWS SDK for JavaScript v3. Specifically, the @aws-sdk/lib-storage package is the recommended abstraction for managing complex multipart logic.

Installing Dependencies

Initialize your environment and install the necessary SDK modules:

npm install @aws-sdk/client-s3 @aws-sdk/lib-storage

Implementation Script

The following script demonstrates how to stream a local file to S3 efficiently, handling concurrency and monitoring progress via events.

const { S3Client } = require("@aws-sdk/client-s3");
const { Upload } = require("@aws-sdk/lib-storage");
const fs = require("fs");
const path = require("path");

const s3Client = new S3Client({ region: "us-east-1" });

async function uploadLargeFile(filePath, bucketName, s3Key) {
  const fileStream = fs.createReadStream(filePath);

  try {
    const parallelUploads3 = new Upload({
      client: s3Client,
      params: { Bucket: bucketName, Key: s3Key, Body: fileStream },
      queueSize: 4, 
      partSize: 1024 * 1024 * 10, // 10 MB part size
      leavePartsOnError: false, 
    });

    parallelUploads3.on("httpUploadProgress", (progress) => {
      console.log(`Uploaded Part: ${progress.part} | Loaded: ${progress.loaded} bytes`);
    });

    await parallelUploads3.done();
    console.log("Upload completed successfully!");
  } catch (error) {
    console.error("Upload failed:", error);
  }
}

uploadLargeFile("large-dataset.tar.gz", "my-production-bucket", "backups/large-dataset.tar.gz");

Method 3: Direct Browser Uploads using Pre-signed URLs

Routing large files through your application server can lead to bandwidth bottlenecks. The most performant architecture for a file upload in AWS S3 involves client-side uploads. By using Pre-signed URLs, you delegate the heavy lifting of the data transfer directly to the browser while maintaining security.

• Step 1: The client requests an upload authorization from your backend server.
• Step 2: The backend interacts with the S3 API to initiate the multipart upload and returns an UploadId.
• Step 3: The backend generates signed PUT URLs for each chunk and sends them to the client.
• Step 4: The browser uploads binary chunks directly to S3.
• Step 5: Upon completion, the client notifies the backend to assemble the object.

Optimizing Costs: Managing Incomplete Multipart Uploads

A critical cost optimization strategy for file upload in AWS S3 is managing orphaned parts. When a multipart upload fails or is interrupted, the uploaded chunks remain in the bucket, accruing storage costs indefinitely. According to AWS billing best practices, you must automate the deletion of these incomplete parts.

Configuring an S3 Lifecycle Rule:

1. Navigate to the Management tab of your bucket in the AWS Management Console.
2. Select Create lifecycle rule.
3. Select Delete expired object delete markers or incomplete multipart uploads.
4. Enable Delete incomplete multipart uploads and set a retention period (e.g., 7 days).
5. Save the rule to ensure your bucket remains clean and cost-efficient.

Conclusion

Successfully managing file upload in AWS S3 requires an understanding of how to handle large objects through multipart mechanisms. Whether you are using the CLI for administrative tasks, the AWS SDK v3 for backend integration, or Pre-signed URLs for client-side uploads, the focus should always be on concurrency, fault tolerance, and cost control. By implementing Lifecycle rules and optimizing your part sizes, you create a robust data pipeline that avoids common pitfalls and keeps your cloud architecture performing at its peak.