How to Upload Large Files to AWS S3: The Definitive Guide

Understanding AWS S3 Upload Limits and Challenges

For developers and system architects, performing a file upload in AWS S3 is a fundamental task, but it comes with specific architectural constraints. Amazon Simple Storage Service (AWS S3) is the industry standard for object storage, yet standard single PUT requests are limited to a maximum upload size of 5 GB. Uploading files larger than this, or managing large datasets over unstable networks, requires more sophisticated handling to avoid timeouts and data corruption.

When an upload fails at 99%, restarting the entire process is highly inefficient. It wastes expensive bandwidth, introduces latency, and degrades the user experience. To ensure high availability, speed, and overall performance, AWS provides dedicated mechanisms for Multipart Uploads. This feature splits data into manageable, resumable segments that can be processed in parallel. In this guide, we will explore how to master the file upload in AWS S3 process using the AWS CLI, AWS SDK v3, and browser-based strategies, ensuring your cloud storage architecture remains resilient, secure, and cost-effective.

The Physics of S3 Uploads: Single PUT vs. Multipart Uploads

To construct a highly reliable storage backend, it is important to understand why single-stream uploads fail over public networks. Standard HTTP PUT operations rely on a continuous TCP connection. As packet latency or network jitter increases, the TCP window size fluctuates, which drastically limits throughput. If a connection is interrupted even momentarily, the session drops, requiring a complete restart.

The table below highlights the structural differences between these two file upload methods in AWS S3:

Feature / Parameter Single PUT Upload Multipart Upload Maximum Object Size 5 GB 5 TB Minimum Object Size 0 Bytes 5 MB (except the last part) Parallelism No (Single connection stream) Yes (Up to 10,000 parallel parts) Resiliency to Network Drops Low (Must restart upload from 0%) High (Only re-upload failed parts) Part Size Limits N/A 5 MB to 5 GB Max Number of Parts N/A 10,000 parts

What is AWS S3 Multipart Upload?

AWS S3 Multipart Upload is an API feature that allows users to upload a single object as a set of smaller parts. By breaking a large file into individual chunks, the upload process gains significantly improved reliability and performance.

• Improved Throughput: Parts can be uploaded in parallel, allowing you to maximize available network bandwidth.

• Resiliency: If a network failure occurs, you only need to re-upload the affected chunk rather than the entire file.

• Scalability: This method supports objects ranging from 5 MB up to 5 TB in total size.

• Pause and Resume: Upload parts can be streamed over time, paused, or resumed based on system resource availability and network state.

How the S3 Multipart Workflow Works Under the Hood

The multipart process is split into three core API phases:

1. Initiation: You notify S3 that you intend to upload an object. S3 responds with a unique UploadId.

2. Upload Parts: You upload individual chunks of the file in any order. Each upload request must contain the UploadId and a sequential Part Number (from 1 to 10,000). S3 returns an ETag header for each completed chunk. Your system must track these values.

3. Completion: You send a request containing the UploadId along with an ordered list of part numbers and their respective ETag values. S3 then reconstructs the complete file inside your bucket.

Architectural Rule: Because S3 permits up to 10,000 parts, your choice of part size determines the maximum file size you can upload. For instance, if you use the minimum part size of 5 MB, the maximum object you can upload is 10,000 * 5 MB = 50 GB. To upload a 5 TB object, you must scale your part size up to 500 MB.

Method 1: Uploading Large Files with AWS CLI

The AWS Command Line Interface (CLI) is the most accessible method for managing a large file upload in AWS S3. It automatically manages multipart logic behind the scenes, splitting files larger than the default threshold of 8 MB into smaller segments and executing parallel transfers.

Step 1: Install and Configure the AWS CLI

Verify your installation by running the command below. If you do not have it installed, follow the official AWS CLI installation guide.

aws --version

Configure your local environment with proper IAM credentials, selecting your default region (e.g., us-east-1):

aws configure

Step 2: Execute the Upload

Use the standard aws s3 cp command. The CLI detects the file size, reads your configuration, and automatically invokes parallel multipart uploads if the file exceeds the threshold:

aws s3 cp /path/to/local/large-file.mp4 s3://your-bucket-name/large-file.mp4

Step 3: Tuning Performance Settings

To optimize performance for high-bandwidth networks (such as AWS EC2 instances running inside a VPC), you can adjust CLI configurations. This enables you to maximize your network card's capabilities:

# Configure the multipart threshold to 64 MB (files below this use standard PUT)
aws configure set default.s3.multipart_threshold 64MB

# Set individual part size to 16 MB
aws configure set default.s3.multipart_chunksize 16MB

# Increase concurrent request limit to speed up transfers
aws configure set default.s3.max_concurrent_requests 20

If you need to sync entire directories containing numerous large media assets, use the aws s3 sync command instead, which utilizes the exact same background multipart configurations:

aws s3 sync /data/local-folder s3://your-bucket-name/data-folder --storage-class STANDARD_IA

Method 2: Programmatic Uploads via Node.js (AWS SDK v3)

Developers implementing a file upload in AWS S3 within an application should use the modern AWS SDK for JavaScript v3. Specifically, the @aws-sdk/lib-storage package is the recommended high-level abstraction for automatically managing complex multipart uploads.

Why Use lib-storage Instead of the Client S3 Command Directly?

The base package @aws-sdk/client-s3 requires you to manually track your UploadId, keep an array of ETag objects in memory, manage individual HTTP streams, and handle manual re-tries. The lib-storage library abstracts this complexity, automatically switching between a single PUT and a multipart upload depending on the input size.

Installing Dependencies

Initialize your Node.js environment and install the modular SDK v3 packages:

npm install @aws-sdk/client-s3 @aws-sdk/lib-storage

Implementation Script

The following production-ready script demonstrates how to stream a local file to S3 efficiently, handle concurrency, and monitor upload progress:

const { S3Client } = require("@aws-sdk/client-s3");
const { Upload } = require("@aws-sdk/lib-storage");
const fs = require("fs");
const path = require("path");

// Instantiate the S3 client
const s3Client = new S3Client({ region: "us-east-1" });

/**
 * Uploads a large file to AWS S3 using lib-storage
 * @param {string} filePath - Absolute path to the local file
 * @param {string} bucketName - Target S3 Bucket name
 * @param {string} s3Key - S3 destination path/key
 */
async function uploadLargeFile(filePath, bucketName, s3Key) {
  // Check if file exists
  if (!fs.existsSync(filePath)) {
    throw new Error(`File not found at path: ${filePath}`);
  }

  const fileStream = fs.createReadStream(filePath);
  const fileStats = fs.statSync(filePath);
  const totalSizeBytes = fileStats.size;

  console.log(`Starting upload for ${path.basename(filePath)} (${(totalSizeBytes / (1024 * 1024)).toFixed(2)} MB)...`);

  try {
    const parallelUploads3 = new Upload({
      client: s3Client,
      params: { 
        Bucket: bucketName, 
        Key: s3Key, 
        Body: fileStream,
        ContentType: "application/octet-stream"
      },
      // Number of concurrent upload parts
      queueSize: 4, 
      // Individual chunk size (10 MB in this case)
      partSize: 1024 * 1024 * 10, 
      // Set to false to clean up parts if the overall process fails
      leavePartsOnError: false, 
    });

    parallelUploads3.on("httpUploadProgress", (progress) => {
      const percentage = progress.total ? ((progress.loaded / progress.total) * 100).toFixed(2) : "Unknown";
      console.log(`[Progress] Uploaded Part: ${progress.part} | Loaded: ${progress.loaded} bytes | Total: ${progress.total || "N/A"} (${percentage}%)`);
    });

    const output = await parallelUploads3.done();
    console.log("Upload completed successfully! Location:", output.Location);
  } catch (error) {
    console.error("An error occurred during S3 multipart upload:", error.message);
    throw error;
  }
}

// Example usage:
// uploadLargeFile("large-dataset.tar.gz", "my-production-bucket", "backups/large-dataset.tar.gz");

Method 3: Direct Browser Uploads using Pre-signed URLs

Routing large files (such as 1 GB+ video files) through your application server can quickly lead to bandwidth bottlenecks, high RAM utilization, and server timeouts. The most performant architecture for user-facing applications involves client-side uploads directly to S3.

By using Pre-signed URLs, you delegate the heavy lifting of the data transfer directly to the client's browser while maintaining strict security controls on your backend.

The Architectural Workflow

1. Step 1: The client browser requests an upload authorization from your backend server, providing the filename and size.

2. Step 2: Your backend server uses AWS SDK credentials to request a multipart initiation from S3, which returns an UploadId.

3. Step 3: The backend calculates the number of parts needed and generates signed PUT URLs for each chunk. For instance, a 100 MB file split into 10 MB chunks requires 10 pre-signed URLs.

4. Step 4: Your backend returns the UploadId and the list of signed URLs to the client.

5. Step 5: The client's browser reads the file locally, slices it into chunks using the JavaScript Blob API, and uploads each chunk to its corresponding pre-signed URL using parallel HTTP PUT requests.

6. Step 6: Once all uploads succeed, the client sends a completion request containing the UploadId and the list of returned ETags to your backend, which then marks the upload complete in S3.

Node.js Backend Blueprint: Generating Multipart Pre-signed URLs

Below is a backend example demonstrating how to initiate the upload and generate pre-signed URLs for each part:

const { S3Client, CreateMultipartUploadCommand, UploadPartCommand } = require("@aws-sdk/client-s3");
const { getSignedUrl } = require("@aws-sdk/s3-request-presigner");

const s3Client = new S3Client({ region: "us-east-1" });

async function getPresignedUrlsForMultipart(bucketName, s3Key, totalParts) {
  // Step 1: Initiate Multipart Upload to get UploadId
  const initResponse = await s3Client.send(new CreateMultipartUploadCommand({
    Bucket: bucketName,
    Key: s3Key,
  }));
  
  const uploadId = initResponse.UploadId;
  const presignedUrls = [];

  // Step 2: Generate a Pre-signed URL for each part
  for (let partNumber = 1; partNumber <= totalParts; partNumber++) {
    const command = new UploadPartCommand({
      Bucket: bucketName,
      Key: s3Key,
      UploadId: uploadId,
      PartNumber: partNumber,
    });

    // URLs will remain valid for 1 hour (3600 seconds)
    const url = await getSignedUrl(s3Client, command, { expiresIn: 3600 });
    presignedUrls.push({ partNumber, url });
  }

  return { uploadId, presignedUrls };
}

Security and Compliance Best Practices

When implementing S3 uploads, protecting data in transit and at rest is a critical requirement. You should configure proper IAM policies, use encryption, and secure public access paths.

1. IAM Policy Principles

Your IAM roles or service accounts should only have the minimum permissions required to perform uploads. Avoid broad wildcards like s3:*. The policy below restricts access to the exact operations needed for multipart uploads within a specific bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Resource": "arn:aws:s3:::my-production-bucket/*"
    }
  ]
}

2. Server-Side Encryption (SSE)

To enforce data encryption during transit and storage, require all PUT operations to include encryption headers. You can configure S3 to use managed S3 keys (SSE-S3) or KMS keys (SSE-KMS). When utilizing pre-signed URLs or programmatic Upload APIs, ensure encryption flags are set within your request parameters.

Optimizing Costs: Managing Incomplete Multipart Uploads

A critical cost-optimization strategy for S3 is managing orphaned, incomplete parts. When a multipart upload is initiated but never completed (due to network drops, client browser crashes, or process terminations), the uploaded parts remain stored in your bucket indefinitely. S3 continues to charge you standard storage rates for these incomplete parts, even though the final object is not visible in your bucket.

In high-throughput environments, this can lead to TBs of "ghost" storage, costing hundreds or thousands of dollars in hidden charges. To prevent this, you should set up an S3 Lifecycle rule to automatically clean up incomplete uploads.

Step-by-Step Configuration in the AWS Console

1. Open the AWS Management Console and navigate to your target S3 bucket.

2. Click on the Management tab.

3. Under the **Lifecycle rules** section, click Create lifecycle rule.

4. Provide a descriptive name, such as CleanUpIncompleteMultipartUploads.

5. Under **Rule scope**, select Apply to all objects in the bucket. Acknowledge the warning.

6. Under **Lifecycle rule actions**, select Delete expired object delete markers or incomplete multipart uploads.

7. Check the box for Delete incomplete multipart uploads.

8. Set the number of days to keep incomplete uploads active (e.g., 7 days). This gives users plenty of time to resume active uploads while ensuring failed ones are eventually cleaned up.

9. Click Create rule to save your changes.

Defining the Rule via CloudFormation or Terraform

For infrastructure-as-code (IaC) pipelines, you can easily define this rule within your templates. Here is an example S3 Lifecycle Configuration block written in Terraform:

resource "aws_s3_bucket_lifecycle_configuration" "cleanup_rule" {
  bucket = aws_s3_bucket.my_bucket.id

  rule {
    id     = "abort-incomplete-multipart-uploads"
    status = "Enabled"

    filter {} # Applies to all objects in the bucket

    abort_incomplete_multipart_upload {
      days_after_initiation = 7
    }
  }
}

Troubleshooting Common S3 Upload Errors

When building production integrations, you are likely to encounter a few common S3 API exceptions. Here is how to diagnose and resolve them:

1. EntityTooSmall (HTTP 400 Bad Request)

• The Cause: You attempted to upload a part that is smaller than 5 MB. AWS S3 requires every part in a multipart upload to be at least 5 MB, with the exception of the very last part.

• The Fix: Check your slicing algorithm on your client or backend. Ensure that every part (except the last one) is at least 5,242,880 bytes in size.

2. InvalidPart (HTTP 400 Bad Request)

• The Cause: The part list sent during the *Complete Multipart Upload* phase did not match the ETags or part numbers generated during the *Upload Parts* phase, or a part was modified after upload.

• The Fix: Verify that your database or state management system preserves the exact, case-sensitive ETag string returned by S3, and that you assemble the parts list in ascending, sequential order.

3. RequestTimeout (HTTP 408)

• The Cause: High latency or poor upload bandwidth is causing your HTTP requests to time out before S3 receives the full part payload.

• The Fix: Reduce the part size (e.g., from 100 MB down to 10 MB or 5 MB) and implement an exponential backoff retry mechanism inside your upload code.

Conclusion & Architectural Checklist

Successfully managing a file upload in AWS S3 requires choosing the right approach for your scale. Whether you are using the CLI for administrative migrations, the AWS SDK v3 to build a secure backend stream, or Pre-signed URLs for highly scalable, direct-from-browser uploads, focusing on parallelism and cleanup will save you time and lower your cloud costs.

Before launching your next high-volume S3 integration, verify your setup against this checklist:

• Is your default part size configured to handle your maximum expected file size within the 10,000-part limit?

• Are you using the high-performance AWS SDK v3 lib-storage helper library instead of raw client calls?

• Are large client uploads utilizing pre-signed URLs to protect your backend servers from network and memory limits?

• Have you deployed an S3 Lifecycle Rule to abort incomplete multipart uploads and prevent hidden charges?

• Are your IAM policies configured to follow the principle of least privilege, allowing only necessary permissions?