How to upload a local folder with subfolders to AWS S3

Photo by Gabriel Heinzer on Unsplash

AWS CLI

To upload a folder (including all subfolders and files) to an S3 bucket using the AWS CLI, you can use the aws s3 cp command with the --recursive flag.

✅ Basic Command

aws s3 cp /path/to/local/folder s3://your-bucket-name/your/prefix/ --recursive

📌 Example

aws s3 cp ./my-folder s3://my-bucket/data/ --recursive

This uploads:

• ./my-folder/file1.txt → s3://my-bucket/data/file1.txt
• ./my-folder/subdir/file2.txt → s3://my-bucket/data/subdir/file2.txt

🔍 Flags Explanation

📂 Sync Alternative

You can also use aws s3 sync, which only uploads new or changed files:

aws s3 sync ./my-folder s3://my-bucket/data/

This is more efficient for frequent uploads or large directories.

Boto3

To upload a folder (with subfolders and files) to AWS S3 using the Boto3 S3 client (in Python), you must manually walk through the folder structure using os.walk() because the S3 client doesn’t have a built-in recursive upload method (unlike the S3 resource's Bucket.upload_file method or AWS CLI).

Here’s how you can do it with the S3 client:

✅ Full Example: Uploading a folder (recursively) using Boto3 S3 client

import boto3
import os

def upload_folder_to_s3(bucket_name, folder_path, s3_prefix=""):
    s3_client = boto3.client('s3')

    # Walk through all directories and files
    for root, dirs, files in os.walk(folder_path):
        for file in files:
            local_path = os.path.join(root, file)

            # Create the relative S3 key
            relative_path = os.path.relpath(local_path, folder_path)
            s3_key = os.path.join(s3_prefix, relative_path).replace("\\", "/")  # Ensure forward slashes

            print(f"Uploading {local_path} to s3://{bucket_name}/{s3_key}")
            s3_client.upload_file(local_path, bucket_name, s3_key)

# Example usage:
upload_folder_to_s3(
    bucket_name='your-bucket-name',
    folder_path='/path/to/local/folder',
    s3_prefix='optional/s3/folder/prefix'  # Can be empty string
)

🔍 Key Points:

• os.walk() is used to recursively go through folders.
• os.path.relpath() calculates the path relative to the root folder you're uploading.
• We replace backslashes with forward slashes to conform to S3 key standards (especially on Windows).
• upload_file() is the S3 client method

Optional Improvements:

• Add error handling with try/except.
• Set ACLs or metadata via ExtraArgs.
• Use threading or multiprocessing for faster uploads if the dataset is large.