Uploading Large Files to S3 with Pre-Signed Url [React + Python]

This article will explain how to upload files to an S3 bucket using pre-signed URLs.

Prerequisites:

• Understanding of S3 and an S3 bucket for storing files.
• Understanding of Python and React Js.

Introduction:

S3 stands for Simple Storage Service that helps to store and retrieve files.
Now in many circumstances, we want our front-end/user to upload files directly to S3 but without knowing about details of the S3 bucket, credentials as it may contain other confidential data.

A pre-signed URL is basically a URL generated using IAM credentials that provide permission to write to the bucket. This URL will be then provided to the user which can be used to directly upload a file to S3. pre-signed URLs have an expiration time which tells when a file can be uploaded, after time expires access is denied for the link.

Create a Pre-Signed URL:

For generating URL, we will use python, boto3 (AWS-SDK for python).
We will create a function (API) that will call the s3 function : generate_presigned_url . The code looks like this:

This function will return a pre-signed URL which the front-end can consume to directly upload a file to the S3 bucket. The content type also can be restricted in Params if we want to restrict upload to specific file types.

Consume URL and upload to S3 directly — React JS

Now once we have the URL we can implement a form field in front that takes a file input and uses this URL to upload files directly.
Form field e.g:

Once the user selects a file from the system onChange will be triggered and will execute as below:

That’s it, Using this you can create a Pre-signed URL and upload a file directly to S3.

Few important things to consider:

We need to define cors policy for the bucket which users will try to upload files.
This helps us in restricting users from uploading files from selected domains and selected methods:

Cors Policy looks like this:

[
    {
        "AllowedHeaders": [
            *
        ],
        "AllowedMethods": [
            "PUT"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": []
    }
]

Another thing is bucket policy which defines actions that can be performed on your bucket, it looks like

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "your_s3_arn/*"
        }
    ]
}

This tells that anyone from the internet (principal: *) can perform GetObject action on your resource i.e anyone can access files of your s3 bucket.

I hope you will be able to generate a pre-signed URL and upload a file using it.
Please comment on the article if you are not able to understand or get stuck in any step.

Thanks, I hope you find it useful.