Creating S3 presigned URLs using Python Boto3 - On AWS Lambda

In this article, we are going to see how to create pre-signed S3 URLs using Boto3 Python and deploy the program to Lambda for accessibility and reusability.

So this article could be a good example for starting your serverless journey too.

with no further ado, let us move on to the objective.

Before we go, let's revise once what is S3 Presigned URLs mean and what they are used for

 

What is S3 Presigned URL

By default, all S3 objects are private. Only the object owner has permission to access them. However, the object owner can optionally share objects with others by creating a pre-signed URL, using their own security credentials, to grant time-limited permission to download the objects.

The presigned URLs are valid only for the specified duration.

If you created a presigned URL using a temporary token, then the URL expires when the token expires, even if the URL was created with a later expiration time.

Anyone who receives the presigned URL can then access the object.

These presigned URLs are helpful in various cases

• To distribute a Software/Video Download URL to customers with limited validity
• To use in business workflows like Document Signing which are valid for a specific duration.
• Using masked URLs for Videos or Digital content posted in your blog or website
• Restrict the permissions and define conditions on who can download the objects and for how long.

Various Products and companies are using S3 pre-signed URLs.

One best example I can give you here is Notion.

Yes, Notion use pre-signed URLs and create temporary URLs as you upload images/videos in your article which might not be available if you hit directly or available for a duration.

Python Program to Create PreSigned URL - Boto3

We have a real use case where we are distributing our open source product SQSCLI  using S3 pre-signed URLs.

SQSCLI - A simple Desktop Application to browse AWS SQS Messages

We are using Lambda to host this Python Application which generates the presigned S3 links when the user needs a download link.

However, you can test it without lambda too with local AWS CLI.

 

For Local Development - using Local AWS CLI profile

Here is the simple python program which can run fine on your local(laptop/desktop) using the AWS CLI profile available in your local.

import logging
import boto3
import botocore
from botocore.exceptions import ClientError


def create_presigned_url(bucket_name, object_name, expiration=600):

    # Choose AWS CLI profile, If not mentioned, it would take default
    boto3.setup_default_session(profile_name='personal')

    # Generate a presigned URL for the S3 object
    s3_client = boto3.client('s3',region_name="ap-south-1",config=boto3.session.Config(signature_version='s3v4',))
    try:
        response = s3_client.generate_presigned_url('get_object',
                                                    Params={'Bucket': bucket_name,
                                                            'Key': object_name},
                                                    ExpiresIn=expiration)
    except Exception as e:
        print(e)
        logging.error(e)
        return "Error"

    # The response contains the presigned URL
    print(response)
    return response


create_presigned_url('devopsjunction','sqscli-windows.zip',600)%

This can run fine on the local but not on AWS Lambda. If you want to host this python program on AWS Lambda you need to make some changes.

 

For Serverless Setup - using Lambda and IAM role

Here is the python program hosted on Lambda. This is specially designed to host on Lambda as it has an extra AWS Lambda function lambda_handler

import json
import logging
import boto3
import botocore
from botocore.exceptions import ClientError

def create_presigned_url(bucket_name, object_name, expiration=600):

    # Generate a presigned URL for the S3 object
    s3_client = boto3.client('s3',region_name="ap-south-1",config=boto3.session.Config(signature_version='s3v4',))
    try:
        response = s3_client.generate_presigned_url('get_object',
                                                    Params={'Bucket': bucket_name,
                                                            'Key': object_name},
                                                    ExpiresIn=expiration)
    except Exception as e:
        print(e)
        logging.error(e)
        return "Error"

    # The response contains the presigned URL
    return response
    

def lambda_handler(event, context):
    
    URLs=[]
    URLs.append(create_presigned_url('devopsjunction','SQSCLI.exe.zip',3600))
    URLs.append(create_presigned_url('devopsjunction','SQSCLI.dmg.zip',3600))

As you can see we are using this Python Boto3 code to create presigned URLs for the files stored in our s3 bucket named devopsjunction

we are creating short-lived S3 URLs or presigned URLs here for two files

• SQSCLI.exe.zip
• SQSCLI.dmg.zip

We have a function named create_presigned_url  which accepts the following parameters

• bucket name
• object name
• expiration in seconds ( default 600)

Let us see how this function is designed in detail

First, we are establishing a connection to AWS

s3_client = boto3.client('s3',region_name="ap-south-1",config=boto3.session.Config(signature_version='s3v4',))

Inside a try block, we are trying to generate a pre-signed URL

using the method generate_presigned_url

s3_client.generate_presigned_url('get_object', Params={'Bucket': bucket_name, 'Key': object_name}, ExpiresIn=expiration)

If you run this function on the server or your laptop, it would try to use AWS CLI profile as authentication.

In our case, we are hosting this function in AWS Lambda so we need to assign S3 read access to this Lambda. function

 

How to deploy Python Code to Lambda

If you are new to AWS Lambda and wondering how to deploy your Python code. You can learn it from our other exclusive article.

AWS Lambda Python Example – With ALB Integration | Devops Junction

 

Giving Permission to AWS Lambda Function to access S3 Bucket

By default, when you are creating an AWS lambda function you will have an IAM profile created for you or you can choose to create one.

Irrespective of you choose to create an IAM profile or use the existing one. Just make sure that your IAM profile has the following policy mapped to it

In our case SQSCLIDownload lambda function has the IAM profile named SQSCLIDownload-role-0a4iyemi which is auto-created.

We are editing this IAM role and adding an Amazon Managed S3ReadOnlyAccess policy. It enables our Lambda function to access our S3 objects

This is what Amazon S3 Read Only Access Provides.

 

Testing your Lambda Python Function - Presigned URLs

Now we have deployed the code that creates S3 Presigned URLs.

Time to test it.

If all goes well when you click on Test on the Lambda Function console. you should see the output

In my case, the test would return two pre signed URLs which customers can use to download two different distributions of our SQSCLI product

If you want this function to be available publically you can add a Trigger with your Lambda function

• API Gateway
• Application Load Balancer

We have covered this in detail on our exclusive article 

As you can see it on the screenshot, My Lambda function is already having the API Gateway Trigger

If you access the URL given by the API Gateway Trigger using CURL. you can see the presigned URLs.

You can use this directly in your system for any workflow automation.

Conclusion

This is how you can use python boto3 to create a pre-signed S3 URL.  Additionally, This Python app is hosted on AWS Lambda as a serverless application.

In this article, we have learnt

• How to create a python boto3 program to create S3 pre-signed URLs
• Deploying the Python boto3 application to AWS Lambda and exposing it with API Gateway
• Configuring IAM permissions for Lambda to access S3 objects.
• Accessing the Lambda Python Application using API Gateway trigger.

Cheers
Sarav AK

---