Copy files from EC2 to S3 Bucket in 4 steps

Table of Contents

Introduction to S3 and EC2 file copy

In this article, we are going to see how to copy files from ec2 instance to S3 bucket and vice versa, with the help of IAM roles and without having to use the AWS access key and secrets in the server.

In AWS technical terms.

Copying files from EC2 to S3 is called Uploading the file

Copying files from S3 to EC2 is called Downloading the files

The first three steps are the same for both upload and download and should be performed only once when you are setting up a new EC2 instance or an S3 bucket. the last and the fourth step is same except the change of source and destination.

Hope the following image helps you understand the difference and between upload and download.

Let see how to get both of this done.

Steps to copy files from EC2 instance to S3 bucket (Upload)

Create an IAM role with S3 write access or admin access
Map the IAM role to an EC2 instance
Install AWS CLI in EC2 instance
Run the AWS s3 cp command to copy the files to the S3 bucket

1. Create an IAM role with S3 write access or admin access

Create a new IAM role with S3 Admin Access which can be lateral mapped to the EC2 instance for easy S3 and EC2 integration.

Refer the following GIF to know how to create a new IAM role for S3 access

2. Map the IAM role to an EC2 instance

Choose the EC2 instance you want to assign this IAM role to.

Click on Actions > Security> Modify IAM Role

Search for the IAM Role we have created in the previous step and select it and hit Save.

3. Install AWS CLI in EC2 instance

Refer the following GIF to know how to install AWS CLI in Linux machine

to know more about AWS CLI commands and how to use them refer this article

4. Run the AWS s3 cp command to copy the files to the S3 bucket

Now as we have installed the aws command line you can simply use the following commands to copy the files to S3 Bucket from EC2.

# To List the S3 Bucket 
aws s3 ls s3://<S3bucketName>

# To copy the files from EC2 to S3
aws s3 cp <Fully Qualified Local filename> s3://<S3BucketName>

Refer the following GIF to know how it works in real-time.

Hope this quick article helps you to copy files from S3 to EC2 instance directly without any programmatic access and Keys

Steps to copy files from S3 bucket to EC2 instance (Download)

To copy the files from S3 bucket to EC2 instance, you have to follow the same steps except changing the source and destination on the Fourth Step

Create an IAM role with S3 write access or admin access
Map the IAM role to an EC2 instance
Install AWS CLI in EC2 instance
Run the AWS s3 cp command to copy the files from S3 to EC2

To copy the files from S3 to EC2, Keep the source as the bucket URL and the destination to your local directory or filename

# To copy the files from S3 to EC2
aws s3 cp s3://<S3BucketName> <Fully Qualified Local filename/Directory>

In the previous command, you can see the difference. Here the source is S3 Bucket URL and the destination is a local file name or directory name.

If you are interested to use S3 with ansible. refer this article.

How to use ansible with S3 – Ansible aws_s3 examples | Devops Junction

Hope it helps

What you think about this article ?. let me know in comments

Cheers
Sarav AK

Follow me on Linkedin My Profile
Follow DevopsJunction onFacebook orTwitter
For more practical videos and tutorials. Subscribe to our channel

Signup for Exclusive "Subscriber-only" Content