Clone EBS Volumes between AWS EC2 with Pulumi | Devops Junction

Updated on:

I had an Infrastructure as an automation requirement recently, where I had to create a solution that will find EBS volumes attached to an EC2 instance and create snapshots of the same and clone it and attach these cloned volumes to another EC2 instance.

I know it was a little complex to put in a single sentence. Let's go once again

Requirement is to

  1. Scan the Source EC2 instance and find the EBS volumes attached
  2. Create Snapshots ( Copy ) for all the EBS Volumes found
  3. Using the Snapshot, create a new EBS Volumes ( Clone )
  4. Attach the newly created Clone EBS Volumes to the Target EC2 instance

I did start with Terraform and spent some time and noticed it was a little complex as usual.

No question that Terraform is highly customizable and I am a big fan of it too.

However, I was looking for another option and wanted to give it a try for pulumi A Modern Infrastructure as Code.

AWS Pulumi

So basically my solution is expected to do the below set of tasks

  • Get details of all EBS volumes attached to an EC2 instance (Source - EC2 Instance)
  • Create a Snapshot of each EBS volume attached to the source EC2 Instance
  • Create volumes from the above-created snapshots (Here we need to clone EBS volumes)
  • Attach these newly created cloned EBS volumes to target EC2 Instance

 

For this use case, I chose 'Pulumi'  over Terraform for its simplicity.

What is Pulumi?

Pulumi is an open-source infrastructure as a code tool for creating, deploying and managing cloud infrastructure.

It is similar to terraform which can be termed as its twin sister,

Unlike Terraform, Pulumi can fit right into your application environment and it can be written in any of the following Languages

  • Python
  • TypeScript
  • JavaScript
  • Go
  • C#
  • F#

So that you don't have to learn an additional language like HCL for Terraform.

I will try to explain how I was able to achieve our objective using pulumi.

  In fact, we were also able to acheive the same in Terraform after spending hours of efforts.  We feel Pulumi is the winner here as it is simple and straight forward and easy to adopt and learn.
we will explain how did we do this objective on Terraform in our next article. Stay Connected.

Read more about the Terraform vs pulumi here

 

Pre-requisites

  • An AWS Account
  • An AWS CLI needs to be installed
  • Your AWS credentials.
    • You can create a new Access Key on this page.link
  • Python3 needs to be installed in machine
  • Pulumi Engine needs to be installed in machine. link

Set-up Instructions

  • Install Python3, pulumi locally.

  • Set up AWS credentials in ~/.aws/credentials.

  • once aws-cli is installed in the host machine, set up AWS credentials using the below command

    $ aws configure
    • This will ask for your AWS access key and AWS Secret as below
      • AWS Access Key ID [None] : ******
      • AWS Secret Access Key [None] : ******
      • Default region name : [leave it none, press enter]
      • Default output format : [leave it none, press enter]

Creating a workspace and Initializing Pulumi

Create an empty directory and then run 'pulumi new' command as this command generates a set of pre-defined files (requirements.txt, Pulumi.yaml,__main__.py)

$ pulumi new

The pulumi new command creates a new Pulumi project with some basic set of files based on the cloud and language specified.

If you are running pulumi new or most other pulumi commands for the first time, you will be prompted to log in to the Pulumi service. The Pulumi CLI works in the background with the Pulumi service in order to deliver a reliable experience. It is free for individual use.

Create a directory with some valid name

mkdir pulumi-devops-junction

 

Get into that directory

cd pulumi-devops-junction

 

Inside that directory execute the pulumi new command

pulumi new

 

Once you execute the 'Pulumi new' command, you should see an output similar to below

Using the up-down arrow key, select the aws-python option. Press Enter

Specify the following information

    1. Project Name
    2. Project Description
    3. Stack Name
    4. Aws Region

If you hit enter without giving any values pulumi will consider default values.  Once you specify all details, pulumi will run and try to install all necessary libs. You should see output similar to below


Now our pulumi project is set up and ready!! and we are ready to go and make changes in the placeholder code.

 

 

Pulumi Python script to Clone EBS Volumes and attach to other EC2

Now we have the default files created, Its time to update the placeholder file with our sourcecode.

copy the below content into __main.py__and replace the existing content.

"""An AWS Python Pulumi program"""
import pulumi
import pulumi_aws as aws
import datetime
import logging

logger = logging.getLogger(__name__)
ct = datetime.datetime.now()
ts = ct.timestamp()
config = pulumi.Config()
data = config.require_object("data")
# print ("Source Instance :",data.get("source_instance_id"))
source_instance = aws.ec2.get_instance(instance_id=data.get("source_instance_id"))
target_instance = aws.ec2.get_instance(instance_id=data.get("target_instance_id"))

ebslist = source_instance.ebs_block_devices
snapshot_name = "vol_snapshots"+"-"+str(ts)
cloned_vol_name = "cloned_vol"+"-"+str(ts)
vol_attachment = "vol_attachment"+"-"+str(ts)
try:
    for i in ebslist:
        logger.info("Loop over EBS list to get volume-id and device_name")
        try:
         source_snapshot = aws.ebs.Snapshot(snapshot_name+str(i.volume_id),volume_id=i.volume_id)
        except RuntimeError:
            logger.exception("Snapshot creation failed")
            raise
        try:
            source_cloned_volume = aws.ebs.Volume(cloned_vol_name+str(i.volume_id),availability_zone=data.get("az"),snapshot_id=source_snapshot.id)
        except RuntimeError:
            logger.exception("Volume creation from snapshots failed")
            raise
        try:
            cloned_vol_attachment = aws.ec2.VolumeAttachment(vol_attachment+str(i.volume_id),device_name=i.device_name,volume_id=source_cloned_volume.id,instance_id=target_instance.instance_id)
        except RuntimeError:
            logger.exception("Volume attachment to target instance failed")
            raise
except RuntimeError:
    logger.exception("Looping through EBS block list failed")
    raise

Yes, that's all and it is simple Python. In terraform it was not this easy ( wait for our next article to compare )

The Preceding python program will take Source_Instance_Id, Target_Instance_Id and region as input parameter and it will checks volumes attached to the Source instance and create snapshots of it.

Next, It will create a second set of volumes from these snapshots and attach these volumes to the target instance.

 

Update the Values and Pulumi up ( Run the script )

Please change the following set of parameters in pulumi.dev.yaml file.

  • source_instance
  • target_instance
  • region

aws pulumi

Pulumi.*.yaml is a configuration file where we can maintain our input parameters and the same can be used in our code. more like a variables file. in terraform it is *.tfvars file

Now run the Pulumi up command.

Like Terraform plan  or Terraform apply command Pulumi will try to get your confirmation before moving on.

You can simply choose no if you just want to plan or dry run and choose yes to apply the changes.

You should see output similar to below

Select Yes  for the question Do you want to perform this update question and pulumi will create your desired infrastructure for you.

 

That's all. Sit back and relax for a minute or a few.

You can validate and confirm if your infra is created as per requirement by logging into the AWS console

 

Once this exercise is done, do not forget to clean up resources using pulumi destroy command in order to avoid unnecessary charges

Voila!!.. You have achieved your requirement with pulumi.

This highly scalable solution where we can read and clone multiple numbers of volumes attached to EC2 instances in no time

 

GitHub Link for the Sourcecode

You can download the source code from this github link 

 

Conclusion

We have learnt how to clone EBS Volumes attached to One Ec2 instance (Source) and attach them to another EC2 instance (Target)

During the implementation, we learned how Pulumi is set up and configured.

Hope you would give it a go for Pulumi. Stay connected with us to learn how to do this objective on Terraform.

We are already halfway through and it is not as easy as Pulumi. Stay Connected.

For any DevOps or IaaS consultation reach out to us at [email protected]

 

Co Authored by

Tapan Hegde  & Sarav AK

Follow me on Linkedin My Profile
Follow DevopsJunction onFacebook orTwitter
For more practical videos and tutorials. Subscribe to our channel

Buy Me a Coffee at ko-fi.com

Signup for Exclusive "Subscriber-only" Content

More from Middleware Inventory

  • AWS CLI EC2 Stop and Terminate command Examples

    In AWS infrastructure, We create a lot of EC2 instances on demand and we tend to forget about the stopped instances considering that it is stopped and not being billed for. But the fact is that Even the stopped instances would cost you a couple of pennies in various forms…

  • AWS EC2 CLI List Examples - Describe instances | Devops Junction

    AWS CLI is a very great help when it comes to efficiently managing your AWS Cloud Infrastructure and your EC2 instances. While we are managing our AWS Infrastructure,  we cannot always afford to login to the AWS console all the time and it is not recommended from the security perspective…

  • Ansible EC2 Create
    Ansible EC2 Example - Create EC2 instance with Ansible

    Infrastructure as Code is getting all attention it deserves and everyone is trying to find their way to the `Completely automated Infrastructure Provisioning & Management` While there are a lot of tools available now in the market starting from Terraform, AWS CloudFormation, Chef, Puppet, Salt Stack There are some differences…

  • Terraform EC2 user_data example
    Terraform AWS EC2 user_data example - aws_instance| Devops Junction

    In this article we are going to see how to create AWS EC2 instance with user_data. While launching AWS Ec2 instances we might want to add some customizations to it like adding a key and executing few shell commands or scripts to make sure they are ready to launch your…

  • Oracle DB backup S3
    Automated Oracle Database Backup to S3

    The Beautiful thing about Digital world is that we have a Ctrl+z button, snapshots and backups to go back in time and correct our mistakes, which is not possible in our real life. Enough with a philosophy. I would not have to tell you how crucial backups are and when…