Terraform import All AWS Security Groups - How to

In this post, we are going to see how to manage existing and already created AWS Security groups with Terraform. The new era of Infrastructure revolution has begun already and we already started provisioning, managing, administrating our Infra as a code with help of Configuration management tools like Ansible, Terraform, SaltStack etc.

If you are new to Terraform and to know how to use Terraform with AWS basics, Refer this article and come back here.

With no further ado, let us jump into today's mission.

The Problem we are trying to Solve

We know how powerful terraform and how efficient we can manage or create the Entire AWS/GCP/DigitalOcean Cloud Infrastructure with Terraform but we often find ourselves having a few additional resources which are created earlier often manually and not managed by Terraform.

To bring all of these unmanaged resources (security groups) into Terraform and make them as Infra as code. We can use Terraform import command.

But it is not productive to import these resources one by one. So I tried to automate it with Ansible and this is how it goes.

 

Importing and Managing Existing AWS Security Groups with Terraform

Still, many DevOps engineers are logging into AWS Management console to update the Security group Inbound and Outbound traffic routes manually like opening ports, enabling traffic route etc. There is nothing wrong about it but the problem is you cannot track the changes your team is making unless you have it under revision control. that is where the Infra as code is winning.

Imagine you have your security group as a code and every change you make is properly committed and manged in your Git repositories like BitBucket or Github.  Every change would be tracked with git log and commit message and there would be a backup of the configuration too in case you want to roll back. Not just that. I listed a few reasons why you should be managing your AWS Security group as a code.

• It enables you to track all your changes like Open/Close ports and the reasons for the changes.
• You can do better Security Auditing with Commit messages
• Easy and efficient management
• Enabling you to automate your infrastructure further
• Easily find your publically open routes and prevent any security incident before happening
• Better Continuity of Business/Disaster Recovery strategy.
• Allow/Disallow IP/Port/Group with a single click of Jenkins job

So I cannot insist enough to manage your AWS Security groups as a code. so now let us see how I managed to import my existing AWS security groups ( All of them ) into Terraform and manging it.

Let us see it in a practical way. how it is possible

 

Ansible + Terraform - to the help

I am a fan of both Ansible and Terraform, Both are wonderful when it comes to managing the infrastructure, while Ansible is pythonic, Terraform has its own language called Hashicorp Configuration Language and supports JSON too.

The reason why we are going to use Python here is that we are going to fetch all the Security groups and its information using Ansible's ec2_group_facts module this can be done with the AWS CLI too but I preferred this way for better data processing.

here is the pictorial representation of how our design is going to be.

If you are not able to understand my handwriting in the picture, Here I write the same for additional reference.

In our setup. Ansible playbook is the main component which invokes both Terraform and EC2 data collection using ec2_group_facts module.

For Ansible to be able to access AWS you need to have AWS programmatic access enabled and Authentication keys in your environment variable.

$ export AWS_ACCESS_KEY_ID=AK************IEVXQ
$ export AWS_SECRET_ACCESS_KEY=gbaIbK*********************iwN0dGfS

This is enough to enable Ansible and Terraform to access your AWS Infrastructure. But however, there are few hidden details and problems you may encounter ( like i did ) you can refer the following articles in case you are stuck or let me know in comments, I will try to help as soon as I can

• Setup Boto python for Ansible AWS
• Setup programmatic Access for AWS – Key and Secret
• Install and configure AWS CLI
• Terraform AWS Configuration Basics

Having your environment ready for Ansible and Terraform we can proceed further.

 

How Ansible and Terraform works together

Having completed the Access setup for AWS and Ansible integration,  You would be able to reach your AWS account from ansible and execute all AWS related modules.

In this playbook also we are going to use such a module named ec2_group_facts which helps us to fetch all the information about the security groups.  that is going to be our first step

Then we will create a directory for each security group we are fetching from the AWS account and creating the terraform configuration files using the terraform import command and the security group id

Once the import is complete, we need to remove a few configuration elements in the terraform configuration file such as ownerid, arn, group id etc, these values/elements should be auto-populated by Terraform so terraform would not let you define it prior.

Once we have removed these auto-generated elements/variables from the configuration file, we are going to validate the file using the terraform validate command

So these are the four steps we are going to perform in order to get all your Security groups managed by Terraform.

 

The Ansible Playbook to import all security groups and add to Terraform

So, The time has come for some code, Enough with the theory.

Few Points to remember before executing the playbook

• You need to update the destdir variable where ansible would create new directories for each security group it is fetching. the directory name would be as same the security group name ( if there are spaces in the security group name it would be converted to - hyphen )
• You need to replace the vpc-id variable of your corresponding vpc

 

The Playbook

---
 - name: Security Group Playbook
   hosts: localhost
   vars:
      destdir: /apps/gritfy/Terraform/SecGroups
      itemstochange: ['arn\s+=.+$','\sid\s+=.+$','owner_id\s+=.+$']

   tasks:
    - name: ec2 security group information fetch
      ec2_group_facts:
        filters:
          vpc-id: vpc-0a8ae2c90f5ca6cfa
      register: result

    - name: Creating a dictionary of Security Group IDs and Names
      set_fact: 
        secdict: "{{ secdict | default ([]) + [ { 'name': item.group_name.replace(' ','-'), 'id': item.group_id } ] }} "
      with_items: "{{result.security_groups}}"
      loop_control:
        label: "{{ item.group_name}}"

    - name: Create the Directory  
      file:
          path: "{{destdir}}/{{item.name}}{{item.id[0:7]}}" # required. Path to the file being managed.
          state: directory 
      register: dircrt
      loop: "{{secdict}}"

    - name: Terraform Import
      shell: |
        git init
        echo 'provider "aws" {\n\tregion = "us-east-1"\n} \n\nresource  "aws_security_group" "elb_sg" {\n\n}' > main.tf
        terraform init
        terraform import aws_security_group.elb_sg {{item.id}} 
        echo 'provider "aws" {\n\tregion = "us-east-1"\n} \n' > main.tf
        terraform show -no-color >> main.tf
        git add .
        git status
        git commit -m "Updated Git"
        pwd && ls -lrt
      args: 
        chdir: "{{destdir}}/{{item.name}}{{item.id[0:7]}}"
      loop: "{{secdict}}"
      when: dircrt is changed

    - name: Change config
      lineinfile:
          path: "{{destdir}}/{{item.0.name}}{{item.0.id[0:7]}}/main.tf"
          regexp: "{{ item.1 }}"
          state: absent
          backup: yes
      with_nested:
        - "{{ secdict }}"
        - "{{ itemstochange }}"
      
    - name: Terraform Validate
      shell: |
        terraform validate 
      args:
        chdir: "{{destdir}}/{{item.name}}{{item.id[0:7]}}"
      loop: "{{secdict}}"
      register: tfvalidate
      failed_when: "'Success' not in tfvalidate.stdout"

 

Ansible Playbook tasks explained

Let us go through the playbook and understand what each task is designed to do.

Task1: EC2 information fetch

This task is using the module named ec2_group_facts and it uses the AWS access key and secret from your environment directly to connect to your AWS account and fetch all the security groups belong to the specific vpc which is mentioned in the arguments.  make sure to update the vpc-id filter value before running the playbook.

 

Task2:  Creating a Dictionary with the Collected Values

ec2_group_facts module would result in a lot of information about the security group but all we need is a security group name and security group id

So we are iterating through the collected output from the previous task and creating a dictionary named secdict we are using set_fact module to create a variable during the runtime

 

Task3:  Creating a Directory for each security group - Naming Convention

As mentioned earlier, we would be creating dedicated directories for each security group where the Terraform configuration file of that corresponding sec group would be saved.

The Name of this directory would be combo of the security group name and the first 7 characters of the security group id, this is to avoid duplicates. AWS does not have unique name constraint for the security group naming, so there can be one more security groups with the same name.

 

Task4:  Terraform Importing tasks

in this task, we are running multiple commands one after another just like we type in a shell prompt. using ansible shell module  These are list of tasks we are going to do with this task

1. we initialize a git repo under the newly created sec group directory
2. create a terraform configuration file named main.tf with AWS provider specifications
3. Performing terraform init to initialize terraform on the directory
4. Performing the import process with terraform import command and the corresponding security group's id
5. Writing the imported configuration back into main.tf configuration file we have created at step2
6. Rest of the steps are for version controlling changes like add, commit etc.

 

Task5:  Terraform file correction and removing the unwanted fields

In this task, we are removing the unwanted lines from the created main.tf configuration file which otherwise makes the file syntactically incorrect

Task6:  Validate the configuration after the changes

Finally, we can validate if the Terraform configuration file main.tf is valid for each security group using the terraform validate command which would be executed inside all the security group directories. It has been designed to fail when the validate command returns a failure message using ansible failed_when

Playbook Execution and Result

If you have done everything correct. You would be able to import all the security groups belong to the specific VPC in your AWS account and would be able to manage using Terraform.

You would have directories created with the security group names beneath the workspace(destdir) directory you have defined in the playbook

 

Make Changes in Terraform File and Apply it to verify

Once the configuration files are imported and ready go to any of the security group directory created and edit the main.tf file and make changes like adding a new ingress rule or changing the CIDR IP address for the allowed port number etc

Once you have done the changes. you are good to go and terraform plan and then terraform apply it

You would be able to see that your security group is successfully managed by Terraform.

Any further change you would like to make to the security group can be done with Terraform Infrastructure as code technique. (Refer the video for more information)

Congratulations.

 

Video Guide - Screen record

Here is the screen record of me, talking about this setup and testing this playbook in my aws account.

 

Hope this article is helpful

 

Cheers
Sarav AK சரவணன் அ.க

---