Using SOPS with AWS KMS - Encrypt and Decrypt files | Devops Junction

Secret Management is an essential part of infrastructure and DevOps. how do you store your Secrets and Environment variables define how safe your infrastructure is

As a good engineering practice, most companies guide their developers to not hard code the secrets and environment variable directly but out of convenience or necessity Engineers tend to commit the secrets/environments to the repositories

It is good to stop them from doing so and employ some constraints and code quality checks to prevent these unintended data from getting committed or entering into the code base.

Another simple efficient way is to encrypt the sensitive data and decrypt it when needed.

Yes. This is a simple ancient encryption and decryption am talking about but not in the ancient ways of creating your own system and keys and sharing the keys amongst the engineers.

I know the management of that KEY itself is going to be tedious and sharing the key between teams and ensuring it's not falling into the wrong hands is another nightmare

If the key gets to the middleman during the sharing or if one of the engineers is compromised then there is no point of security, is not it?

So let's see how to do this ancient technique of encryption/decryption in a modern way

 

In this article, we are going to discuss an efficient way of encrypting and decrypting files with native AWS KMS keys and IAM

Before we move further, we need to learn about an open-source product we are going to use for these secret operations and it is named SOPS

Sops is indeed a short form of Secret Operations.

 

What is SOPS and how can we use it

SOPS is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with the following key mechanisms

• AWS KMS
• GCP KMS
• Azure Key Vault
• age,
• PGP. (demo)

Not just these files SOPS can encrypt various other formats also.

When you are using SOPS to encrypt the YAML, JSON, ENV and INI formats - SOPS smartly can encrypt only the values.

In the following diagram, we used the AWS KMS key to encrypt a file test.yaml with some username and password

You can see the output is encrypted with key-related information after executing SOPS -e command.

If you look closely, you can see the keys are untouched only values are encrypted.

With a Single command I was able to encrypt these files in an instant and using the AWS KMS key ensures that only selected IAM users can leverage this KEY

sops -e – in-place – kms "arn:aws:kms:us-east-1:<ACCOUNT ID>:key/mrk-KEYFILENAME" test.yaml

 

AWS KMS and IAM with SOPS

SOPS can use various types of secret encryption mechanisms like PGP, AGE etc where we need to create a Key ourselves and retain it, store it, and share it to be able to decrypt the encrypted files.

Tools such as git-crypt have the same problem - where you have to manage the keys securely.

If the key is lost or tampered the security of the encrypted files would be compromised

Using KMS and IAM brings the perfect balance of Authorization and Authentication in using SOPS at your Organization

With IAM - you ensure only the intended engineers are onboarded to the AWS infrastructure as well and you also define who can use this KMS Key to encrypt and decrypt the files

With KMS Key - You ensure there is no Silos involved and a Single of point of failure such as the Key getting lost or tampered.

As you see in the preceding diagram, When you are using SOPS with AWS KMS two actions happen simultaneously

1. With your IAM permission, the KMS key is being fetched
2. Once the KMS Key is retrieved, SOPS use it to perform the encryption and/or decryption

 

Encrypt and Decrypt files with SOPS - using AWS KMS

Now we have clarity on how we are going to use AWS KMS with SOPS, These are the steps to enable SOPS with AWS KMS encryption

Tasks at the AWS console - require admin or elevated privileges

The following steps have to be done on the aws console. To perform these steps you might need admin or elevated privileges

1. Create an AWS account with the following IAM permissions

   Encrypt, Decrypt, ReEncrypt, GenerateDataKey, DescribeKey

2. Get the Access Key ID and Secret of the user created - needed to configure AWS CLI
3. Now install and configure AWS CLI with the access key and secret - we got from the previous step ( the values given below are just placeholder)

   ➜ ~ aws configure
   AWS Access Key ID [None]: AKIAS790KQGK63WUK6T5
   AWS Secret Access Key [None]: kkQEiBjJSKrDkWBLO9G/JJKQWIOKL/CpHjMGyoiJWW
   Default region name [None]: us-east-1
   Default output format [None]:

4. Create a Symmetric KMS key - for Encryption and Decryption
5. Add Key administrators and key users - administrators have full access to keys so keep them limited and users are those who can use the key for encryption and decryption
   
   
6. Add the user created in Step 1, either as a key administrator or as a key user - based on the requirement. in most cases key user is sufficient
7. Complete the KMS key creation.

Installation of SOPS in your local PC

Now we have done all the changes required on the AWS front and now it is time to put them into use.

Our first requirement is to install the SOPS on our local PC

Choose the rightful binaries for your PC and install it - from this GitHub link

 

Using SOPS with KMS - Encryption and Decryption example

With the AWS CLI installed and configured, you will be able to use the SOPS to encrypt and decrypt the files

Here is the test Yaml file we have created for testing.

~|⇒ cat test.yml
username: sarav
password: secretpassword

Now let us run the encryption command with the KMS key ARN as follows (you need to replace the ARN with yours)

sops -e – in-place – kms "arn:aws:kms:us-east-1:<AWS-ACCOUNT-ID>:key/mrk-<KEY-ID>" test.yml

Now the file is encrypted, To validate the file is encrypted and it has used the KMS key we provided just open the file and you will see some content like this

~|⇒ cat test.yml
username: ENC[AES256_GCM,data:yTwd8V4=,iv:CDUoeilD7K/oLwOHYYxotRM5NaoHb0ROuHoH57bEBlc=,tag:maP1uemDnm4PB3S/nf0jog==,type:str]
password: ENC[AES256_GCM,data:YLk9dYVzBxOjI5ieU30=,iv:l1KoEH02JnGAHeLEkaE6JLP3DVfVJ6/7Uou/SOML2fQ=,tag:/l5YUekg7tdmSIHuAdnJPg==,type:str]
sops:
kms:
- arn: arn:aws:kms:us-east-1:<AWS-ACCOUNT-ID>:key/mrk-<KMS-KEY-ID>
created_at: "2023-09-10T11:35:49Z"
enc: AQICAHhsV+xo/bxJefOk1Gz+1QoGeaNwiYhZtac4EKmVVKaU3AE/ZSl3S6n50iIAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMIJVfLxyEWr0Qf3KtAgEQgDuNpkxS1PDpe9DlsFtaHVs7LJ5sKiYu5FIjaQITtA/xqgACKl55nwlDviBbs/6fxiTWldCl939oVHWuQA==
aws_profile: ""
gcp_kms: []
azure_kv: []
hc_vault: []
age: []
lastmodified: "2023-09-10T11:35:50Z"
mac: ENC[AES256_GCM,data:Yo1C7bo/roL+X925EHKbNSS5OW/M9X6FEp4Lt7YtlrV7L0rt9k8YhqqnmecvQN0BquSqA4pPZhT0Rt3JaGiVjjD9yKgKN7LbGVAfd8mE2EUdnJL+Y9cNXeXLx/bAUCq3/jgnPAXbLhpjjS8fwbM3z1nx+uZfdLFA8=,iv:ppxgtxNdYQ7uNHkV4dWMCqAgmO0R+DPdHUkUoulmHGs=,tag:C/WR1zicupSPMpwbLyU65A==,type:str]
pgp: []
unencrypted_suffix: _unencrypted
version: 3.7.3

You can see which mechanism was used to encrypt the file along with the key and hash details

Now you can decrypt the file to view the encrypted content using the sops -d command

sops -d – in-place – kms "arn:aws:kms:us-east-1:<AWS-ACCOUNT-ID>:key/mrk-<KMS-KEY-ID>" test.yml

 

Here are all the commands and their outputs as executed on my end

That's how SOPS can be used with AWS KMS for securing the files - encryption and decryption.

 

Thanks
Sarav AK

---