Backup RDS database snapshots to S3 | AWS Architecture

Updated on:

Relation Database Service (RDS) is a managed database option provided by AWS. It is a great option for those who want to use a relational database without having to manage the underlying infrastructure. RDS provides a number of database engines to choose from including MySQL, PostgreSQL, Oracle, and SQL Server. RDS also provides a number of features including automated backups, point-in-time recovery, and read replicas.

When we talk about backups in RDS, we are talking about two different types of backups. The first is automated backups. Automated backups are enabled during the creation (optionally) and are stored in S3. Users can configure the retention period for automated backups and the window in which the backup occurs.

RDS backup to S3

The second type of backup is a manual snapshot. Manual snapshots are user-initiated and are also stored in S3. Both types of backups are stored in S3 and are accessible via the AWS console or the AWS CLI.

In this article, we will look at how to back up RDS databases to S3 using Boto. Boto is the AWS SDK for Python. It allows you to write Python scripts to interact with AWS resources. We will use Boto to connect to AWS and create a database snapshot. We will then copy the snapshot to S3 and delete the snapshot.

In other words, we are going to export the Snapshot to S3.

 

 

RDS Backup to S3 - An opensource tool/script

I have created a simple tool that will allow you to export the snapshot to S3. The tool is written in Python and uses the Boto SDK to connect to AWS.

The script does the following tasks

  1. Connects to AWS using Boto
  2. Creates a snapshot of the database
  3. Copies the snapshot to S3
  4. Deletes the snapshot once the export is completed

 

How to Download and Use this Tool

  1. Clone git repository
  2.  git clone https://github.com/AKSarav/RDSbackupS3.git
  3. Create a virtual environment 
     python3 -m venv venv
  4. Install the requirements using the following command 
     pip install -r requirements.txt
  5. Run the script 
     python3 rds_backup_to_s3.py [--instance INSTANCE] [--region REGION] [--bucket BUCKET]
[--prefix PREFIX] [--role ROLE] [--kms KMS]
  6. Check the S3 bucket for the snapshot

 

Prerequisites

The script expects the following arguments.

  • --instance RDS instance name to backup*
  • --region AWS region of RDS instance
  • --bucket S3 bucket name to copy the snapshot to
  • --prefix S3 prefix ( folder ) to copy the snapshot to
  • --role IAM role to use to copy the snapshot to S3
  • --kms  key to use to copy the snapshot to S3

 

Source code

For your convenience, I have pasted the source code of this script below you can quickly copy and use it

But I do insist you clone the repository to get the latest version and not miss any future updates.

import boto3
import datetime
import sys
import time
import argparse


# get the current date and time
now = datetime.datetime.now()

# set the region and create the RDS client
rds = boto3.client('rds', region_name='us-east-1')

if __name__ == '__main__':

    # argparse get inputs
    parser = argparse.ArgumentParser(description='Backup RDS instance')
    parser.add_argument('--instance', help='RDS instance name')
    parser.add_argument('--region', help='AWS region')
    parser.add_argument('--bucket', help='S3 bucket name')
    parser.add_argument('--prefix', help='S3 prefix')
    parser.add_argument('--role', help='IAM role')
    parser.add_argument('--kms', help='KMS key')
    args = parser.parse_args()

    # if args are not passed, throw error
    if None in (args.instance, args.region, args.bucket, args.prefix, args.role, args.kms):
        print('Usage: python backup-rds.py [instance name] [region] [bucket] [prefix] [role] [kms]')
        sys.exit(1)

    # Display all the values given
    print("-"*50)
    print("RDSbackupS3 | Version: 0.0.1 | Github: ")
    print("-"*50)
    for arg in vars(args):
        print("{:<10}: '{}'".format(arg, getattr(args, arg)))
    print("-"*50)


    instance_name = args.instance

    # get the instance id from the instance name
    response = rds.describe_db_instances(DBInstanceIdentifier=instance_name)

    if not response['DBInstances']:
        print('No instance found with name: ' + instance_name)
        sys.exit(1)
        
    instance_id = response['DBInstances'][0]['DBInstanceIdentifier']

    # create the snapshot
    year=str(now.year)
    month=str(now.month)
    day=str(now.day)

    snapshot_name = instance_name + '-manual-' + now.strftime('%Y-%m-%d-%H-%M')
    print('Creating snapshot: ' + snapshot_name)
    response = rds.create_db_snapshot(
        DBSnapshotIdentifier=snapshot_name,
        DBInstanceIdentifier=instance_id,
        Tags=[
            {
                'Key': "BackupDateTime",
                'Value': now.strftime('%Y-%m-%d-%H-%M'),
            },
            {
                'Key': "BackupType",
                'Value': "Manual",
            },
        ]
    )

    if response['ResponseMetadata']['HTTPStatusCode'] != 200:
        print('Error creating snapshot')
        sys.exit(1)

    # print the snapshot name
    print("Snapshot Initiated successfully: "+snapshot_name)  

    # wait for the snapshot to be created
    waiter = rds.get_waiter('db_snapshot_completed')  

    print('Waiting for snapshot to complete')
    waiter.wait(
        DBSnapshotIdentifier=snapshot_name,
        WaiterConfig={
            'Delay': 30,
            'MaxAttempts': 60
        }
    )

    # print the snapshot status
    response = rds.describe_db_snapshots(DBSnapshotIdentifier=snapshot_name)
    snapshot_status = response['DBSnapshots'][0]['Status']
    print('Snapshot status: ' + snapshot_status)

    # if the snapshot fails, print the status message
    if snapshot_status == 'failed':
        print('Snapshot failed: ' + response['DBSnapshots'][0]['StatusMessage'])
        sys.exit(1)

    # if the snapshot succeeds, print the arn
    print('Snapshot created with arn: ' + response['DBSnapshots'][0]['DBSnapshotArn'])


    # Export the Snapshots to S3
    export_response=rds.start_export_task(
        ExportTaskIdentifier='export-'+snapshot_name,
        SourceArn=response['DBSnapshots'][0]['DBSnapshotArn'],
        S3BucketName=args.bucket,
        IamRoleArn=args.role,
        KmsKeyId=args.kms,
        S3Prefix=args.prefix,
    )

    # print the export status
    response = rds.describe_export_tasks(ExportTaskIdentifier='export-'+snapshot_name)
    export_status = response['ExportTasks'][0]['Status']
    print('Export status: ' + export_status)

    export_status = export_status.lower()

    # wait until the export status changes to 'completed'
    while export_status == 'in_progress' or export_status == 'starting':
        i = 0
        response = rds.describe_export_tasks(ExportTaskIdentifier='export-'+snapshot_name)
        export_status = response['ExportTasks'][0]['Status']
        print('Export status: ' + export_status)
        time.sleep(30)
        i = i + 1

        # Lowercase the export status for comparison
        export_status = export_status.lower()

        # break if no of attempts are more than 30 - 15 minutes
        if i > 30:
            break
        if export_status == 'failed' or export_status == 'failed' or export_status == 'canceled':
            break


    if export_status == 'failed':
        print('Export failed: ' + str(response))
        sys.exit(1)
    
    elif export_status == 'canceled':
        print('Export canceled: ' + str(response))
        sys.exit(1)

    elif export_status == 'complete':
        print('Export completed Successfully: ' + str(response))

    else:
        print('Unhandled Export status: ' + export_status)
        print('Response: ' + str(response))
        print('Not removing the snapshot as the export status is not complete')
        sys.exit(1)

    # Delete the snapshot
    print('Deleting snapshot: ' + snapshot_name)
    response = rds.delete_db_snapshot(
        DBSnapshotIdentifier=snapshot_name
    )

    if response['ResponseMetadata']['HTTPStatusCode'] != 200:
        print('Error deleting snapshot')
        sys.exit(1)

    # print the snapshot name
    print("Snapshot deleted successfully: "+snapshot_name)

 

Sample Output

Here is the execution and the output of this script. Some confidential information are obfuscated

# python3 backup-rds.py  – instance mysqlproddb – region us-east-1 – bucket gritfy-backup-drive – prefix databases/mysql/2023/06/18 – kms arn:aws:kms:us-east-1:XXXXXXXXXXX:key/zzzzzzz-1313-4091-xxxx-z18c81001zz – role arn:aws:iam::XXXXXXXXXXX:role/rds-s3-export-support-role
--------------------------------------------------
RDSbackupS3 | Version: 0.0.1 | Github: https://github.com/AKSarav/RDSbackupS3
--------------------------------------------------
instance  : 'mysqlproddb'
region    : 'us-east-1'
bucket    : 'gritfy-backup-drive'
prefix    : 'databases/mysql/2023/06/18'
role      : 'arn:aws:iam::XXXXXXXXXXX:role/rds-s3-export-support-role'
kms       : 'arn:aws:kms:us-east-1:XXXXXXXXXXX:key/zzzzzzz-1313-4091-xxxx-z18c81001zz'
--------------------------------------------------
Creating snapshot: mysqlproddb-manual-2023-06-18-21-00
Snapshot Initiated successfully: mysqlproddb-manual-2023-06-18-21-00
Waiting for snapshot to complete
Snapshot status: available
Snapshot created with arn: arn:aws:rds:us-east-1:XXXXXXXXXXX:snapshot:mysqlproddb-manual-2023-06-18-21-00
Export status: STARTING
Export status: STARTING
Export status: STARTING
Export status: STARTING
Export status: STARTING
Export status: STARTING
Export status: INPROGRESS
Export status: COMPLETE
Export completed Successfully:
Deleting snapshot: mysqlproddb-manual-2023-06-18-21-00
Snapshot deleted successfully: mysqlproddb-prime-manual-2023-06-18-21-00

 

Conclusion

Hope this tool helps you to export your RDS snapshot to S3. If you have any questions or comments please let us know in the comments section

 

Further References

  1. AWS RDS Documentation
  2. AWS RDS S3 Export Documentation
  3. Boto3 RDS Documentation

Cheers
Sarav AK

Follow me on Linkedin My Profile
Follow DevopsJunction onFacebook orTwitter
For more practical videos and tutorials. Subscribe to our channel

Buy Me a Coffee at ko-fi.com

Signup for Exclusive "Subscriber-only" Content

More from Middleware Inventory

  • How to Apply Pending Modifications in RDS - AWS | Devops Junction

    This short article lets us see how to apply pending modifications to AWS RDS. Sometimes when we modify the RDS database instance for reasons such as Changing the Instance type Changing Parameter Group Scale up the Disk Size allocated and the maximum disk Size Upgrading the DB Version (major/minor) Or…

  • aws s3 cp
    AWS S3 CP Examples - How to Copy Files with S3 CLI | Devops Junction

    AWS CLI has become a life saver when you want to manage your AWS infrastructure efficiently. This also provides lot of possibilities for Automation and reduce the number of times that you have to login to AWS Management console. If you are new to AWS CLI. Please consider reading this…

  • Ansible S3
    How to use ansible with S3 - Ansible aws_s3 examples | Devops Junction

    S3 has become the cheap and best object storage and it is used widely across enterprises and products, to securely save and access data over the internet. We have various ways to interact with the S3 bucket,  to upload, download and copy objects back and forth to S3. SDKs (…

  • Oracle DB backup S3
    Automated Oracle Database Backup to S3

    The Beautiful thing about Digital world is that we have a Ctrl+z button, snapshots and backups to go back in time and correct our mistakes, which is not possible in our real life. Enough with a philosophy. I would not have to tell you how crucial backups are and when…

  • aws s3 sync
    AWS S3 Sync Examples - Sync S3 buckets AWS CLI

    In this article, we are going to see how to use the powerful aws s3 sync CLI command. In the Linux world, if we want to compare two directories and copy the files on the same machine or between two machines over the remote (ssh) we use rsync Those who…