Schedule Amazon RDS stop and start using AWS Lambda

Amazon Relational Database Service (Amazon RDS) makes it easy to set up, operate, and scale a relational database in the cloud. Traditional relational databases require time spent on capacity planning, maintenance, backup, and recovery; a substantial amount of a database administrator’s time is lost to these tasks. Amazon RDS helps DBAs to focus on other important tasks that add value to the organization by automating most routine tasks.

In a typical development environment, dev and test databases are mostly utilized for 8 hours a day and sit idle when not in use. However, the databases are billed for the compute and storage costs during this idle time. To reduce the overall cost, Amazon RDS allows instances to be stopped temporarily. While the instance is stopped, you’re charged for storage and backups, but not for the DB instance hours. Please note that a stopped instance will automatically be started after 7 days.

This post presents a solution using AWS Lambda and Amazon EventBridge that allows you to schedule a Lambda function to stop and start the idle databases with specific tags to save on compute costs. The second post presents a solution that accomplishes stop and start of the idle Amazon RDS databases using AWS Systems Manager.

Solution overview

AWS Lambda is a compute service that lets you run code without managing any servers. You don’t have to worry about provisioning servers, configuring the operating systems, installing applications, and so on.

Amazon EventBridge uses simple rules in which you can create events and assign specific actions in response to them.

Amazon RDS provides different instance types optimized to fit different relational database use cases. The compute costs for most of these instances are billed on an hourly basis. Database instances provisioned in dev or test environments that remain idle for extended periods of time (due to the intermittent nature of the tasks performed on them) can be automatically stopped on a nightly basis and started before business hours by using Lambda functions and EventBridge rules. In this proposed solution, we use a Lambda function to store the code that stops or starts all the RDS instances that have a tag DEV-TEST, and use EventBridge Events rules to trigger the Lambda functions.

The following diagram illustrates our solution architecture.

To implement the solution, you complete the following high-level steps:

1. Provision the following resources:
   1. Tags for your RDS instances.
   2. An AWS Identity and Access Management (IAM) policy and role for your Lambda.
   3. Two Lambda functions to stop and start your databases.
2. Create Amazon EventBridge rules to trigger the Lambda functions as needed.

Prerequisites

To follow the steps in this post, you need the following:

• An AWS account with administrator access to Amazon RDS.
• An RDS instance that you want to shut down and start on a schedule.

Provision the resources

The following steps explain how to create tags, an IAM policy and role for Lambda, and the Lambda functions that we schedule to stop or start the databases.

Create tags

You can assign tags while creating a DB instance or by modifying the instance after it’s created. The following steps walk you through assigning tags for a scheduled stop and start:

1. On the Amazon RDS console, choose a database and the instance within that database that you want to add tags to.
2. On the Tags tab underneath the instance details, choose Add tags.
   
3. For Tag key, enter DEV-TEST.
4. For Value, enter Auto-Shutdown.
5. Choose Add.
   

You can now see the added tags on the Tags tab.

Create an IAM policy and role for Lambda

We now create an IAM policy and role for Lambda to start and stop the instances.

1. On the IAM console, under Access management in the navigation pane, choose Policies.
2. Choose Create policy.
   
3. On the JSON tab, enter the following policy code:

   {
     "Version": "2012-10-17",
     "Statement": [
         {
             "Sid": "VisualEditor0",
             "Effect": "Allow",
             "Action": [
                 "rds:DescribeDBClusterParameters",
                 "rds:StartDBCluster",
                 "rds:StopDBCluster",
                 "rds:DescribeDBEngineVersions",
                 "rds:DescribeGlobalClusters",
                 "rds:DescribePendingMaintenanceActions",
                 "rds:DescribeDBLogFiles",
                 "rds:StopDBInstance",
                 "rds:StartDBInstance",
                 "rds:DescribeReservedDBInstancesOfferings",
                 "rds:DescribeReservedDBInstances",
                 "rds:ListTagsForResource",
                 "rds:DescribeValidDBInstanceModifications",
                 "rds:DescribeDBInstances",
                 "rds:DescribeSourceRegions",
                 "rds:DescribeDBClusterEndpoints",
                 "rds:DescribeDBClusters",
                 "rds:DescribeDBClusterParameterGroups",
                 "rds:DescribeOptionGroups"
             ],
             "Resource": "*"
         }
     ]
   }

4. Choose Review policy.
5. For Name, enter rdsstopstart.
   
6. Choose Create policy.
   We now create the IAM role.
7. In the navigation pane, choose Roles.
8. Choose Create role.
   
9. For Select type of trusted entity, choose AWS service.
10. For Common use cases, choose Lambda.
11. Choose Next: Permissions.
    
12. Search for and select the policy you created (rdsstopstart).
    
13. Choose Next: Tags.
14. In the Tags section, provide your key and value (Optional)
15. Choose Review.
16. For Role name, enter rdsLambda.
17. Review the attached policies and choose Create role.
    

You can now attach this role while creating your Lambda functions.

Create your Lambda function to stop the database

For this post, we create two Lambda functions that can be called to stop and start the databases. We first walk you through creating the stop function.

1. On the Lambda console, choose Functions in the navigation pane.
2. Choose Create function.
   
3. For Function name, enter stoprds.
4. For Runtime, choose Python 3.7.
5. For Execution role, select Use an existing role.
6. For Existing role, choose the role you created (rdsLambda).
   
7. Choose Create function.
8. On the function details page, navigate to the function code.
9. Delete the sample code and enter the following.

   # this Code will help to schedule stop the RDS databasrs using Lambda
   # Yesh 
   # Version -- 2.0
   
   import boto3
   import os
   import sys
   import time
   from datetime import datetime, timezone
   from time import gmtime, strftime
   
   def shut_rds_all():
       region=os.environ['REGION']
       key=os.environ['KEY']
       value=os.environ['VALUE']
   
       
       client = boto3.client('rds', region_name=region)
       response = client.describe_db_instances()
       v_readReplica=[]
       for i in response['DBInstances']:
           readReplica=i['ReadReplicaDBInstanceIdentifiers']
           v_readReplica.extend(readReplica)
       
       for i in response['DBInstances']:
   #The if condition below filters aurora clusters from single instance databases as boto3 commands defer to stop the aurora clusters.
           if i['Engine'] not in ['aurora-mysql','aurora-postgresql']:
   #The if condition below filters Read replicas.
               if i['DBInstanceIdentifier'] not in v_readReplica and len(i['ReadReplicaDBInstanceIdentifiers']) == 0:
                   arn=i['DBInstanceArn']
                   resp2=client.list_tags_for_resource(ResourceName=arn)
   #check if the RDS instance is part of the Auto-Shutdown group.
                   if 0==len(resp2['TagList']):
                       print('DB Instance {0} is not part of autoshutdown'.format(i['DBInstanceIdentifier']))
                   else:
                       for tag in resp2['TagList']:
   #If the tags match, then stop the instances by validating the current status.
                           if tag['Key']==key and tag['Value']==value:
                               if i['DBInstanceStatus'] == 'available':
                                   client.stop_db_instance(DBInstanceIdentifier = i['DBInstanceIdentifier'])
                                   print('stopping DB instance {0}'.format(i['DBInstanceIdentifier']))
                               elif i['DBInstanceStatus'] == 'stopped':
                                   print('DB Instance {0} is already stopped'.format(i['DBInstanceIdentifier']))
                               elif i['DBInstanceStatus']=='starting':
                                   print('DB Instance {0} is in starting state. Please stop the cluster after starting is complete'.format(i['DBInstanceIdentifier']))
                               elif i['DBInstanceStatus']=='stopping':
                                   print('DB Instance {0} is already in stopping state.'.format(i['DBInstanceIdentifier']))
                           elif tag['Key']!=key and tag['Value']!=value:
                               print('DB instance {0} is not part of autoshutdown'.format(i['DBInstanceIdentifier']))
                           elif len(tag['Key']) == 0 or len(tag['Value']) == 0:
                               print('DB Instance {0} is not part of auroShutdown'.format(i['DBInstanceIdentifier']))
               elif i['DBInstanceIdentifier'] in v_readReplica:
                   print('DB Instance {0} is a Read Replica. Cannot shutdown a Read Replica instance'.format(i['DBInstanceIdentifier']))
               else:
                   print('DB Instance {0} has a read replica. Cannot shutdown a database with Read Replica'.format(i['DBInstanceIdentifier']))
   
       response=client.describe_db_clusters()
       for i in response['DBClusters']:
           cluarn=i['DBClusterArn']
           resp2=client.list_tags_for_resource(ResourceName=cluarn)
           if 0==len(resp2['TagList']):
               print('DB Cluster {0} is not part of autoshutdown'.format(i['DBClusterIdentifier']))
           else:
               for tag in resp2['TagList']:
                   if tag['Key']==key and tag['Value']==value:
                       if i['Status'] == 'available':
                           client.stop_db_cluster(DBClusterIdentifier=i['DBClusterIdentifier'])
                           print('stopping DB cluster {0}'.format(i['DBClusterIdentifier']))
                       elif i['Status'] == 'stopped':
                           print('DB Cluster {0} is already stopped'.format(i['DBClusterIdentifier']))
                       elif i['Status']=='starting':
                           print('DB Cluster {0} is in starting state. Please stop the cluster after starting is complete'.format(i['DBClusterIdentifier']))
                       elif i['Status']=='stopping':
                           print('DB Cluster {0} is already in stopping state.'.format(i['DBClusterIdentifier']))
                   elif tag['Key'] != key and tag['Value'] != value:
                       print('DB Cluster {0} is not part of autoshutdown'.format(i['DBClusterIdentifier']))
                   else:
                       print('DB Instance {0} is not part of auroShutdown'.format(i['DBClusterIdentifier']))
   
   def lambda_handler(event, context):
       shut_rds_all()

10. Choose Save.
    The above lambda function needs 3 parameters (REGION, KEY, VALUE) to be passed as environment variables. REGION is where the RDS instances are currently running, KEY and VALUE are the Tags that we have attached for the instances that require auto shutdown in the previous steps. Please note the values that we attached to the RDS instances should match exactly to the environment variables. Please follow the below steps to enter these.
11. Navigate to the ‘Configuration’ tab and choose ‘Environment Variables’. Click on the EDIT and add the Environment Variables as shown below.
    
12. Choose Test to test the function.
    
    The Configure test event page opens.
13. For Event name, enter stop.
14. Choose Create.
    
15. Choose Test again to test the function.

The test stops the database with the tags specified in the function. You can see on the function detail page that the function was successful.

Create your Lambda function to start the database

Repeat the same steps to create the start function, called rdsstart. Use the following code:

# this Code will help to schedule start the RDS databasrs using Lambda
# Yesh 
# Version -- 2.0

import boto3
import os
import sys
import time
from datetime import datetime, timezone
from time import gmtime, strftime

def start_rds_all():
    region=os.environ['REGION']
    key=os.environ['KEY']
    value=os.environ['VALUE']
    client = boto3.client('rds', region_name=region)
    response = client.describe_db_instances()

    v_readReplica=[]
    for i in response['DBInstances']:
        readReplica=i['ReadReplicaDBInstanceIdentifiers']
        v_readReplica.extend(readReplica)
    
    for i in response['DBInstances']:
#The if condition below filters aurora clusters from single instance databases as boto3 commands defer to start the aurora clusters.
        if i['Engine'] not in ['aurora-mysql','aurora-postgresql']:
#The if condition below filters Read replicas.
            if i['DBInstanceIdentifier'] not in v_readReplica and len(i['ReadReplicaDBInstanceIdentifiers']) == 0:
                arn=i['DBInstanceArn']
                resp2=client.list_tags_for_resource(ResourceName=arn)
#check if the RDS instance is part of the Auto-Shutdown group.
                if 0==len(resp2['TagList']):
                    print('DB Instance {0} is not part of autoshutdown'.format(i['DBInstanceIdentifier']))
                else:
                    for tag in resp2['TagList']:
                        if tag['Key']==key and tag['Value']==value:
                            if i['DBInstanceStatus'] == 'available':
                                print('{0} DB instance is already available'.format(i['DBInstanceIdentifier']))
                            elif i['DBInstanceStatus'] == 'stopped':
                                client.start_db_instance(DBInstanceIdentifier = i['DBInstanceIdentifier'])
                                print('Started DB Instance {0}'.format(i['DBInstanceIdentifier']))
                            elif i['DBInstanceStatus']=='starting':
                                print('DB Instance {0} is already in starting state'.format(i['DBInstanceIdentifier']))
                            elif i['DBInstanceStatus']=='stopping':
                                print('DB Instance {0} is in stopping state. Please wait before starting'.format(i['DBInstanceIdentifier']))
                        elif tag['Key']!=key and tag['Value']!=value:
                            print('DB instance {0} is not part of autoshutdown'.format(i['DBInstanceIdentifier']))
                        elif len(tag['Key']) == 0 or len(tag['Value']) == 0:
                            print('DB Instance {0} is not part of autoShutdown'.format(i['DBInstanceIdentifier']))
            elif i['DBInstanceIdentifier'] in v_readReplica:
                print('DB Instance {0} is a Read Replica.'.format(i['DBInstanceIdentifier']))
            else:
                print('DB Instance {0} has a read replica. Cannot shutdown & start a database with Read Replica'.format(i['DBInstanceIdentifier']))

    response=client.describe_db_clusters()
    for i in response['DBClusters']:
        cluarn=i['DBClusterArn']
        resp2=client.list_tags_for_resource(ResourceName=cluarn)
        if 0==len(resp2['TagList']):
            print('DB Cluster {0} is not part of autoshutdown'.format(i['DBClusterIdentifier']))
        else:
            for tag in resp2['TagList']:
                if tag['Key']==key and tag['Value']==value:
                    if i['Status'] == 'available':
                        print('{0} DB Cluster is already available'.format(i['DBClusterIdentifier']))
                    elif i['Status'] == 'stopped':
                        client.start_db_cluster(DBClusterIdentifier=i['DBClusterIdentifier'])
                        print('Started Cluster {0}'.format(i['DBClusterIdentifier']))
                    elif i['Status']=='starting':
                        print('cluster {0} is already in starting state.'.format(i['DBClusterIdentifier']))
                    elif i['Status']=='stopping':
                        print('cluster {0} is in stopping state. Please wait before starting'.format(i['DBClusterIdentifier']))
                elif tag['Key'] != key and tag['Value'] != value:
                    print('DB Cluster {0} is not part of autoshutdown'.format(i['DBClusterIdentifier']))
                else:
                    print('DB Instance {0} is not part of autoShutdown'.format(i['DBClusterIdentifier']))

def lambda_handler(event, context):
    start_rds_all()

The above lambda function needs 3 parameters (REGION, KEY, VALUE) to be passed as environment variables. REGION is where the RDS instances are currently running, KEY and VALUE are the Tags that we have attached for the instances that require auto shutdown in the previous steps. Please note the values that we attached to the RDS instances should match exactly to the environment variables. Please follow the below steps to enter these.

Navigate to the ‘Configuration’ tab and choose ‘Environment Variables’. Click on the EDIT and add the Environment Variables as shown below.

Testing the function should provide the following output.

After you create and test both functions, we can create EventBridge rules to trigger these functions as needed.

Create your Amazon EventBridge Rule

Amazon EventBridge rules trigger the functions we created to either stop or start the tagged database. For this post, we configure them to trigger on a schedule.

1. On the EventBridge console, under Events in the navigation pane, choose Rules. Under the Create rule section, define a name for the rule as shown below.
   
2. Under the Define Pattern section, select Schedule and click on Cron expression and Enter 0 23 ? * MON-FRI * (This cron expression stops databases Monday to Friday at 11:00 PM GMT) as shown below.
   
3. In the Select event bus section, choose AWS default event bus and Enable the rule on the selected event bus (by default they will be enabled).
   
4. In the Select Targets section, choose Lambda Function In the first drop-down box.
5. For Function, choose the stop function you created (stoprds).
   
6. Review the details entered and select Create at the bottom of the page to create the Rule.
   The EventBridge rule now triggers the stoprds Lambda function at the scheduled time.
7. Repeat these steps to create a rule to trigger the rdsstart Lambda function at the preferred scheduled time.

Summary

This post demonstrated how to schedule a Lambda function to stop and start RDS databases in dev and test environments when they’re not in use. The benefits of automating the startup and shutdown of RDS DB instances using Lambda allows organizations to further reduce compute costs and simplify the administration of database environments that don’t need to be running continuously.

We encourage you to try this solution and take advantage of all the benefits of using AWS Lambda and Amazon Relational Database Service. You can also accomplish Amazon RDS Stop and Start using AWS Systems Manager, check out the blog post “Schedule Amazon RDS stop and start using AWS Systems Manager”

Please feel free to reach out with questions or requests in the comments.

About the authors

Yesh Tanamala is a Database Migration Consultant with AWS Professional Services. He works as a database migration specialist to help internal and external Amazon customers move their on-premises database environment to AWS data stores.

Sharath Lingareddy is a Database Architect with the Professional Services team at Amazon Web Services. He has provided solutions using relational databases including Amazon RDS. His focus area is homogeneous and heterogeneous migrations of on-premise databases to Amazon RDS and Aurora PostgreSQL.

Varun Mahajan is a Solutions Architect at Amazon Web Services. He works with enterprise customers helping them align their business goals with the art of the possible using cloud-based technologies. He enjoys working with data and solving problems using the AWS database and analytics portfolio.