How to limit AWS Lambda budget on test environment

We can prevent spending thousands of dollars caused by a mistake using AWS Budget + SNS + Lambda

Guy looking at the money flying away

Unfortunately, there is no ready solution available on AWS to prevent that we lose a lot of money on Lambdas unintentionally. Like in this case: https://medium.com/@asankha/lambda-programming-errors-that-could-cost-you-thousands-of-dollars-a-day-265dfac354f

After facing some similar issues where I was working, I came up with a workaround that can be useful for a lot of other users. However, I recommend it to development or sandbox environments only, because it will force all lambdas in the account to stop working. After the bleeding has stopped, you can go back on the lambdas and reconfigure the concurrencies according to your needs.

The solution

You probably already know that AWS has a solution called AWS Budgets where you can configure the amount of money you are planning to spend by month and set some thresholds and alarms. These alarms can send emails or send a message to Amazon SNS.

AWS Bugdget console

That’s the key to the solution because Lambdas can be triggered by SNS and we can use a function to set the concurrency of the other lambdas to zero (and make them stop running) using aws cli.

To sum up:

Creating the Lambda

Before jumping into the code, we need to be able to use aws cli inside the Lambda, and we can do that by creating a layer with awscli and the required libraries.
Run the commands below to create the zip file using Linux:

mkdir awscli-virtualenv
virtualenv awscli-virtualenv
cd awscli-virtualenv/bin/
source activate
pip install awscli
# replace the first line of aws file with #!/var/lang/bin/python
sed -i "1s/.*/\#\!\/var\/lang\/bin\/python/" aws
deactivate
cd ../..
mkdir awscli-lambda-layer
cd awscli-lambda-layer
cp ../awscli-virtualenv/bin/aws .
cp -r ../awscli-virtualenv/lib/python3.6/site-packages/* .
zip -r ../awscli-lambda-layer.zip *

If you want to use a different version of Python, replace the python3.6 by your version.

Upload the zip into a Lambda Layer:

Create an IAM role for this Lambda function and remember that you need to create a specific policy for it containing the PutFunctionConcurrency permission to all the Lambda resources. Otherwise, you won’t be able to set the concurrency.

Create the Lambda function with the runtime set to the Python version you are using and configure the layer you have created before.

The basic logic of the function is:

  • Copy the aws commands to /tmp and change the permission so we can execute them.
  • Get all the function names
  • Loop through the function names and set the concurrency to zero except for the function itself (setConcurrency lambda function).

The lambda code I used was the following:

import subprocess
import logging
import os
from distutils.dir_util import copy_tree
import json
def lambda_handler(event, context): # copy the files so I could be able to change the permission and execute
copy_tree('/opt/', '/tmp/')
os.chmod('/tmp/aws', 0O775)
# list all the function to get the names
cmdline = ['/tmp/aws', 'lambda', 'list-functions']
result = subprocess.run(cmdline, shell=False, stdout=subprocess.PIPE)
json_result = json.loads(result.stdout)
# loop through the functions and set the concurrency to zero
for func in json_result['Functions']:
if func['FunctionName']=='setConcurrency': # use this function name instead of setConcurrency
print("Stopped ",func['FunctionName'])
cmdline = ['/tmp/aws', 'lambda', 'put-function-concurrency', ' - function-name', func['FunctionName'], ' - reserved-concurrent-executions', '0']
result = subprocess.run(cmdline, shell=False, stdout=subprocess.PIPE)
print(result)

Final configurations

Now that we have the Lambda done, we need to create an SNS topic.

Besides the standards configuration, we need to add in the Access Policy the following code so AWS Budget can send the alerts to it:

    {
"Sid": "ExampleSid123456789012",
"Effect": "Allow",
"Principal": {
"Service": "budgets.amazonaws.com"
},
"Action": "SNS:Publish",
"Resource": "arn:aws:sns:XX-XXXX-1:XXXXXXXXXX:stopLambdas"
}
SNS Topic configuration

Configure the SNS Topic as a trigger on the Lambda.

Lambda trigger: SNS

Set the SNS Topic to receive the alarms from AWS Budget.

AWS Budget Alert

It’s done!

Now we can test whatever we want regarding Lambdas on our development environment and we won’t be surprised by an enormous bill! So choose wisely your budget and let’s get this done!

Data Scientist

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store