How to Use Pandas in AWS Lambda

Prepare Lambda layers using Docker and your favourites python packages

Maybe your function will run from there. (Photo by Christina Morillo on pexels.com)

The idea to write this article happened when I had to personally tackle this issue and I did not find any helpful resource.

This article is short and is meant to be short. Lambda functions are made to make you save time, so this article is.

Sometimes tutorials are too long, sometimes they just show a particular use case which you don’t care about, something it’s just written so badly, however, after the need to create a microservice triggered by an API to process some data the struggle started. But chill, I have an answer!

What you’ll need:

• Docker
• A Bash terminal

Let’s get started:

First, you need a requirements.txt file with the packages you will need:

pandas
pytz
numpy

Second step: we will pull the python image from Docker by using this script named get_layer_packages.sh :

#!/bin/bash
export PKG_DIR="python"
rm -rf ${PKG_DIR} && mkdir -p ${PKG_DIR}
docker run --rm -v $(pwd):/foo -w /foo lambci/lambda:build-python3.6 \
pip3 install -r requirements.txt -t ${PKG_DIR}

Of course, change the python version based on what you need and on what Lambda is compatible with.

At this point, we create the lambda_function.py file, which will be run by Lambda (this is an example code):

import pandas as pd
def lambda_handler(event, context):
d = {‘col1’: [1, 2], ‘col2’: [3, 4]}
df = pd.DataFrame(data=d)
return df.to_json()

Now we need to give permissions to our get_layer_packages.sh , open the folder you are working with a Bash terminal and run:

chmod +x get_layer_packages.sh

then:

./get_layer_packages.sh

This operation can take up to a few minutes, depending on if you have the Python image on your local machine and on your internet connection.

Now move the lambda_function.py into the pyhton folder created by the script we just launched and cd into the python directory

mv lambda_function.py python && cd python

Zip the folder:

zip -r pandas-lambda.zip .

At this point your .zip layer is ready, now go to your Lambda Console AWS:

In your function page, under “Function code”, from the dropdown menu choose “Upload a .zip file” and upload the .zip folder we created.

You can now test your function by creating a test, leave the default values and give it a name, then click on test:

Success!

Here you are now you can use your favourite pandas methods with Lambda!

You can, of course, add your favourite layers to trigger your function, for example, an API Gateway to turn your function into a REST API.

I have a newsletter 📩.
Every week I’ll send you a brief findings of articles, links, tutorials, and cool things that caught my attention. If tis sounds cool to you subscribe.
That means a lot for me.

Mills form