How to Use Pandas in AWS Lambda

Prepare Lambda layers using Docker and your favourites python packages

Maybe your function will run from there. (Photo by Christina Morillo on pexels.com)

he idea to write this article happened when I had to personally tackle this issue and I did not find any helpful resource.

This article is short and is meant to be short. Lambda functions are made to make you save time, so this article is.

Sometimes tutorials are too long, sometimes they just show a particular use case which you don’t care about, something it’s just written so badly, however, after the need to create a microservice triggered by an API to process some data the struggle started. But chill, I have an answer!

What you’ll need:

  • Docker
  • A Bash terminal

Let’s get started:

First, you need a requirements.txt file with the packages you will need:

pandaspytznumpy

Second step: we will pull the python image from Docker by using this script named get_layer_packages.sh :

#!/bin/bashexport PKG_DIR="python"
rm -rf ${PKG_DIR} && mkdir -p ${PKG_DIR}
docker run --rm -v $(pwd):/foo -w /foo lambci/lambda:build-python3.6 \
pip3 install -r requirements.txt -t ${PKG_DIR}

Of course, change the python version based on what you need and on what Lambda is compatible with.

At this point, we create the lambda_function.py file, which will be run by Lambda (this is an example code):

import pandas as pddef lambda_handler(event, context):d = {‘col1’: [1, 2], ‘col2’: [3, 4]}df = pd.DataFrame(data=d)return df.to_json()

Now we need to give permissions to our get_layer_packages.sh , open the folder you are working with a Bash terminal and run:

chmod +x get_layer_packages.sh

then:

./get_layer_packages.sh

This operation can take up to a few minutes, depending on if you have the Python image on your local machine and on your internet connection.

Now move the lambda_function.py into the pyhton folder created by the script we just launched and cd into the python directory

mv lambda_function.py python && cd python

Zip the folder:

zip -r pandas-lambda.zip .

At this point your .zip layer is ready, now go to your Lambda Console AWS:

In your function page, under “Function code”, from the dropdown menu choose “Upload a .zip file” and upload the .zip folder we created.

You can now test your function by creating a test, leave the default values and give it a name, then click on test:

Success!

Here you are now you can use your favourite pandas methods with Lambda!

You can, of course, add your favourite layers to trigger your function, for example, an API Gateway to turn your function into a REST API.

Business-oriented Data Scientist, full-stack developer in the free time. Fascinated by everything that merge business with tech.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store