How to Install Pip Packages in AWS Lambda Using Docker and ECR

 
 

Deploying AWS Lambda functions often involves packaging various dependencies. Managing these dependencies directly within Lambda can be cumbersome and error-prone. Using Docker and Amazon Elastic Container Registry (ECR) can streamline this process, ensuring consistency and reliability in your deployments. In this blog, we will walk through how to install pip packages in Lambda packages using Docker and ECR.

AWS Lambda is a powerful service that allows you to run code without provisioning or managing servers. However, managing dependencies for your Lambda functions can be challenging, especially when dealing with complex Python packages. By leveraging Docker, you can package your Lambda function and its dependencies in a container, which ensures a consistent runtime environment. Additionally, using ECR to store your Docker images simplifies the deployment process.

Before reading the remainder, be sure to subscribe and support the channel if you have not!
Subscribe:
Support:
Hire me at UpWork to build your IoT projects:

Prerequisites

Before we get started, make sure you have the following installed and configured:

  • AWS CLI
  • Docker

Step 1: Install AWS CLI and Configure

First, install the AWS CLI and configure it with your credentials.

Installation

  • On Windows: Download the AWS CLI MSI installer from the official website. Run the installer and follow the on-screen instructions.
  • On macOS: Use Homebrew to install the AWS CLI:
brew install awscli
  • On Linux: Use the following commands:
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

Configure the AWS CLI by running:

aws configure

Enter your AWS Access Key ID, Secret Access Key, Default region name, and Default output format when prompted.

Step 2: Create an ECR Repository

If you haven't already created an ECR repository, do so by running:


aws ecr create-repository --repository-name my-lambda-repo --region us-east-1

This command creates a new ECR repository where you'll store your Docker images.

Step 3: Authenticate Docker to ECR

Authenticate your Docker CLI to your ECR registry, make sure you are signed into Docker desktop and have it running on your computer:


aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com

Replace <aws_account_id> with your actual AWS account ID.

Step 4: Create a .dockerignore File

Create a .dockerignore file in your project directory to exclude unnecessary files from the Docker build context:

__pycache__
.git
tests
*.md
*.txt

Step 5: Create a Dockerfile

Create a Dockerfile in your project directory. This file defines the environment in which your Lambda function will run and specifies the necessary dependencies.

# Builder stage - you can use other AWS runtimes and python versions
FROM public.ecr.aws/lambda/python:3.12-x86_64 as builder

# Install Python packages (replace as needed)
RUN pip install --no-cache-dir requests numpy pandas --target "/var/task"

# Final stage
FROM public.ecr.aws/lambda/python:3.12-x86_64

# Copy necessary files from the builder stage - add whatever code you need
COPY --from=builder /var/task /var/task
COPY handler.py ./

# Set the CMD to your handler
CMD ["handler.lambda_handler"]

Step 6: Build the Docker Image

Build the Docker image using the following command:


docker build -t my-lambda-image .

This command creates a Docker image named my-lambda-image.

Step 7: Tag the Docker Image

Tag the Docker image for pushing to your ECR repository:

docker tag my-lambda-image <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/my-lambda-repo:latest

Step 8: Push the Docker Image to ECR

Push the Docker image to your ECR repository:

docker push <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/my-lambda-repo:latest

Step 9: Update Lambda Function

Finally, update your Lambda function to use the new Docker image. You can do this through the AWS Management Console or using the AWS CLI:

aws lambda update-function-code --function-name my-lambda-function --image-uri <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/my-lambda-repo:latest

Replace my-lambda-function with the name of your Lambda function.

Alternatively, you can update your Lambda function through the AWS Management Console:

  1. Navigate to the AWS Lambda console.
  2. Select your Lambda function from the list.
  3. In the "Code" tab, choose "Image" as the code source.
  4. Select the ECR image you pushed earlier.
  5. Save the changes.

Creating a Lambda Function from Scratch with a Docker Image

You can also create a new AWS Lambda function from scratch using a Docker image. This is useful when you are starting a new project or want to create a fresh Lambda function with a specific Docker environment. Here's how to do it through the AWS Management Console:

  1. Navigate to the AWS Lambda console.
  2. Click on "Create function."
  3. Select "Container image" as the code source.
  4. Specify your container image details by selecting the ECR repository and image tag you pushed earlier.
  5. Configure the remaining settings (function name, role, etc.).
  6. Click on "Create function."

This process sets up a new Lambda function that uses your Docker image as its execution environment, ensuring all dependencies and configurations are included from the start.

Conclusion

Using Docker and ECR to manage dependencies for your AWS Lambda functions significantly streamlines the deployment process. This method ensures that your functions have a consistent runtime environment, reducing the potential for errors caused by missing or incompatible dependencies.

By following the steps outlined in this tutorial, you can efficiently package, test, and deploy your Lambda functions with all necessary dependencies included. This approach not only simplifies the deployment process but also enhances the reliability and performance of your serverless applications.

Remember to subscribe to the channel and support our content creation efforts:

Create a free account to access full content.

All access to code and resources on ShillehTek.

Signup Now

Already a member? Sign In

Explore More on Our Blog

Interfacing the MPU6050 with Raspberry Pi Pico W in C++

Interfacing the MPU6050 with Raspberry Pi Pico W in C++

Interface with the MPU6050 using the Raspberry Pi Pico W in C++.

How to Write your First C++ Program on the Raspberry Pi Pico W

How to Write your First C++ Program on the Raspberry Pi Pico W

Write your first C++ Program on the Pico W in a few simple steps.

How to Use ThingSpeak with the Raspberry Pi Pico W

How to Use ThingSpeak with the Raspberry Pi Pico W

Learn how to create a real-time environmental monitoring system with the Raspberry Pi Pico W and ThingSpeak!

How to Use ADS1115 with the Raspberry Pi (Part 1)

How to Use ADS1115 with the Raspberry Pi (Part 1)

Discover how to expand your Raspberry Pi projects by integrating the ADS1115 ADC for precise analog signal reading....

How to Install Pip Packages in AWS Lambda Using Docker and ECR

How to Install Pip Packages in AWS Lambda Using Docker and ECR

Learn how to streamline AWS Lambda deployments by using Docker and Amazon Elastic Container Registry (ECR) to package...

Create Tabular Product Descriptions on Your Shopify Store

Create Tabular Product Descriptions on Your Shopify Store

Enhance your Shopify store's product pages with our comprehensive guide on implementing tabular descriptions. Learn how to add a...

SSH Into Raspberry Pi with Tailscale VPN

SSH Into Raspberry Pi with Tailscale VPN

Effortlessly access and manage your Raspberry Pi from anywhere using Tailscale's secure mesh VPN.

Send Email with Lua and the ESP32

Send Email with Lua and the ESP32

In this tutorial, we delve into sending emails with the ESP32-S3 using Lua, focusing on the Xedge IDE's built-in SMTP...

How to Code with Lua on ESP32 with XEdge32

How to Code with Lua on ESP32 with XEdge32

Learn how to set up Xedge32 and start coding on the ESP32-S3 with Lua programming!

Stream Audio From Raspberry Pi to Local Computer

Stream Audio From Raspberry Pi to Local Computer

Discover the simplicity of streaming live audio directly from a USB microphone connected to your Raspberry Pi to...

Back to blog

Leave a comment

Please note, comments need to be approved before they are published.