How to Install Pip Packages in AWS Lambda Functions (Method 1)

In this tutorial, I'll demonstrate a streamlined method to incorporate pip packages into your AWS Lambda functions by executing the pip install command directly within the Lambda environment. This approach, while not adhering to best practices, is particularly suited for simple projects and packages, offering a pragmatic solution that I frequently employ in my testing environments to significantly save time. The only prerequisite for following along with this guide is to have an AWS account. So, let's dive in and get started.

Before reading the remainder, be sure to subscribe and support the channel if you have not!
Subscribe:
Support:
Hire me at UpWork to build your IoT projects:

Part 1: Script We Want to Run

import json
import jwt

def lambda_handler(event, context):
    secret_key = 'secret-key'
    
    # Data payload you want to encode
    payload = {
        'user_id': 123,
        'email': 'user@example.com'
    }

    try:
        # Encoding the payload with the secret key
        encoded_jwt = jwt.encode(payload, secret_key, algorithm='HS256')
        print(f"Encoded JWT: {encoded_jwt}")

        # Decoding the payload with the secret key
        decoded_jwt = jwt.decode(encoded_jwt, secret_key, algorithms=['HS256'])
        print(f"Decoded JWT: {decoded_jwt}")

        return {
            'statusCode': 200,
            'body': json.dumps({
                'message': 'JWT encoded and decoded successfully!',
                'encoded': encoded_jwt,
                'decoded': decoded_jwt
            })
        }

    except Exception as e:
        print(e)
        return {
            'statusCode': 500,
            'body': json.dumps({
                'message': 'An error occurred',
                'error': str(e)
            })
        }

This script outlines a process for incorporating the pyjwt package into an AWS Lambda function, which is essential for encoding and decoding a payload. Installing this package directly within Lambda encounters challenges due to the environment's compatibility issues. Local installations on your system do not suffice as they might not be compatible with Amazon Linux, which Lambda functions run on. This discrepancy necessitates alternative approaches to ensure the necessary packages are accessible within the Lambda environment.

To circumvent these issues, a practical solution involves leveraging an additional Lambda function specifically for installing required packages into the /tmp directory. This directory is particularly suitable for such operations due to its ephemeral nature, allowing temporary installations without persisting beyond the execution context. The process involves using this auxiliary Lambda function to execute a pip install command for the pyjwt package, subsequently packaging the installed files into a zip archive. This archive is then stored in an Amazon S3 bucket.

Following the storage of the package in S3, the next step is to create a Lambda layer from the uploaded zip file. Layers in AWS Lambda are a powerful feature allowing you to manage and share dependencies across multiple Lambda functions. By adding the pyjwt package as a layer, you effectively make it available to the original Lambda function without directly embedding it within its deployment package. This approach not only simplifies dependency management but also enhances the maintainability of your Lambda functions by segregating the function code from its dependencies.

This method offers a streamlined and efficient means of integrating external packages into Lambda functions, avoiding the complexities and potential incompatibilities associated with direct installations or manual management of wheel files. By leveraging Lambda layers for dependency management, you ensure a more organized and scalable architecture for your serverless applications.

Part 2: Lambda Helper

Let's put our plan into action! Begin by creating a new Lambda function in the Python version corresponding to your target environment. For example, if your desired packages need to be compatible with Python 3.12, ensure your helper Lambda function also uses Python 3.12. It's crucial to maintain consistency in both Python version and architecture (e.g., arm64 vs. x64) across your environments to avoid compatibility issues.

Below is a Python script designed for the new Lambda function. This script automates the process of installing packages, packaging them, and uploading the package to an Amazon S3 bucket:

https://github.com/shillehbean/youtube-p2/blob/main/lambda_pip_helper.py

This script leverages Python's subprocess module to execute pip install commands, directing the installation to the /tmp directory for ephemeral storage. Replace the packages list with any other packages you need, following the same format.

There are some considerations to keep in mind:

  • Resource Limits: If the packages are large, you might encounter memory or storage limitations. To mitigate this, increase the memory, timeout, and ephemeral storage settings in the Lambda function's Configuration tab to their maximum values. However, some packages might still be too large.
  • S3 and IAM Permissions: Ensure the Lambda function has the necessary permissions to write to S3 buckets. This may require modifying the Lambda's execution role in the IAM service to include S3 access. Additionally, create an S3 bucket in the appropriate AWS region and update the bucket_name variable in the script accordingly.

This approach offers a streamlined method to package and deploy Python dependencies in AWS Lambda environments, overcoming the challenges of direct installations or managing dependencies locally.

Part 3: Create Pip Layer and Test

After successfully executing the helper Lambda function for package installation, your next step involves interacting with the AWS S3 bucket to retrieve the zipped package file. Here’s a straightforward guide to integrating the package into your Lambda environment using Layers:

  1. Download the Package from S3: Navigate to the S3 bucket where your Lambda helper function uploaded the zipped package. Download this zip file to your local machine.

  2. Create a New Lambda Layer:

    • In the AWS Lambda console, locate the “Layers” option in the left menu.
    • Click on “Create layer”.
    • Provide a name for your layer and a brief description to identify its purpose or contents.
    • Under "Upload a .zip file", select the zip file you downloaded from S3.
    • For "Compatible runtimes", choose the Python version that matches your Lambda function (e.g., Python 3.12 if that's what you're using).
    • Specify the operating system if required, then click on "Create".
  3. Add the Layer to Your Lambda Function:

    • Return to the Lambda function where you intend to use the installed package.
    • Scroll down to the “Layers” section and click on “Add a layer”.
    • Choose “Custom layers” and then select the layer you just created.
    • Confirm the selection and version if prompted, then add the layer to your function.

With the layer now added to your Lambda function, you should have access to the third-party package (pyjwt in this case) and be able to execute your code as intended.

Reflecting on the Process

If this method enabled you to successfully run your code or introduced you to a new technique, consider the broader context of deploying applications in AWS. While using Lambda Layers for package management is effective, it's important to recognize its limitations, particularly for more complex dependencies or larger applications.

Exploring Further: Docker in AWS Lambda

For scenarios demanding greater control over the execution environment, Docker offers a robust solution. Docker containers allow you to package your application with all its dependencies into a single unit, ensuring consistency across development, testing, and production environments. I plan to cover Docker's integration with AWS Lambda in a future tutorial, showcasing it as the gold standard for managing dependencies and deploying applications in serverless architectures.

Stay tuned for more insights, and don't hesitate to reach out with questions or topics you'd like to explore further. Your engagement helps shape our content, ensuring it remains relevant and informative. Thank you for your support!

Create a free account to access full content.

All access to code and resources on ShillehTek.

Signup Now

Already a member? Sign In

Back to blog

Leave a comment

Please note, comments need to be approved before they are published.