Discover how to automate image thumbnail creation using AWS Lambda, S3 Event Notifications, and Terraform, ensuring efficient and scalable processing.

In today’s digital landscape, visually appealing thumbnails are essential for capturing audience attention across various platforms like YouTube, Instagram, and more. Manually creating these thumbnails can be time-consuming and may require significant design skills. Fortunately, leveraging AWS S3 event-driven processing with AWS Lambda and Terraform can streamline this process, enabling automated and scalable thumbnail generation. This guide will walk you through setting up an automated system to create image thumbnails seamlessly.

Architecture Overview

The solution employs a serverless architecture that leverages AWS services to handle image processing efficiently. The primary components include:

S3 Buckets: One for storing original images (source) and another for storing generated thumbnails (destination).
AWS Lambda: Serverless functions that process images upon triggering.
Terraform: Infrastructure as Code (IaC) tool to manage and deploy AWS resources consistently.
CloudWatch: For monitoring and logging Lambda function executions.

AWS Architecture

Step 1: Create Source and Destination Buckets

Begin by setting up two S3 buckets using Terraform: one for the source images and another for the thumbnails.

# S3 Source Image Bucket
resource "aws_s3_bucket" "source_image_bucket" {
  bucket = var.source_bucket_name
  tags = {
    Name = "${local.naming_prefix}-s3-source-bucket"
  }
}

# S3 Thumbnail Image Bucket
resource "aws_s3_bucket" "thumbnail_image_bucket" {
  bucket = var.thumbnail_bucket_name
  tags = {
    Name = "${local.naming_prefix}-s3-thumbnail-bucket"
  }
}

This configuration ensures that all original images are stored separately from their corresponding thumbnails, promoting organized storage and easier management.

Step 2: Create IAM Policy for Lambda

Next, define an IAM policy that grants the necessary permissions for the Lambda function to access the S3 buckets.

# S3 Policy to Get and Put objects
resource "aws_iam_policy" "lambda_s3_policy" {
  name   = "LambdaS3Policy"
  policy = jsonencode({
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "s3:GetObject",
        "Resource": "${aws_s3_bucket.source_image_bucket.arn}/*"
      },
      {
        "Effect": "Allow",
        "Action": "s3:PutObject",
        "Resource": "${aws_s3_bucket.thumbnail_image_bucket.arn}/*"
      }
    ]
  })
}

This policy permits the Lambda function to retrieve images from the source bucket and upload the generated thumbnails to the destination bucket.

Step 3: Create IAM Role for Lambda

Establish an IAM role that the Lambda function will assume, attaching the previously created policy and the basic Lambda execution role.

# Lambda IAM role to assume the role
resource "aws_iam_role" "lambda_s3_role" {
  name               = "LambdaS3Role"
  assume_role_policy = jsonencode({
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "Service": "lambda.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
      }
    ]
  })
}

# Assign policy to the role
resource "aws_iam_policy_attachment" "assigning_policy_to_role" {
  name       = "AssigningPolicyToRole"
  roles      = [aws_iam_role.lambda_s3_role.name]
  policy_arn = aws_iam_policy.lambda_s3_policy.arn
}

resource "aws_iam_policy_attachment" "assigning_lambda_execution_role" {
  name       = "AssigningLambdaExecutionRole"
  roles      = [aws_iam_role.lambda_s3_role.name]
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

This setup ensures that the Lambda function has the necessary permissions to execute and interact with S3 buckets securely.

Step 4: Create the Lambda Function

Develop a Python-based Lambda function responsible for generating thumbnails. Compress the function code and its dependencies before deploying.

# Compressing lambda_handler function code
data "archive_file" "thumbnail_lambda_source_archive" {
  type        = "zip"
  source_dir  = "${path.module}/lambda"
  output_path = "${path.module}/lambda_function.zip"
}

# Creating Lambda Function
resource "aws_lambda_function" "create_thumbnail_lambda_function" {
  function_name    = "CreateThumbnailLambdaFunction"
  filename         = "${path.module}/lambda_function.zip"
  runtime          = "python3.12"
  handler          = "thumbnail_generator.lambda_handler"
  memory_size      = 256
  timeout          = 300
  environment {
    variables = {
      DEST_BUCKET = aws_s3_bucket.thumbnail_image_bucket.bucket
    }
  }
  source_code_hash = data.archive_file.thumbnail_lambda_source_archive.output_base64sha256
  role             = aws_iam_role.lambda_s3_role.arn
  layers           = [
    "arn:aws:lambda:us-east-1:770693421928:layer:Klayers-p312-Pillow:2"
  ]
}

The Lambda function utilizes the Pillow library for image processing and interacts with S3 using the boto3 library. Ensure that the function is bundled correctly with all dependencies.

Lambda Function Code Example

import logging
import boto3
from io import BytesIO
from PIL import Image
import os

logger = logging.getLogger()
logger.setLevel(logging.INFO)

s3_client = boto3.client('s3')

def lambda_handler(event, context):
    logger.info(f"event: {event}")
    logger.info(f"context: {context}")

    # Get the S3 bucket and object key from the event
    bucket = event["Records"][0]["s3"]["bucket"]["name"]
    key = event["Records"][0]["s3"]["object"]["key"]

    # Define the destination bucket and thumbnail key
    thumbnail_bucket = os.environ['DEST_BUCKET']
    thumbnail_name, thumbnail_ext = os.path.splitext(key)
    thumbnail_key = f"{thumbnail_name}_thumbnail{thumbnail_ext}"

    logger.info(f"Bucket name: {bucket}, file name: {key}, Thumbnail Bucket name: {thumbnail_bucket}, file name: {thumbnail_key}")

    # Open the image using Pillow
    file_byte_string = s3_client.get_object(Bucket=bucket, Key=key)['Body'].read()
    img = Image.open(BytesIO(file_byte_string))
    logger.info(f"Size before compression: {img.size}")

    # Create a thumbnail
    img.thumbnail((500, 500))
    logger.info(f"Size after compression: {img.size}")

    # Save the thumbnail to a BytesIO object
    buffer = BytesIO()
    img.save(buffer, "JPEG")
    buffer.seek(0)

    # Upload the thumbnail to the destination bucket
    sent_data = s3_client.put_object(
        Bucket=thumbnail_bucket,
        Key=thumbnail_key,
        Body=buffer
    )
    if sent_data['ResponseMetadata']['HTTPStatusCode'] != 200:
        raise Exception(f"Failed to upload image {key} to bucket {bucket}")

    return event

This function listens for new image uploads, processes them to create thumbnails, and stores the thumbnails in the designated S3 bucket.

Step 5: Configure S3 Event Notifications

Set up S3 event notifications to trigger the Lambda function whenever a new image is uploaded to the source bucket.

# Lambda Function Permission to allow S3 to invoke it
resource "aws_lambda_permission" "thumbnail_allow_bucket" {
  statement_id  = "AllowExecutionFromS3Bucket"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.create_thumbnail_lambda_function.arn
  principal     = "s3.amazonaws.com"
  source_arn    = aws_s3_bucket.source_image_bucket.arn
}

# Creating S3 Notification for Lambda
resource "aws_s3_bucket_notification" "thumbnail_notification" {
  bucket = aws_s3_bucket.source_image_bucket.id

  lambda_function {
    lambda_function_arn = aws_lambda_function.create_thumbnail_lambda_function.arn
    events              = ["s3:ObjectCreated:*"]
  }

  depends_on = [aws_lambda_permission.thumbnail_allow_bucket]
}

This configuration ensures that every time a new image is added to the source bucket, the Lambda function is automatically invoked to generate a thumbnail.

Step 6: Monitor with CloudWatch

Create a CloudWatch log group to capture and monitor logs from the Lambda function.

# Creating CloudWatch Log group for Lambda Function
resource "aws_cloudwatch_log_group" "create_thumbnail_lambda_function_cloudwatch" {
  name              = "/aws/lambda/${aws_lambda_function.create_thumbnail_lambda_function.function_name}"
  retention_in_days = 30
}

Monitoring is crucial for maintaining the health and performance of your AWS S3 event-driven processing pipeline. CloudWatch provides insights into function executions and potential issues.

Step 7: Deploy with Terraform

Initialize and apply your Terraform configuration to deploy the resources.

terraform init
terraform plan
terraform apply -auto-approve

Upon successful deployment, Terraform will output relevant information confirming the creation of resources.

Testing the Setup

To ensure everything is configured correctly:

Upload an Image: Add a large image to the source S3 bucket.
Verify Thumbnail Creation: Check the destination S3 bucket for the generated thumbnail.
Review Logs: Inspect the CloudWatch log group to confirm the Lambda function executed without errors.

This verification step ensures that your automated thumbnail generation process is functioning as intended, providing efficient and scalable image processing.

Cleanup

To avoid incurring unnecessary charges, clean up the AWS resources once you’re done testing.

terraform destroy -auto-approve

Remember to empty the S3 buckets before destroying them to ensure a smooth cleanup process.

Conclusion

By integrating AWS S3 event-driven processing with AWS Lambda and Terraform, you can automate the creation of image thumbnails efficiently. This serverless approach not only scales seamlessly with your needs but also reduces the overhead of managing infrastructure. Such automation is invaluable for content creators who strive to maintain a consistent and engaging online presence without dedicating excessive time to manual thumbnail design.

Embrace this solution to enhance your content creation workflow, ensuring your visuals always stand out and attract your target audience effectively.

Ready to streamline your thumbnail creation process? Get started with ThumblifyAI today!