Serverless S3 Pipeline

Yet another AWS service to generate thumbnails. This one is based on AWS Lambda, S3, and SQS.

It can be deployed to AWS or LocalStack using the Serverless Framework.

It also includes:

• Unit tests.
• Functional tests, which are executed against LocalStack.
• Pre-commit hooks: Black, ISort, Flake8, and MyPy.
• A Makefile with useful commands.

Design

1. An image is stored to the images bucket
2. The image creation event is queued into an SQS queue
3. The lambda function tries to generate the thumbnails
4. • Success: Thumbnails are saved to the thumbnails bucket
   • Error:
     1. The message is retried as many times as configured in the queue
     2. If it continues to fail, the message is sent to a DLQ
     3. You can manually invoke another lambda function that dequeues from the DLQ and sends the messages back to the original queue to be retried

Structure

The serverless.yml file contains the Serverless configuration to deploy the stack to either AWS or LocalStack.

The lambda functions are located in the functions package. Each AWS Lambda handler function is on a separate file. Common code is in the same package.

Unit tests are in the functions/tests package.

Integration tests are in the integration_tests package.

You can find useful commands in the Makefile.

Python requirements:

1. The requirements.txt file contains the essential Python dependencies required by the application logic to run.
2. The requirements.dev.txt file contains the Python dependencies you need to have installed in your environment to contribute to the application logic.
3. The requirements.test.txt file contains the Python dependencies required to run tests.

Setup

Install the Serverless Framework

npm install -g serverless

Install LocalStack:

pip install localstack

Install Serverless Framework Plugins

Go to the root directory of this repo and install the plugins:

cd serverless_s3_pipeline

npm i

Install and Configure the AWS CLI

Follow these instructions to install the AWS CLI.

To interact with LocalStack through the AWS CLI, you can create a profile with dummy region and access key.

Add this to your ~/.aws/config file:

[profile localstack]
region = us-east-1
output = json

And this to your ~/.aws/credentials file:

[localstack]
aws_access_key_id = dummyaccesskey
aws_secret_access_key = dummysecretaccesskey

Deploy to LocalStack

Start LocalStack:

localstack start

Deploy to LocalStack:

serverless deploy --stage local

You should get something like the following. Notice the endpoint URL:

✔ Service deployed to stack thumbnails-service-local

functions:
  generate_thumbnails: thumbnails-service-local-generate_thumbnails
  retry_from_dlq: thumbnails-service-local-retry_from_dlq

You can alternatively start localstack as a daemon and deploy with a single command:

make deploy_local

Makefile commands

Install GIT Hooks

make install_git_hooks

Run GIT Hooks

make run_git_hooks

Create Python virtualenv

make create_virtualenv

Install requirements into the virtualenv

make install_requirements
make install_dev_requirements
make install_test_requirements

# Or just
make install_all_requirements

Run tests

make run_unit_tests
make run_integration_tests

Deploy stack to LocalStack

Restarts LocalStack (if running) before deploying.

make deploy_local

Deploy only the Lambda functions to LocalStack

make deploy_functions_local

Show logs of the Lambda function that generates the thumbnails

make tail_generate_thumbnails_function_logs

Show logs of the Lambda function that enqueues failed messages to retry

make tail_retry_from_sqs_function_logs

Upload a few test images to the input bucket

make upload_test_images_to_s3

Show up to 10 messages in the DLQ

make show_messages_in_dlq

Invoke the Lambda function that enqueues failed messages to retry

make retry_from_dql