Monitoring Lambda Functions with OpenTelemetry

This repository contains all of the instructions and files you need to instrument & monitor an AWS Lambda function with OpenTelemetry across several common programming languages.

Prerequisites

The following required software must be accessible from your command line:

• Terraform (required)
  • Check with terraform --version
  • Installation docs
• Configured AWS CLI (required)
  • Check with aws --version
  • Installation docs
  • Configuration docs
  • Configuration video
• Go >1.18 SDK (only required for Golang)
  • Check with go version
  • Installation docs
• Maven >3.x (only required for Java)
  • Check with mvn -v
  • Download docs
  • Installation docs
• Java >11 SDK (only required for Java)
  • Check with java -version
  • Installation docs (Amazon Coretto 17)
• Python >3.10 (only required for Python)
  • Check with python3 --version
  • Installation docs (3.11)

Architecture

In order to demonstrate different scenarios, a sample data processing environment has been build for this workshop. The structure of the environment can be seen in the inmage below:

Refer to this documentation to learn more about what you will be deploying!

Getting started

In a new terminal window, clone this repository and cd into it:

git clone https://github.com/utr1903/monitoring-lambda-with-opentelemetry
cd monitoring-lambda-with-opentelemetry

Currently, every programming language in this repo has its own environment:

• Golang
• Java
• Python

The folder structure of each language is the same:

- apps
  - check
  - create
  - delete
  - update
- infra
  - scripts
  - terraform

Account Setup

In this step, you will associate your AWS region and New Relic license key to the application so that telemetry data will be sent to your New Relic account. If you don't already have a New Relic account, set one up for free here

Once you have your region and license key, set the environment variables with the following CLI commands:

export AWS_REGION="XXX" # example: eu-west-1
export NEWRELIC_LICENSE_KEY="XXX"

Deploy AWS Resources

Now we will deploy the AWS resources using a preconfigured Terraform script in the Golang directory.

1. Switch to the {language}/infra/scripts directory using the command below:

cd golang/infra/scripts

2. run the 00_deploy_aws_resources.sh script:

bash 00_deploy_aws_resources.sh

3. After the Terraform deployment is complete, the public URL of the API Gateway will be prompted to your terminal. You can generate some traffic by triggering it with the following one-line curl loop:

while true; do; curl -X POST "${API_GATEWAY_PUBLIC_URL}/create"; sleep 1; done

Example:

while true; do; curl -X POST "https://mmzght1j5l.execute-api.eu-west-1.amazonaws.com/prod/create"; sleep 1; done

4. At this point, you have deployed and instrumented the Golang service. To view this data in New Relic immediately run the following query in your account:

SELECT * FROM Span WHERE instrumentation.provider = 'opentelemetry'

After 5 minutes have passed since running the terraform script, navigate to All Entities in the New Relic platform and locate the service(s) named golang-lambda-***-otel

Complete Instrumentation

The full instrumentation of each service in each programming language contains components of auto and manual instrumentation.

• The Lambda functions included in this environment are wrapped with OpenTelemetry auto-instrumentation layers which instrument the inbound & outbound calls to/from your services.
• In order to track custom KPIs, additional instrumentation (known as manual instrumentation) is needed. For example, creating custom span events or adding additional attributes to individual spans.

The telemetry data generated within the Lambda function is then sent to an OpenTelemetry collector which is mounted to the Lambda function either as a zip package or a layer (see code). The collector is then responsible for forwarding the telemetry data to your backend of choice!

REMARK: Currently this repo is designed to send data to New Relic by default. However, it is possible to make small edits to the OTel collector config and Terraform Lambda environment variables to send data elsewhere.

Capability Matrix

The table below visualizes the level of instrumentation required for each service.

In each cell:

• 1st mark is for the implementation of the code
• 2nd mark is for the implementation of the auto-instrumentation
• 3rd mark is for the implementation of the manual-instrumentation

Language	create	update	check	delete
Golang	✅ ✅ ✅	✅ ✅ ✅	✅ ✅ ✅	✅ ✅ ✅
Java	✅ ✅ ✅	✅ ❌ ❌	✅ ✅ ✅	✅ ✅ ✅
Python	✅ ✅ ✅	✅ ✅ ✅	✅ ✅ ✅	✅ ✅ ✅

Golang

• The trace context propagation between Lambdas fails to be established due to a lack of SDK functionality.

Java

• The update Lambda cannot be auto-instrumented currently because of the bug.

TODOs

• Propagate trace context
• Send metrics
• Send logs (in context with traces)