v1.12.0

Changes

This release brings a number of minor but importance enhancements in Logger, Parameters, Data Classes, Idempotency.

Also, this release wouldn't have been possible without @michaelbrewer relentless contribution - Thank you, Michael!!

Logger

You can now inject correlation IDs coming from any Event Source using JMESPath expressions, or by manually injecting a string with set_correltion_id.

Another enhancement that goes nicely with any Logging Analytics solution such as Amazon CloudWatch Logs Insights, Kibana, Datadog, Splunk, Loggly, etc is the ability to enumerate exceptions across all your functions - A new field exception_name will be added to your Logs when using logger.exception().

Data Classes

You can now have a self-documented schema for AppSync Lambda Resolvers whether you use Direct Lambda Resolver feature, or @function GraphQL Transformer with Amplify CLI. More importantly, scalar type utilities such as timestamps, timezone offsets, UUIDs, are also available!

Another enhancement is additional documentation for Cognito custom auth challenges.

Parameters

A minor enhancement but important for sensitive parameters that cannot be cached. You can now use force_fetch=True parameter to always fetch the latest from your preferred parameter provider be that Parameter Store, Secrets Manager, AppConfig, DynamoDB, or your own.

We have also refreshed Parameters documentation using the new Getting started vs Advanced, including surfacing how you can control in-memory cache TTL for parameters retrieved.

Idempotency

Besides error handling improvements, thanks to @Dunedan and @michaelbrewer (#318), you can now reuse a single DynamoDB table to store idempotency state. All new idempotency states stored in your DynamoDB table, as part of this release, will now contain your function name as a prefix for the hash key.

This is one step forward towards GA for Idempotency - The last missing feature before it's GA is allowing an exception callback mechanism so you can handle any exceptions to more easily return custom responses for event sources like API Gateway, AppSync, etc.

🌟New features and non-breaking changes

• feat(parameters): Add force_fetch option (#341) by @michaelbrewer
• feat(data-classes): Add appsync scalar_types_utils and add support for timezone offsets (#339) by @michaelbrewer
• feat(idempotent): Include function name in the idempotent key (#326) by @michaelbrewer
• feat(data-classes): AppSync Resolver Event (#323) by @michaelbrewer
• feat(logging): Add correlation_id support (#321) by @michaelbrewer
• feat(logging): Include exception_name (#320) by @michaelbrewer

📜 Documentation updates

• docs: surface new 1.12.0 features and enhancements (#344) by @heitorlessa
• docs(data-classes): Add more cognito code examples (#340) by @michaelbrewer
• docs: Correct code examples (#317) by @michaelbrewer
• docs(metrics): Correct code examples in markdown (#316) by @michaelbrewer
• docs(metrics): Corrections to the docstrings code examples (#314) by @michaelbrewer
• docs(idempotency): Correct examples and line highlights (#312) by @michaelbrewer

🐛 Bug and hot fixes

• docs(tracer): Fix Tracer typing hinting for Pycharm (#345) by @michaelbrewer
• fix(idempotency): TypeError when calling is_missing_idempotency_key with an int (#315) by @michaelbrewer
• fix(idempotency): Correctly handle save_inprogress errors (#313) by @michaelbrewer

Internal

• refactor(parameters): Consistently reference env (#319) by @michaelbrewer

This release was made possible by the following contributors:

@heitorlessa and @michaelbrewer

v1.11.0

Changes

This is our most special release this quarter, as we are happy to announce:

1. Up to 75% performance improvement for customers not using Tracer utility due to lazy loading
2. Up to 40% performance improvement for customers using Metrics as large operations are now <1ms
3. Brand new Idempotency utility for safely retrying operations that have side-effects, one of the most awaited feature since we've gone GA

Performance improvements

Tracer effect on import time flame graph before and after initial round of optimizations

Warm execution logging, serializing, and validating 99 Metrics to CloudWatch asynchronously on the lowest Lambda fn memory (128) after all optimizations

New idempotency utility

Initially led by @igorlg RFC on Idempotency, and its implementation that started in mid December by @cakepietoast, with a gigantic effort by both @cakepietoast and @michaelbrewer on UX and correctness following Amazon Builder's Library and Stripe designs -- We now have it in Beta as part of this release.

Given the amount of serious work for this utility and its logic, we've decided it'd be best to launch as Beta as we'd like feedback on UX before we release as stable.

Key features at launch:

• Use DynamoDB as a persistence layer for state including Bring Your Own Persistence Store Layer
• Optional in-memory caching, idempotency state auto-expiration for cost reasons
• JMESPath support to select a subset of your payload to serialize and hash as idempotency key
• Payload validation in case we shouldn't return the previous result in case X parameter has changed in your request
• Last but not least, the documentation is super comprehensive with over 600 lines of content.

We truly hope you enjoy it :-)

🌟New features and non-breaking changes

• feat(data-classes): Add connect contact flow event (#304) by @Sordie
• feat(idempotency): Add raise_on_no_idempotency_key flag (#297) by @michaelbrewer
• feat: Idempotency helper utility (#245) by @cakepietoast

🌟 Minor Changes

• refactor(idempotent): Change UX to use a config class for non-persistence related features (#306) by @michaelbrewer
• refactor(metrics): optimize validation and serialization (#307) by @heitorlessa
• feat(idempotent): Add support for jmespath_options (#302) by @michaelbrewer
• fix(core): import time latency by lazily loading high level modules (#301) by @heitorlessa

📜 Documentation updates

• docs(dataclasses): new Connect Contact Flow (#310) by @heitorlessa
• docs(idempotency): tidy up doc before release (#309) by @heitorlessa
• docs(batch): add example on how to integrate with sentry.io (#308) by @heitorlessa
• docs(idempotent): Fix typos and code formatting (#305) by @michaelbrewer
• docs(data-classes): Correct import for DynamoDBRecordEventName (#299) by @michaelbrewer

🐛 Bug and hot fixes

• test(general): Add some missing code coverage (#296) by @michaelbrewer
• feat(idempotency): Fix KeyError when local_cache is True and an error is raised in the lambda handler (#300) by @michaelbrewer
• fix: correct behaviour to avoid caching "INPROGRESS" records (#295) by @cakepietoast

This release was made possible by the following contributors:

@Sordie, @cakepietoast, @heitorlessa and @michaelbrewer

v1.10.5

Changes

No code changes. Release to re-trigger layer publish.

• chore: version bump to 1.10.5 (#292) by @cakepietoast

This release was made possible by the following contributors:

@cakepietoast

v1.10.4

Changes

Releasing a new version primarily to update the published "extras" Lambda layer, as it was not being built correctly (see #290). Also including some fixes for the recently updated documentation thanks to @michaelbrewer.

📜 Documentation updates

• docs(data-classes): Fix anchor tags to be lower case (#288) by @michaelbrewer
• chore: Correct the docs location for the labeller (#289) by @michaelbrewer

Internal

• chore: version bump to 1.10.4 (#291) by @cakepietoast

This release was made possible by the following contributors:

@cakepietoast, @heitorlessa and @michaelbrewer

v1.10.3

Changes

This patch release fixes a) a bug in Tracer when multiple traces are collected and reach Lambda X-Ray agent payload limit, and b) a minor type issue in Parser that made mypy unable to validate models.

This release also marks our documentation system migration from Gatsby to MKdocs, including navigation improvements to address customers feedback on finding getting started vs advanced information.

🌟New features and non-breaking changes

• test: add internal benchmark on AWS Lambda for core utilities (#261) by @nmoutschen

📜 Documentation updates

• docs: migrate documentation from Gatsby to MkDocs material (#279) by @am29d

🐛 Bug and hot fixes

• fix: fix typing hit for envelope parse model (#286) by @nmoutschen
• fix: disable batching of X-Ray subsegments (#284) by @nmoutschen

This release was made possible by the following contributors:

@am29d, @heitorlessa and @nmoutschen

v1.10.2

Changes

This quick release fixes:

• batch processing utility when dealing with multiple exceptions when processing SQS records
• package size for Python <3.8 where typing_extensions module was being installed unnecessarily
• Typos in the documentation

🌟 Minor Changes

• fix: remove unnecessary typing-extensions for py3.8 (#281) by @nadobando

📜 Documentation updates

• docs(appconfig): Use correct import for docstring (#271) by @michaelbrewer
• chore: typo in parser docs (#274) by @heitorlessa

🐛 Bug and hot fixes

• fix: batch processing exceptions (#276) by @cakepietoast

Internal

• chore: bump to 1.10.2 (#282) by @heitorlessa
• chore: fix immer and socket.io CVEs (#278) by @heitorlessa

This release was made possible by the following contributors:

@cakepietoast, @heitorlessa, @michaelbrewer and @nadobando

v1.10.1

Changes

This release patches a model mismatch when using SNS -> SQS -> Lambda as opposed to SNS -> Lambda. The former changes three keys that are incompatible with the model we derived from Lambda:

• MessageAttributes key is not present
• UnsubscribeUrl becomes UnsubscribeURL
• SigningCertUrl becomes SigningCertURL

This release also introduces a new envelope, SnsSqsEnvelope, to make this process seamless and easier when processing messages in Lambda that come from SNS -> SQS. This will extract the original SNS published payload, unmarshall it, and parse it using your model.

from aws_lambda_powertools.utilities.parser import BaseModel, envelopes, event_parser

class MySnsBusiness(BaseModel):
    message: str
    username: str

@event_parser(model=MySnsBusiness, envelope=envelopes.SnsSqsEnvelope)
def handle_sns_sqs_json_body(event: List[MySnsBusiness], _: LambdaContext):
    assert len(event) == 1
    assert event[0].message == "hello world"
    assert event[0].username == "lessa"

🌟 Minor Changes

• feat: add support for SNS->SQS protocol (#272) by @heitorlessa

Maintenance

• chore: bump to 1.10.1 (#273) by @heitorlessa

This release was made possible by the following contributors:

@heitorlessa

v1.10.0

Changes

This release adds a number of new features to Logger, Tracer, Validator, and Parameters, a new Lambda Layer with extra packages installed (e.g. parser), and documentation fixes.

Detailed information about these features are at the bottom.

For the next release (1.11.0), we'll be focusing on a new Idempotency utility, import time performance improvements for those not using Tracer utility, and possibly a new Circuit Breaker utility.

---

This release was made possible by the following contributors:

@am29d, @heitorlessa, @michaelbrewer, @n2N8Z, @risenberg-cyberark and @suud

🌟New features and non-breaking changes

• feat: Add AppConfig parameter provider (#236) by @risenberg-cyberark
• feat: support extra parameter in Logger messages (#257) by @heitorlessa

🌟 Minor Changes

• feat: toggle to disable log deduplication locally for pytest live log #262 (#268) by @heitorlessa
• improv: add support for custom lambda handlers with kwargs #242 (#269) by @heitorlessa
• improv: override Tracer auto-capture response/exception via env vars (#259) by @heitorlessa
• feat: support custom formats in JSON Schema validation (#247) by @n2N8Z

📜 Documentation updates

• docs: fix import (#267) by @suud
• docs: add info about extras layer (#260) by @am29d
• docs: add missing parser models (#254) by @risenberg-cyberark

Internal

• chore: bump to 1.10.0 (#270) by @heitorlessa
• chore: move env names to constant file (#264) by @heitorlessa
• chore: general simplifications and cleanup (#255) by @michaelbrewer
• chore: docs npm high vuln. (#256) by @heitorlessa

Details

Logger

Extra parameter

Logger now supports extra parameter in the standard logging when logging new messages. You can pass in any dictionary to this new parameter, and its keys and values will be available within the root of the structure - This is ephemeral and keys do not persist with its counterpart structure_logs(append=True, ...) method.

Excerpt

from aws_lambda_powertools import Logger

logger = Logger(service="payment")

fields = { "request_id": "1123" }

logger.info("Hello", extra=fields)

Log sample

{
   "timestamp": "2021-01-12 14:08:12,357",
   "level": "INFO",
   "location": "collect.handler:1",
   "service": "payment",
   "sampling_rate": 0.0,
   "request_id": "1123", // highlight-line
   "message": "Collecting payment"
}

Pytest Live Log support

Pytest Live Log support will output logging records as they are emitted directly into the console with colours.

Since Logger drops duplicate log records as of 1.7.0, you can now explicitly override this protection when running tests locally to make use of Pytest Live Log:

POWERTOOLS_LOG_DEDUPLICATION_DISABLED="1" pytest -o log_cli=1

Tracer

Override auto-capture response and exception

When using Tracer decorators, capture_method or capture_lambda_handler, we auto-capture its responses and exceptions, serialize them and inject as tracing metadata to ease troubleshooting.

There are times when serializing objects can cause side effects, for example reading S3 streaming objects if not read before. You can now override this behaviour either by parameter or via env var: POWERTOOLS_TRACER_CAPTURE_RESPONSE, POWERTOOLS_TRACER_CAPTURE_ERROR

Parameters

AppConfig support

You can now retrieve and cache configuration stored in AppConfig natively - Thanks to Ran from CyberArk.

from aws_lambda_powertools.utilities import parameters

def handler(event, context):
    # Retrieve a single configuration, latest version
    value: bytes = parameters.get_app_config(name="my_configuration", environment="my_env", application="my_app")

Validator

Custom formats

You can now use formats parameter to instruct Validator utility how to deal with any custom integer or string - Thanks to @n2N8Z

Custom format snippet in a JSON Schema

{
	"lastModifiedTime": {
	  "format": "int64",
	  "type": "integer"
	}
}

Excerpt ignoring int64 and a positive format instead of failing

from aws_lambda_powertools.utilities.validation import validate

event = {} # some event
schema_with_custom_format = {} # some JSON schema that defines a custom format

custom_format = {
    "int64": True, # simply ignore it,
	"positive": lambda x: False if x < 0 else True
}

validate(event=event, schema=schema_with_custom_format, formats=custom_format)

---

v1.9.1

Changes

This patch release fixes a bug when multiple parent Loggers with the same name are configured multiple times, as per #249.

from aws_lambda_powertools import Logger


class Class1:
    logger = Logger("class_1")

    @staticmethod
    def f_1():
        Class1.logger.info("log from class_1")


class Class2:
    logger = Logger("class_1")

    @staticmethod
    def f_2():
        Class2.logger.info("log from class_2")
        Class1.f_1()


Class2.f_2()

This release also adds brings staged changes initially planned for 1.10 such as minor improvements in Tracer documentation, brings equality checks to ease testing Event source data classes utility, and initial work for MyPy support (PEP 561).

🌟New features and non-breaking changes

• feat: PEP-561 compliance (#237) by @gmcrocetti

🌟 Minor Changes

• improv: Added eq function to DictWrapper for better equality checks (#233) by @GroovyDan
• test(general): Add missing tests for parser (#232) by @michaelbrewer
• docs: add clarification to Tracer docs for capture_method (#244) by @cakepietoast
• test: DictWrapper equals method (#234) by @Nr18
• chore: implement phony targets correctly (#235) by @Nr18

🐛 Bug and hot fixes

• fix: prevent touching preconfigured loggers #249 (#250) by @heitorlessa

This release was made possible by the following contributors:

@GroovyDan, @Nr18, @cakepietoast, @dependabot, @dependabot[bot], @gmcrocetti, @heitorlessa and @michaelbrewer

v1.9.0

Changes

This release adds support for Kinesis, S3, CloudWatch Logs, Application Load Balancer, and SES models in Parser - Exclusively added by Ran (once again) from CyberArk.

Docs clarified Logger keys that cannot be suppressed, a broken link, and the sidebar menu is now always expanded by default for improved UX.

🌟New features and non-breaking changes

• feat: Add Kinesis lambda event support to Parser utility (#227) by @risenberg-cyberark
• feat: Add S3 lambda event support to Parser utility #224 (#225) by @risenberg-cyberark
• feat: Add CloudWatch lambda event support to Parser utility (#231) by @risenberg-cyberark
• feat: Add ALB lambda event support to Parser utility (#229) by @risenberg-cyberark
• feat: Add SES lambda event support to Parser utility #213 (#214) by @risenberg-cyberark

📜 Documentation updates

• docs: Clarify Logger keys that cannot be suppressed (#219) by @igorlg
• docs: add source code link in nav bar (#223) by @heitorlessa
• docs: fix broken link for github (#222) by @pankajagrawal16

This release was made possible by the following contributors:

@heitorlessa, @igorlg, @pankajagrawal16, @risenberg-cyberark, Pankaj Agrawal and Ran Isenberg