AWS Lambda: Log handler not writing logs to CloudWatch unless use_queues=False

[rkulagowski]

I am using the following bit of initialization code in an AWS Lambda function.

thisModule = 'status'
logger = logging.getLogger(thisModule)
logger.setLevel(logging.INFO)
cwh = watchtower.CloudWatchLogHandler(
    log_group_name='Backend', log_stream_name=hf.today, use_queues=False)
cwh.setLevel(logging.WARNING)
logger.addHandler(cwh)

with the intent that logger.warning('log message')

would go to a dedicated log stream (hf.today would be "2022-03-17", etc), so that the logging traffic doesn't get mixed up in the standard CloudWatch logs which I'm emitting with logger.info()

Anyway, the above didn't work until I added use_queues=False.

Is my initialization code incorrect, or is something within use_queues not working as expected? I saw the comment about Django, but I'm not using Django.

Because this logging is occurring in short-lived AWS Lambda python functions, I'm not too concerned about the queues, other than it seems like it's not working.

If I am using this in a Lambda, and I do want to use the queues, do I need an explicit close / flush? Is that why my log messages aren't getting to CloudWatch?

[kislyuk]

I suppose I'll have to add this to the package documentation.

Watchtower is not suitable or necessary for AWS Lambda. All AWS Lambda logs (i.e. all lines printed to stderr by the runtime in the Lambda) are automatically sent to CloudWatch Logs, into log groups under the /aws/lambda/ prefix (https://console.aws.amazon.com/cloudwatch/home?#logsV2:log-groups$3FlogGroupNameFilter$3D$252Faws$252Flambda).

If you consult the documentation for the AWS Lambda execution environment (https://docs.aws.amazon.com/lambda/latest/dg/runtimes-context.html) you will note that AWS Lambda suspends (freezes) all processes in the environment once the invocation is complete and until the next invocation, if any. This means any asynchronous background processes and threads, including watchtower, will be suspended and inoperable, so watchtower cannot function correctly in this execution model.

If for some reason you want to keep using watchtower in Lambda, you can force the logs to flush by calling flush() on the handler, however that may have performance implications for your Lambda and is not advised.