Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Only 1 watchtower handler works; others fail when logging with "WatchtowerWarning: Received message after logging system shutdown" #206

Closed
ebk46 opened this issue Aug 26, 2024 · 1 comment

Comments

@ebk46
Copy link

ebk46 commented Aug 26, 2024

We are using Django 4.2.15 and Watchtower 3.3.1

We have logging via Watchtower for 2 different CloudWatch streams within the same group. The following is environment-specific settings code within -settings.py files. They share a common initial LOGGING config.

boto3_logs_client = boto3.client("logs", region_name=AWS_REGION)

LOGGING["handlers"]["primary"] = {
  "class": "watchtower.CloudWatchLogHandler",
  "level": "DEBUG",
  "boto3_client": boto3_logs_client,
  "log_stream_name": "primary",
  "formatter": "aws",
}

LOGGING["loggers"]["primary"] = {
  "handlers": ["primary"],
  "level": "DEBUG",
}

LOGGING["handlers"]["secondary"] = {
  "class": "watchtower.CloudWatchLogHandler",
  "level": "DEBUG",
  "boto3_client": boto3_logs_client,
  "log_stream_name": "secondary",
  "formatter": "aws",
}

LOGGING["loggers"]["secondary"] = {
  "handlers": ["secondary"],
  "level": "DEBUG",
}

The "primary" logger works as intended. However, any logs to the "secondary" logger fail with:

WatchtowerWarning: Received message after logging system shutdown

As with many issues in this repo, setting use_queues: False enables logs for both, but that is not suitable for a production deployment.

@ebk46
Copy link
Author

ebk46 commented Aug 27, 2024

It seems that switching our gunicorn worker class from the default ("sync") to "gthread" and using multiple threads per worker somehow resolved this. I'm not sure exactly how, but I'll close this issue since it seems to not be directly related to watchtower. With that said, if that's true, it would probably be a helpful addition to the docs.

@ebk46 ebk46 closed this as completed Aug 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants
@ebk46 and others