Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs: Reimplement PR 5649 on troubleshooting timeout errors #6183

Merged
merged 1 commit into from
May 26, 2022
Merged

Docs: Reimplement PR 5649 on troubleshooting timeout errors #6183

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions docs/sources/operations/troubleshooting.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,27 @@ Promtail yet. There may be one of many root causes:
- Your pods are running with different labels than the ones Promtail is
configured to read. Check `scrape_configs` to validate.

## Loki timeout errors

Loki 504 errors, context canceled, and error processing requests
can have many possible causes.

- Review Loki configuration

- Loki configuration `querier.query_timeout`
- `server.http_server_read_timeout`
- `server.http_server_write_timeout`
- `server.http_server_idle_timeout`

- Check your Loki deployment.
If you have a reverse proxy in front of Loki, that is, between Loki and Grafana, then check any configured timeouts, such as an NGINX proxy read timeout.

- Other causes. To determine if the issue is related to Loki itself or another system such as Grafana or a client-side error,
attempt to run a [LogCLI](../../tools/logcli/) query in as direct a manner as you can. For example, if running on virtual machines, run the query on the local machine. If running in a Kubernetes cluster, then port forward the Loki HTTP port, and attempt to run the query there. If you do not get a timeout, then consider these causes:

- Adjust the [Grafana dataproxy timeout](https://grafana.com/docs/grafana/latest/administration/configuration/#dataproxy). Configure Grafana with a large enough dataproxy timeout.
- Check timeouts for reverse proxies or load balancers between your client and Grafana. Queries to Grafana are made from the your local browser with Grafana serving as a proxy (a dataproxy). Therefore, connections from your client to Grafana must have their timeout configured as well.

## Troubleshooting targets

Promtail exposes two web pages that can be used to understand how its service
Expand Down