Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add unique promtail_instance id to labels for gcptarget #3501

Merged
merged 2 commits into from
Mar 17, 2021
Merged

Add unique promtail_instance id to labels for gcptarget #3501

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c5b7fc1
Add unique promtail_instance id to labels for gcptarget
kavirajk Mar 17, 2021
d65cb83
Update docs/sources/clients/promtail/scraping.md
kavirajk Mar 17, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/sources/clients/promtail/scraping.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,10 @@ Before using `gcplog` target, GCP should be [configured](../gcplog-cloud) with p

It also support `relabeling` and `pipeline` stages just like other targets.

Log entries scraped by `gcplog` will add an additional label called `promtail_instance`. This label uniquely identifies each promtail instance trying to scrape gcplog (from a single `subscription_id`).
We need this unique identifier to avoid out-of-order errors from Loki servers.
Because say two promtail instances rewrite timestamp of log entries(with same labelset) at the same time may reach Loki servers at different times can cause Loki servers to reject it.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think here you should perhaps explain a bit more what causes this conflict in log lines - it may not be immediately obvious to users who are new to this. You've added a config option in the docs (use_incoming_timestamp: false # default rewrite timestamps) but this is optional - correct? If users don't notice that log line, they may be confused as to why we are rewriting the timestamp.

This adds another wrinkle: if users opt to not rewrite the timestamp, is the unique UUID even required? (we can leave it in either way, just curious)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding the wrinkle: I tried with and without rewriting the timestamp during my deployment. In both cases out of order errors appeared, so this UUID is still needed to be able to ingest GCP logs.

Maybe this is not needed when you are limiting the amount of logs, but I already had issues with just the k8s_cluster resource logs.


## Syslog Receiver

Promtail supports receiving [IETF Syslog (RFC5424)](https://tools.ietf.org/html/rfc5424)
Expand Down
1 change: 1 addition & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ require (
github.com/prometheus/client_model v0.2.0
github.com/prometheus/common v0.18.0
github.com/prometheus/prometheus v1.8.2-0.20210215121130-6f488061dfb4
github.com/satori/go.uuid v1.2.1-0.20181028125025-b2ce2384e17b
github.com/segmentio/fasthash v1.0.2
github.com/shurcooL/httpfs v0.0.0-20190707220628-8d4bc4ba7749
github.com/shurcooL/vfsgen v0.0.0-20200824052919-0d455de96546
Expand Down
Loading