Allow promtail to set X-Scope-OrgID header dynamically

[brancz]

We are in a situation where we control both the setup and configuration of promtail and the loki backend, and are running multiple tenants on Kubernetes (a tenant is a namespace), and would like to set the tenant based on said namespace.

We are faced with two possible solutions:

• Perform log splitting in a proxy in front of the loki API, by inspecting the labels of each log stream and creating multiple requests split by the namespace.
• Allow relabelling a value into a __x_scope_orgid__ meta label, that wouldn't be attached to the target, but used as meta-configuration (similar to let's say __params.* in Prometheus), to set the header promtail would use.

In many ways this feels like cadvisor exposing metrics for all namespaces in Kubernetes, where we can also only scrape everything instead of on a per namespace basis.

My feeling is that having promtail be more flexible about this, for scenarios where both sides are controlled, to be a common scenario, for companies that have stricter requirements as to who can view what, in multi tenant environments.

Let me know what you think 🙂 .

@tomwilkie @gouthamve

[tomwilkie]

Yeah running multiple promtails would be a massive pain. Can you think of a nice way to do this, fitting in with the current code? Not against it per se, just worry it might be a bit of a horrible hack.

[brancz]

I have to dive into the code a bit more to be able to say if I can come up with an elegant solution, but more than happy to do so now knowing it's not entirely out there :)

[yubozhao]

Any follow up on this issue? We are heading toward this as well. We might able to do something with it

[brancz]

I haven’t had a chance yet to dive into this, so if you have some time to dive into it, go for it! :)

[kfox1111]

I'm interested in the same.

[base698]

Where is the tenant id/org id set? I get "no org id" logs in promtail, but can't find a place to put it in a config.

[yubozhao]

@base698 Please check my comments at this Issue for reference, #310 (comment). Feel free to ping me, if you have any other questions.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[brancz]

I was wondering if there could be another strategy for this, which would be: have a storage in-cluster, that is purely used for buffering and forwarding it to a remote storage?