consul agent based service discovery for promtail

[trevorwhitney]

Is your feature request related to a problem? Please describe.
The current service discovery mechanism for consul relies on the consul catalog api. This is problematic for very large clusters. Since promtail is often co-located with jobs on a machine also running a local consul agent, it would be nice to have the option to use the agent api instead. This would result in much smaller payloads when discovering services in large clusters.

Describe the solution you'd like
Either as a separate service discovery mechanism, or a configuration option to the existing consul service discovery mechanism, consul service discovery can be instructed to use the consul agent api for service discovery as opposed to the consul catalog api.

Describe alternatives you've considered

1. Use the current catalog api based SD as it exists:

• This results in json payload sizes of 300MB+ for the large cluster in question, causing the SD to struggle to keep up.

Additional context
This solution in particular is for a Nomad cluster. Nomad also has an API that could be leveraged for SD, but it appears to return results for the whole cluster (similar to the consul catalog API) and thus would likely run into the same scaling problems.

[trevorwhitney]

My team and I are volunteering to work on this, but wanted to expose the issue first to gather opinions/feedback.

[owen-d]

This sounds reasonable to me 👍