Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Prometheus remote_write source #4697

Closed
bruceg opened this issue Oct 22, 2020 · 4 comments · Fixed by #5144
Closed

Add support for Prometheus remote_write source #4697

bruceg opened this issue Oct 22, 2020 · 4 comments · Fixed by #5144
Assignees
Labels
domain: sources Anything related to the Vector's sources source: prometheus_scrape Anything `prometheus_scrape` source related type: feature A value-adding code addition that introduce new functionality.

Comments

@bruceg
Copy link
Member

bruceg commented Oct 22, 2020

Between a Prometheus remote_read source (#4162) and a remote_write sink (#2058), there exists a third possibility. Prometheus has support for connecting to a remote system and sending metrics to it through the streaming remote_write protocol. These metrics are a copy of those written to its database as it receives them from other sources.

Vector would take the role of the remote with this new sink that accepts connections from a Prometheus server and receive a stream of metrics using the remote_write protocol. This stream could of course then be transformed and forwarded on to other metrics sinks.

@bruceg bruceg added needs: approval Needs review & approval before work can begin. domain: sources Anything related to the Vector's sources source: prometheus type: feature A value-adding code addition that introduce new functionality. labels Oct 22, 2020
@jamtur01 jamtur01 removed the needs: approval Needs review & approval before work can begin. label Oct 22, 2020
@spencergilbert
Copy link
Contributor

This would indeed be super cool

@langecode
Copy link
Contributor

This and #2058 would make is possible to use Vector-Vector as a channel for streaming Prometheus data - which would be super interesting :-)

@bruceg
Copy link
Member Author

bruceg commented Nov 16, 2020

The data exposed by the remote_write protocol is very primitive—everything more complex than a counter or gauge is translated into multiple simple metrics. Further, there is no "header" like there is in the text protocol to indicate the original type. There is however some evidence in the translated metrics what the original would have been (for example, the presence of metrics named X_min and X_max indicate X is a summary distribution). Should this source be trying to parse out these structures like it does with the text format, or leave that to a downstream transform (yet to be written)?

@langecode
Copy link
Contributor

langecode commented Nov 17, 2020

Well, I know that Vector models different metric types. However, in Prometheus the storage model contains no types. This is dependent on the interpretation of the timeseries - you could call it an implicit contract between the metrics producer and Prometheus. So a histogram will be represented by individual buckets signified by a le label which is picked up if the user applies the histogram_quantile, and so on. The end user writing the queries basically needs to know which functions makes sense to apply to a certain timeseries.

The overall use case we would like to be able to solve is something like:

app <- Prometheus -> Vector  ->  Vector -> remote write (Victoria Metrics, Cortex, ...)

So basically we would like to be able to tunnel each of the metric events through Vector into a remote write capable destination.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: sources Anything related to the Vector's sources source: prometheus_scrape Anything `prometheus_scrape` source related type: feature A value-adding code addition that introduce new functionality.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants