This repository has been archived by the owner on Nov 21, 2024. It is now read-only.
Support "up" metric for Prometheus #8
Assignees
Labels
data-model
design-required
P0
An issue that needs to be addressed immediately. Breaking change.
phase1
phase1 tasks
prom-receiver
Prometheus receiver tasks
Milestone
Comments
|
Note: As long as it is a scrape, a 'up' metric is generated in Prometheus even in an unsuccessful scrape. |
|
I experimented with this, and have a working PoC, in-case anyone wants to see what it would look like. It isn't much, but might be useful to someone: open-telemetry/opentelemetry-collector@main...dashpole:add_up_metric. One thing that came up is that the UP metric doesn't seem to have a type when we receive it. In order to make it work, I had to set the type to a prometheus GAUGE_DOUBLE. I also found that the UP metric isn't super-useful without instance and job labels. Its a bit hard to tell what is (or isn't) up without those: #7. |
|
cc @odeke-em |
|
This needs to be assigned to @odeke-em. |
|
@odeke-em please join the OTel organization. We will sponsor you! :-) |
|
Thank you @jmacd, and great to catch you here! |
odeke-em
added a commit
to orijtech/opencensus-go-exporter-prometheus
that referenced
this issue
Apr 13, 2021
Internal metrics consumed by Prometheus such as "up" indicate
a status of the target, as either "up" with 1.0 or "down" with 0.0.
This change ensures that no ConstLabels, nor namespace prefixes will
be added to such metrics. This ensures that when one exports with
the special name "up", that it passes through up to the Prometheus
exporter, so:
# HELP up up
# TYPE up counter
up{instance="localhost:9999"} 1
instead of:
# HELP tests_up tests/up
# TYPE tests_up counter
tests_up{instance="localhost:9999",service="spanner"} 1
A further assertion can be added to ensure that "up" is a gauge,
but I am not sure that it might be necessary, just in case some
user wants to expose it as a counter.
Updates open-telemetry/prometheus-interoperability-spec#8
odeke-em
added a commit
to orijtech/opencensus-go-exporter-prometheus
that referenced
this issue
Apr 13, 2021
Internal metrics consumed by Prometheus such as "up" indicate
a status of the target, as either "up" with 1.0 or "down" with 0.0.
This change ensures that no ConstLabels, nor namespace prefixes will
be added to such metrics. This ensures that when one exports with
the special name "up", that it passes through up to the Prometheus
exporter, so:
# HELP up up
# TYPE up counter
up{instance="localhost:9999"} 1
instead of:
# HELP tests_up tests/up
# TYPE tests_up counter
tests_up{instance="localhost:9999",service="spanner"} 1
A further assertion can be added to ensure that "up" is a gauge,
but I am not sure that it might be necessary, just in case some
user wants to expose it as a counter.
Updates open-telemetry/prometheus-interoperability-spec#8
odeke-em
added a commit
to orijtech/opentelemetry-collector
that referenced
this issue
Apr 13, 2021
Make a receiver specific view that'll be registered
and used to record the "up" status either "0.0" or "1.0"
when an instance can't be scraped from or can be, respectively.
This ensures that the collector can act as a passthrough
for statuses and it currently outputs:
# HELP up Whether the endpoint is alive or not
# TYPE up gauge
up{instance="0.0.0.0:8888"} 1
up{instance="localhost:9999"} 0
Fixes open-telemetry/prometheus-interoperability-spec#8
Requires census-ecosystem/opencensus-go-exporter-prometheus#24
odeke-em
added a commit
to orijtech/opentelemetry-collector
that referenced
this issue
Apr 13, 2021
Make a receiver specific view that'll be registered
and used to record the "up" status either "0.0" or "1.0"
when an instance can't be scraped from or can be, respectively.
This ensures that the collector can act as a passthrough
for statuses and it currently outputs:
# HELP up Whether the endpoint is alive or not
# TYPE up gauge
up{instance="0.0.0.0:8888"} 1
up{instance="localhost:9999"} 0
I did not take the approach of plainly sending up suffixed metric names.
to recommend instead using relabelling inside the exporter itself like:
- source_labels: [__name__]
regex: "(.+)_up"
target_label: "__name__"
replacement: "up"
because:
* it'd apply ConstLabels on every *_up metric, only want "instance=$INSTANCE"
* other exporters wouldn't be able to use the "up" metric as is if we
inject rewrites
Regardless of if we used a label rewrite, the end result would be the
following:
up{instance="localhost:8888",job="otlc"}
up{exported_instance="0.0.0.0:9999",instance="localhost:8888",job="otlc"}
up{exported_instance="0.0.0.0:1234",instance="localhost:8888",job="otlc"}
which this change accomplishes without having to inject any label
rewrites, but just by the new imports and upgrade of the prometheus
exporter.
Fixes open-telemetry/prometheus-interoperability-spec#8
Requires census-ecosystem/opencensus-go-exporter-prometheus#24
odeke-em
added a commit
to orijtech/opentelemetry-collector
that referenced
this issue
Apr 13, 2021
Make a receiver specific view that'll be registered
and used to record the "up" status either "0.0" or "1.0"
when an instance can't be scraped from or can be, respectively.
This ensures that the collector can act as a passthrough
for statuses and it currently outputs:
# HELP up Whether the endpoint is alive or not
# TYPE up gauge
up{instance="0.0.0.0:8888"} 1
up{instance="localhost:9999"} 0
I did not take the approach of plainly sending up suffixed metric names.
to recommend instead using relabelling inside the exporter itself like:
- source_labels: [__name__]
regex: "(.+)_up"
target_label: "__name__"
replacement: "up"
because:
* it'd apply ConstLabels on every *_up metric, only want "instance=$INSTANCE"
* other exporters wouldn't be able to use the "up" metric as is if we
inject rewrites
Regardless of if we used a label rewrite, the end result would be the
following:
up{instance="localhost:8888",job="otlc"}
up{exported_instance="0.0.0.0:9999",instance="localhost:8888",job="otlc"}
up{exported_instance="0.0.0.0:1234",instance="localhost:8888",job="otlc"}
which this change accomplishes without having to inject any label
rewrites, but just by the new imports and upgrade of the prometheus
exporter.
Fixes open-telemetry/prometheus-interoperability-spec#8
Requires census-ecosystem/opencensus-go-exporter-prometheus#24
odeke-em
added a commit
to orijtech/opentelemetry-collector
that referenced
this issue
Apr 13, 2021
Make a receiver specific view that'll be registered
and used to record the "up" status either "0.0" or "1.0"
when an instance can't be scraped from or can be, respectively.
This ensures that the collector can act as a passthrough
for statuses and it currently outputs:
# HELP up Whether the endpoint is alive or not
# TYPE up gauge
up{instance="0.0.0.0:8888"} 1
up{instance="localhost:9999"} 0
I did not take the approach of plainly sending up suffixed metric names.
to recommend instead using relabelling inside the exporter itself like:
- source_labels: [__name__]
regex: "(.+)_up"
target_label: "__name__"
replacement: "up"
because:
* it'd apply ConstLabels on every *_up metric, only want "instance=$INSTANCE"
* other exporters wouldn't be able to use the "up" metric as is if we
inject rewrites
Regardless of if we used a label rewrite, the end result would be the
following:
up{instance="localhost:8888",job="otlc"}
up{exported_instance="0.0.0.0:9999",instance="localhost:8888",job="otlc"}
up{exported_instance="0.0.0.0:1234",instance="localhost:8888",job="otlc"}
which this change accomplishes without having to inject any label
rewrites, but just by the new imports and upgrade of the prometheus
exporter.
Fixes open-telemetry/prometheus-interoperability-spec#8
Requires census-ecosystem/opencensus-go-exporter-prometheus#24
odeke-em
added a commit
to orijtech/opentelemetry-collector
that referenced
this issue
Apr 16, 2021
Make a receiver specific view that'll be registered
and used to record the "up" status either "0.0" or "1.0"
when an instance can't be scraped from or can be, respectively.
This ensures that the collector can act as a passthrough
for statuses and it currently outputs:
# HELP up Whether the endpoint is alive or not
# TYPE up gauge
up{instance="0.0.0.0:8888"} 1
up{instance="localhost:9999"} 0
I did not take the approach of plainly sending up suffixed metric names.
to recommend instead using relabelling inside the exporter itself like:
- source_labels: [__name__]
regex: "(.+)_up"
target_label: "__name__"
replacement: "up"
because:
* it'd apply ConstLabels on every *_up metric, only want "instance=$INSTANCE"
* other exporters wouldn't be able to use the "up" metric as is if we
inject rewrites
Regardless of if we used a label rewrite, the end result would be the
following:
up{instance="localhost:8888",job="otlc"}
up{exported_instance="0.0.0.0:9999",instance="localhost:8888",job="otlc"}
up{exported_instance="0.0.0.0:1234",instance="localhost:8888",job="otlc"}
which this change accomplishes without having to inject any label
rewrites, but just by the new imports and upgrade of the prometheus
exporter.
Fixes open-telemetry/prometheus-interoperability-spec#8
Requires census-ecosystem/opencensus-go-exporter-prometheus#24
|
Duplicate of #41. |
odeke-em
added a commit
to orijtech/opentelemetry-collector
that referenced
this issue
Apr 26, 2021
Make a receiver specific view that'll be registered
and used to record the "up" status either "0.0" or "1.0"
when an instance can't be scraped from or can be, respectively.
This ensures that the collector can act as a passthrough
for statuses and it currently outputs:
# HELP up Whether the endpoint is alive or not
# TYPE up gauge
up{instance="0.0.0.0:8888"} 1
up{instance="localhost:9999"} 0
I did not take the approach of plainly sending up suffixed metric names.
to recommend instead using relabelling inside the exporter itself like:
- source_labels: [__name__]
regex: "(.+)_up"
target_label: "__name__"
replacement: "up"
because:
* it'd apply ConstLabels on every *_up metric, only want "instance=$INSTANCE"
* other exporters wouldn't be able to use the "up" metric as is if we
inject rewrites
Regardless of if we used a label rewrite, the end result would be the
following:
up{instance="localhost:8888",job="otlc"}
up{exported_instance="0.0.0.0:9999",instance="localhost:8888",job="otlc"}
up{exported_instance="0.0.0.0:1234",instance="localhost:8888",job="otlc"}
which this change accomplishes without having to inject any label
rewrites, but just by the new imports and upgrade of the prometheus
exporter.
Fixes open-telemetry/prometheus-interoperability-spec#8
Requires census-ecosystem/opencensus-go-exporter-prometheus#24
odeke-em
added a commit
to orijtech/opentelemetry-collector
that referenced
this issue
May 2, 2021
Make a receiver specific view that'll be registered
and used to record the "up" status either "0.0" or "1.0"
when an instance can't be scraped from or can be, respectively.
This ensures that the collector can act as a passthrough
for statuses and it currently outputs:
# HELP up Whether the endpoint is alive or not
# TYPE up gauge
up{instance="0.0.0.0:8888"} 1
up{instance="localhost:9999"} 0
I did not take the approach of plainly sending up suffixed metric names.
to recommend instead using relabelling inside the exporter itself like:
- source_labels: [__name__]
regex: "(.+)_up"
target_label: "__name__"
replacement: "up"
because:
* it'd apply ConstLabels on every *_up metric, only want "instance=$INSTANCE"
* other exporters wouldn't be able to use the "up" metric as is if we
inject rewrites
Regardless of if we used a label rewrite, the end result would be the
following:
up{instance="localhost:8888",job="otlc"}
up{exported_instance="0.0.0.0:9999",instance="localhost:8888",job="otlc"}
up{exported_instance="0.0.0.0:1234",instance="localhost:8888",job="otlc"}
which this change accomplishes without having to inject any label
rewrites, but just by the new imports and upgrade of the prometheus
exporter.
Fixes open-telemetry/prometheus-interoperability-spec#8
Requires census-ecosystem/opencensus-go-exporter-prometheus#24
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Implement the "up" metric for Prometheus.
Component: https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/prometheusreceiver
Related issue: open-telemetry/opentelemetry-specification#1078
The text was updated successfully, but these errors were encountered: