Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to set the value to helm release's timestamp #36

Merged
merged 2 commits into from
Jul 28, 2020

Conversation

knatsakis
Copy link
Contributor

@knatsakis knatsakis commented May 15, 2020

Recent versions of Grafana (I am testing with v6.7.1) have a "Series value as timestamp" option which can be used with the timestamp option implemented in this PR, in order to use the helm_chart_info metric for Grafana annotations pulled from Prometheus.

https://grafana.com/docs/grafana/latest/reference/annotations/#querying-other-data-sources

2020-05-16-022125_1270x940_scrot

@sstarcher
Copy link
Owner

If changing the value to a time series what would you be graphing in grafana? Do you have an example of what you are trying to accomplish?

@knatsakis
Copy link
Contributor Author

Do you have an example of what you are trying to accomplish?

2020-05-26-101146_3170x1110_scrot

So, what I am trying to achieve is deployment annotations in grafana, that use the metrics exported by helm-exporter.

In grafana the are two ways to achive this:

  1. By having the "Series value as timestamp" turned off and using a timeseries/metric like this:
null null null null 1 0 0 0 0 0
  1. By having the "Series value as timestamp" turned on and using a timeseries/metric like this:
null null null null 1590478033 1590478033 1590478033 1590478033 1590478033 1590478033

Where 1590478033 is the unix representation of the time the deployment occurred.

@knatsakis
Copy link
Contributor Author

Currently, helm-exporter is providing a timeseries/metric like this:

null null null null 1 1 1 1 1 1

This causes grafana to draw a vertical line for every minute, assuming that a new deployment has been occurring every minute.

I have found no PromQL query that can transform

null null null null 1 1 1 1 1 1

to

null null null null 1 0 0 0 0 0

I hope this helps explain the issue. If not I can provide more info. Thanks!

@brennoo
Copy link

brennoo commented May 26, 2020

we have an annotation for helm sync with count(helm_chart_info {release="$app"}) by (env,release) - 1

image

@sstarcher
Copy link
Owner

Excellent that's a great idea. So with your PR we have two modes.

  1. Metrics collected where the value is the status. This allows you to graph the health of a release.
  2. Metrics where the value is the timestamp. This allows you to put events onto nodes.

Would it make sense to not make them mutually exclusive? Maybe a default mode where metrics for both get generated and a flag to turn each of them off separately?

@knatsakis
Copy link
Contributor Author

@sstarcher, I don't think it makes much sense to have both options enabled simultaneously. I mean what is the point of having the same metrics twice with different values?

Is that something you think must be done in order to merge this PR? If not can you merge it?

@sstarcher
Copy link
Owner

But it's not the same metric. I would recommend renaming it. One allows you to track the status of a release. The one you added allows you to track release events. My main use case is tracking the current and prior status of a release. With the current code if I enable your option to track events I can no longer track release statuses.

This is needed in order to use the helm_chart_info metric for Grafana
annotations.
@knatsakis knatsakis force-pushed the knats-timestamp-in-value branch 3 times, most recently from fccf90a to 8341bc7 Compare July 27, 2020 14:07
statsInfo.WithLabelValues(chart, releaseName, version, appVersion, strconv.FormatInt(updated, 10), namespace, latestVersion).Set(status)
}
if *timestampSeries == true {
statsTimestamp.WithLabelValues(chart, releaseName, version, appVersion, strconv.FormatInt(updated, 10), namespace, latestVersion).Set(float64(updated))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we have two metrics do we need to duplicate all of the information? Or could statsInfo contain all of the information like it has had and statsTimestamp contains just the things needed to make it reference statsInfo such as release and namespace?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 you have requested that we make each metric optional, so what if a user only enables one of them?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ya, with my above suggestion making them optional would not really make sense. I think it would be best to have the existing one always one with statsTimestamp optional, but I'm also fine with how you have it now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll leave it as it is then.

- Add a second metric for timestamps
- Make info and timestamp metrics not mutually exclusive
- Add flags to turn off info and timestamp metrics
@knatsakis knatsakis force-pushed the knats-timestamp-in-value branch from 8341bc7 to b1ad066 Compare July 27, 2020 22:46
@knatsakis
Copy link
Contributor Author

@sstarcher I have spent some time testing this PR and it seems to be working nicely now (after fixing a minor naming bug).

Could you have another look and let me know if you would like anything else?

@sstarcher sstarcher merged commit f63863e into sstarcher:master Jul 28, 2020
@sstarcher
Copy link
Owner

Thanks for your contributions!

@sstarcher
Copy link
Owner

Bah, we should have updated the README.md to add info about the new metric.

@knatsakis knatsakis deleted the knats-timestamp-in-value branch July 28, 2020 19:31
sstarcher pushed a commit that referenced this pull request Oct 5, 2021
add option to set the value to helm release's timestamp
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants