Prometheus metrics & mixin #375
Conversation
- `_unseal_requests_total` Counter is incremented every time the
`unseal()` function is called. Note that some Kubernetes events like
deleting a `SealedSecret` results in this function being called and
the counter incremented. Not bad though.
This Counter gives the best indiciation of controller activity.
- `_unseal_errors_total{reason="<reason>"}` Counter with labels for each
of the unseal failure cases; fetch, status, unmanaged, unseal, update
The default retry behaviour of 5 times in quick succession tends to
result in the Counter being incremented 5 times in event of a failure
and user action is required to rectify, eg: reseal, RBAC, etc.
- `build_info` with `revision` set to `VERSION` from git/tag
All Counters are initialized to 0 during init.
- Basic alerts that will fire if there are any unseal error counter increments. In my experience these all require user action. - Tests for alerts - Simple dashboard plotting total requests and errors. Could be extended with `build_info` panel/etc.
prometheus-mixin/Makefile
Outdated
| @@ -0,0 +1,56 @@ | |||
| # Prometheus Mixin Makefile | |||
There was a problem hiding this comment.
let's put prometheus-mixin in a contrib top level directory
There was a problem hiding this comment.
also please add a README.md file in this directory explaining what it is
|
unfortunately the tests fail because of the way multiarch docker images are currently built. the official docker ecosystem is often very automation unfriendly; there are alternative tools that will allow us to build images and manifests offline without having to push them to a registry; I'll try to play with that and regain the ability to run integration tests in unprivileged PRs |
|
@kskewes, I'm happy to merge this and apply further cleanups myself. Please clarify why you marked this PR as "WIP", do you plan pushing more changes later or it's just to signal that it's not "done" done? |
Just to signal it's not done done. Thanks for the quick feedback. |
Rebased Prometheus PR on master.
Closes: #177
/metrics.There aren't any sensitive endpoints on this port, but if metrics are considered sensitive for some then I can move to a custom port.
build_infoandgo_build_infometricsPodMonitorjsonnet for those using Prometheus Operator (us)Additional thoughts:
make testis currently failing for me on master as well as theprometheusbranch:--- FAIL: TestHttpCert (1.37s)Testing done:
kind