Prometheus alert for pod crashlooping #21394
Conversation
|
The alert is currently set to be pretty sensitive, any pod restart would trigger it. Will adjust later if we feel it's too loud |
|
/cc @alvaroaleman |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: alvaroaleman, chaodaiG The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
| rules: [ | ||
| { | ||
| alert: 'prow-pod-crashlooping', | ||
| expr: 'rate(kube_pod_container_status_restarts_total{namespace=~"default|prow-monitoring",job="kube-state-metrics"}[5m]) * 60 * 5 > 0', |
There was a problem hiding this comment.
FYI for the future: Prometheus has a convenient increase(<metric>[<duration>]) function that can be used instead of rate(<metric>[<duration>]) * <duration in seconds>. It handles breaks in monotonicity just like rate().
Add alert catching pod crashlooping