Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rulers are never removed from the ring if evicted #2058

Closed
bboreham opened this issue Jan 31, 2020 · 1 comment · Fixed by #2587
Closed

rulers are never removed from the ring if evicted #2058

bboreham opened this issue Jan 31, 2020 · 1 comment · Fixed by #2587
Labels
component/rules Bits & bobs todo with rules and alerts: the ruler, config service etc. keepalive Skipped by stale bot

Comments

@bboreham
Copy link
Contributor

Basically the same problem as #1521, but rulers have no in-memory state to flush so it should be easier to ignore (or auto-forget) dead ones. Maybe we should worry a little about a split-brain state, but surely the worst case is doubling up evaluation which isn't fatal.

The error message is:

level=error ts=2020-01-31T09:13:05.403429734Z caller=ruler.go:362 msg="unable to verify rule group ownership ownership, will retry on the next poll" err="at least 1 live ingesters required, could only find 0"
@bboreham bboreham added the component/rules Bits & bobs todo with rules and alerts: the ruler, config service etc. label Jan 31, 2020
@stale
Copy link

stale bot commented Mar 31, 2020

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/rules Bits & bobs todo with rules and alerts: the ruler, config service etc. keepalive Skipped by stale bot
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants