Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend replication when instance heartbeat timeout #4493

Conversation

amoylan2
Copy link
Contributor

@amoylan2 amoylan2 commented Sep 23, 2021

Signed-off-by: lanmengran1 lanmengran1@jd.com

What this PR does:
Currently, there is an occasion that ingester extend writes not work. If an ingester instance is in Unhealthy state for som reason, where it's actually heartbeat timeout but stay in ACTIVE state in ring desc. In such case, extend writes will not work.
It is reasonable to treat the instance state as LEFT when calling ShouldExtendReplicaSetOnState

Which issue(s) this PR fixes:
Fixes #

Checklist

  • [NA] Tests updated
  • [NA] Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Signed-off-by: lanmengran1 <lanmengran1@jd.com>
@amoylan2 amoylan2 force-pushed the fix/extend_replication_when_instance_heartbeat_timeout branch from 9ae157c to 975d051 Compare September 28, 2021 10:17
@amoylan2 amoylan2 marked this pull request as draft September 28, 2021 10:34
@amoylan2
Copy link
Contributor Author

I want to know is this a real problem? If not I will close it then

@bboreham
Copy link
Contributor

bboreham commented Oct 1, 2021

Hi, thanks for your PR; unfortunately it is against the design of Cortex.

The reasons we don't do anything like this are:

  • each timeseries has a significant amount of metadata, so we want to keep the set of timeseries in one ingester stable over time.
  • we already allow redundancy, e.g. write to 3 ingesters but succeed if 1 is down.

It may be that you are trying to address the issue at #1521; better to join in there before contributing code.

@amoylan2
Copy link
Contributor Author

Thanks for your reviewing.
I'm closing this.

@amoylan2 amoylan2 closed this Oct 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants