[stable/elasticsearch] fix cluster outage during master termination

[kimxogus]

What this PR does / why we need it:

This PR introduces announce service which points to each master pod and has cluster ip.
Master node will have network.publish_host as corresponding announce service's cluster ip, so that discovery service can access live ip address of terminated master.
With this PR, it seems cluster outage is gone. Http requests to client service responses immediately.

Which issue this PR fixes

(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged)

• fixes [stable/elasticsearch] Terminating current master pod causes cluster outage of more than 30 seconds #8785

Special notes for your reviewer:

Checklist

[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

• DCO signed
• Chart Version bumped
• Variables are documented in the README.md

[kimxogus]

/assign @desaintmartin

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: kimxogus
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: desaintmartin

If they are not already assigned, you can assign the PR to them by writing /assign @desaintmartin in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• stable/elasticsearch/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kimxogus]

/assign @rendhalver

[rendhalver]

Again I ask why are we effectively disabling the health checks to "fix" a service rather than fixing the broken health check?

[kimxogus]

master service is only for discovery service and I think it's irrelevant to success of http health check because discovery service starts before http health check succeeded.

[kimxogus]

I agree that we need to filter bad citizens.
Removed unready endpoints tolerations

[rendhalver]

/hold
This change will be unnecessary when we migrate to the elastic helm charts discussed here #10543

[kimxogus]

@rendhalver When I dived into #8785 and elastic/elasticsearch#36822 , I also tried https://github.com/elastic/helm-charts/tree/master/elasticsearch . and elastic's own helm chart had the same issue and it doesn't seem to be fixed in elastic's chart. I think this change is needed by elastic's helm chart too.

By the way, migrating elastic's charts is not just a moving helm repo, values.yaml should be rewritten, so there are many considerations needed to migrate elastic's chart like migrating with save pvs(it seems to be next priority) and there are some features that exists here should be added to elastic's chart such as lifecycle hook.

I don't think we can migrate to elastic's chart in near future, so before when we deprecate stable/elasticsearch chart, I don't think we should stop enhancing this chart.

[rendhalver]

Ok maybe we need to open an issue to track this and work out a fix for the problem.
I think effectively ignoring the health check isn't the right way to fix the problem.
Kubernetes relies on health checks and if we start ignoring them we are bad citizens.

The elastic helm-charts run two services which seems like a better way to solve the problem.
The run one service for cluster discovery and one service for writing to the cluster.

I realise migrating to the elastic/helm-charts won't be simple. We are going to write a migration plan to assist people to make the switch.

I would like to slow down development here and focus on working out how to migrate.

[kimxogus]

I agree that we can slow down development of this chart, and I hope I can contribute developing elastic helm chart, but I think this is a kind of major bugfix in docker environment which reduces cluster downtime from a couple of minutes to almost zero.

[desaintmartin]

IMHO we should start to synchronize changes we make on both charts. If the change is almost the same for both Charts, then I think it should be accepted here if it gets accepted in elastic/helm-charts.

This will allow easier switch when we have to switch.