Fix change notification of backend shard #835
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
HAProxy backends can be sharded into smaller pieces, so unchanged backends don't need to be rebuilt, reducing the time spent regenerating the configuration file, specially on deployments with thousands of backends. HAProxy Ingress controls the shards that should be rebuilt via the partial parsing implementation of ingress converter. Partial parsing tracks everything that need to be rebuilt, including backends. The internal HAProxy model controls the shards that should be rebuilt based on the backends tracked by the ingress converter. This means that changes made outside this controlled environment can be out of sync with configurations saved to disk, which is what HAProxy uses when it is restarted. This synchronization can be eventually fixed depending on new received events, sometimes this doesn't trigger HAProxy reloads, leading to the internal model in sync with disk, but not synced with the HAProxy instance. This out of sync is happening when HAProxy needs to be reloaded: all backends are iterated and new empty endpoints are added if needed. If the backend that received new endpoints wasn't tracked due to api changes, it is considered as a read only backend and its configuration file won't be updated. However the internal model has this new endpoint and will eventually use it, leading to a "No such server" from HAProxy client socket, which the controller was ignoring up to v0.12.5. This behavior is mitigated since v0.12.6 where unexpected responses triggers a reload with synced configuration files, and this update fixes a root cause of this behavior. This should be merged as far as v0.11.
8879ddc to
fc635bb
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
HAProxy backends can be sharded into smaller pieces, so unchanged backends don't need to be rebuilt, reducing the time spent regenerating the configuration file, specially on deployments with thousands of backends.
HAProxy Ingress controls the shards that should be rebuilt via the partial parsing implementation of ingress converter. Partial parsing tracks everything that need to be rebuilt, including backends. The internal HAProxy model controls the shards that should be rebuilt based on the backends tracked by the ingress converter. This means that changes made outside this controlled environment can be out of sync with configurations saved to disk, which is what HAProxy uses when it is restarted. This synchronization can be eventually fixed depending on new received events, sometimes this doesn't trigger HAProxy reloads, leading to the internal model in sync with disk, but not synced with the HAProxy instance.
This out of sync is happening when HAProxy needs to be reloaded: all backends are iterated and new empty endpoints are added if needed. If the backend that received new endpoints wasn't tracked due to api changes, it is considered as a read only backend and its configuration file won't be updated. However the internal model has this new endpoint and will eventually use it, leading to a "No such server" from HAProxy client socket, which the controller was ignoring up to v0.12.5.
This behavior is mitigated since v0.12.6 where unexpected responses triggers a reload with synced configuration files, and this update fixes a root cause of this behavior.
This should be merged as far as v0.11.
Related with #810
Should fix the root cause of #807