OCPBUGS-45924: Fix "live get after write" in static pod installer controller. #1929
Conversation
|
@benluddy: This pull request references Jira Issue OCPBUGS-45924, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/hold Getting some E2E signal while working on the tests... |
The static pod installer controller builds an apply configuration for the status of the nodes it manages based on potentially-stale state from an informer cache. The controller is written to assume that it has observed the effect of its own previous writes. If the cache is stale, this assumption can be violated which results in unpredictable installation decisions. A mechanism was recently introduced requiring the installer controller to wait for its lister to catch up to the latest version after performing a write. This mechanism did not work because it depended on keeping state across calls to the Sync method, and because Sync has a value receiver, field writes were not visible on subsequent calls.
314ad28 to
0fc0059
Compare
|
/cc @dgrisonnet @tkashem |
|
/lgtm |
|
/jira refresh |
|
@benluddy: This pull request references Jira Issue OCPBUGS-45924, which is valid. The bug has been moved to the POST state. 3 validation(s) were run on this bug
Requesting review from QA contact: In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@benluddy: all tests passed! Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: benluddy, dgrisonnet, tkashem The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/hold cancel I see this bug occur in practically every 4.19 E2E job, and I don't see it in the presubmits of openshift/cluster-etcd-operator#1392. |
|
@benluddy: Jira Issue OCPBUGS-45924: Some pull requests linked via external trackers have merged: The following pull requests linked via external trackers have not merged:
These pull request must merge or be unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with Jira Issue OCPBUGS-45924 has not been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/cherry-pick release-4.18 |
|
@benluddy: new pull request created: #1932 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
The static pod installer controller builds an apply configuration for the status of the nodes it manages based on potentially-stale state from an informer cache. The controller is written to assume that it has observed the effect of its own previous writes. If the cache is stale, this assumption can be violated which results in unpredictable installation decisions.
A mechanism was recently introduced requiring the installer controller to wait for its lister to catch up to the latest version after performing a write. This mechanism did not work because it depended on keeping state across calls to the Sync method, and because Sync has a value receiver, field writes were not visible on subsequent calls.