OCPBUGS-48523: Bump to Kubernetes 1.32 and latest library-go. #1392

benluddy · 2025-01-29T15:58:07Z

This is motivated by the installer controller fix in library-go, which has already moved to Kubernetes 1.32.

benluddy · 2025-01-29T15:58:12Z

/hold

benluddy · 2025-01-29T15:58:30Z

xref openshift/library-go#1929

benluddy · 2025-01-29T18:29:29Z

/retest-required

benluddy · 2025-01-29T20:52:55Z

/retest-required

benluddy · 2025-01-29T21:07:36Z

/cc @hasbro17

The library-go Kubernetes 1.32 bump merged to master, so I was forced to pull that in to get openshift/library-go#1929. This came along with a Go version bump in go.mod, so I was also forced to bump to the 4.19 base image.

benluddy · 2025-01-29T21:08:15Z

/hold cancel

The fix has merged into library-go so this is no longer pointing at my library-go fork.

benluddy · 2025-01-29T22:05:00Z

This "slow backup" test appears broken by https://tip.golang.org/doc/go1.23#timer-changes, looking into it...

Elbehery · 2025-01-29T22:14:29Z

This "slow backup" test appears broken by https://tip.golang.org/doc/go1.23#timer-changes, looking into it...

fixing ... 👍🏽

hasbro17 · 2025-01-29T22:25:44Z

@benluddy In the interest of getting the library-go bump (unless you've already figured it out) you can add a t.Skip() for TestNewBackupServer_scheduleBackup.
The backupserver is still an ongoing tech preview development for periodic backups that doesn't impact us in GA.

@Elbehery Can you please fix the unit test as a follow up to this PR.

Elbehery · 2025-01-29T22:29:26Z

@benluddy In the interest of getting the library-go bump (unless you've already figured it out) you can add a t.Skip() for TestNewBackupServer_scheduleBackup. The backupserver is still an ongoing tech preview development for periodic backups that doesn't impact us in GA.

@Elbehery Can you please fix the unit test as a follow up to this PR.

yes, raising a fix now 👍🏽

this change is the cause

Second, the timer channel associated with a Timer or Ticker is now unbuffered, with capacity 0. The main effect of this change is that Go now guarantees that for any call to a Reset or Stop method, no stale values prepared before that call will be sent or received after the call. Earlier versions of Go used channels with a one-element buffer, making it difficult to use Reset and Stop correctly. A visible effect of this change is that len and cap of timer channels now returns 0 instead of 1, which may affect programs that poll the length to decide whether a receive on the timer channel will succeed. Such code should use a non-blocking receive instead.

Elbehery · 2025-01-29T22:34:43Z

raised #1393

cc @hasbro17 @benluddy

hasbro17 · 2025-01-29T22:36:09Z

Ah okay so literally an off by 1 change. I'll leave it up to Ben to either pull that in or skip.

benluddy · 2025-01-30T14:39:53Z

#1393 has nearly merged, I'll watch it and rebase this on top.

Elbehery · 2025-01-30T15:41:59Z

#1393 has nearly merged, I'll watch it and rebase this on top.

@benluddy its merged 👍🏽

This is motivated by the installer controller fix in library-go, which has already moved to Kubernetes 1.32.

Elbehery · 2025-01-30T17:13:28Z

@hasbro17 the fix has been merged, and the CI is running now

leaving this 4 u to tag 👍🏽

benluddy · 2025-01-30T17:23:48Z

/retitle OCPBUGS-48523: Bump to Kubernetes 1.32 and latest library-go.

openshift-ci-robot · 2025-01-30T17:23:59Z

@benluddy: This pull request references Jira Issue OCPBUGS-48523, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.19.0) matches configured target version for branch (4.19.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @geliu2016

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

This is motivated by the installer controller fix in library-go, which has already moved to Kubernetes 1.32.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

hasbro17 · 2025-01-30T17:45:08Z

/lgtm
/label acknowledge-critical-fixes-only

openshift-ci · 2025-01-30T17:45:57Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: benluddy, hasbro17

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [hasbro17]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2025-01-30T19:33:45Z

/retest-required

Remaining retests: 0 against base HEAD b23ae82 and 2 for PR HEAD 82d72cb in total

benluddy · 2025-01-30T20:54:12Z

/retest-required

The last run of e2e-aws-ovn-serial passed before timing out during deprovisioning.

Elbehery · 2025-01-31T08:33:52Z

/retest-required

Elbehery · 2025-01-31T08:34:33Z

same e2e operator job failure as in #1391

it just passed

retesting ...

openshift-ci-robot · 2025-01-31T11:33:33Z

/retest-required

Remaining retests: 0 against base HEAD c956ac3 and 1 for PR HEAD 82d72cb in total

openshift-ci-robot · 2025-01-31T13:34:09Z

/retest-required

Remaining retests: 0 against base HEAD c956ac3 and 2 for PR HEAD 82d72cb in total

Elbehery · 2025-01-31T15:23:10Z

e2e operator job fails on a flaky test

=== RUN   TestEtcdctlCommands/etcdctl_defrag
    etcdctl_test.go:96: 
        	Error Trace:	/go/src/github.com/openshift/cluster-etcd-operator/test/e2e/etcdctl_test.go:96
        	Error:      	Received unexpected error:
        	            	exit status 1
        	Test:       	TestEtcdctlCommands/etcdctl_defrag
        	Messages:   	output was {"level":"warn","ts":"2025-01-31T14:52:30.430068Z","logger":"etcd-client","caller":"v3@v3.5.17/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000466000/10.0.17.68:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
        	            	Failed to defragment etcd member[https://10.0.17.68:2379] (context deadline exceeded)
        	            	Finished defragmenting etcd member[https://10.0.41.156:2379]
        	            	Finished defragmenting etcd member[https://10.0.83.181:2379]
        	            	command terminated with exit code 1
    --- FAIL: TestEtcdctlCommands/etcdctl_defrag (7.70s)

: Run multi-stage test e2e-operator - e2e-operator-test container test expand_less | 28m26s
-- | --
{  igrate SKIP: github.com/openshift/cluster-etcd-operator/test/e2e TestEtcdctlCommands/etcdctl_role_add SKIP: github.com/openshift/cluster-etcd-operator/test/e2e TestEtcdctlCommands/etcdctl_role_delete SKIP: github.com/openshift/cluster-etcd-operator/test/e2e TestEtcdctlCommands/etcdctl_role_get SKIP: github.com/openshift/cluster-etcd-operator/test/e2e TestEtcdctlCommands/etcdctl_role_grant-permission SKIP: github.com/openshift/cluster-etcd-operator/test/e2e TestEtcdctlCommands/etcdctl_role_list SKIP: github.com/openshift/cluster-etcd-operator/test/e2e TestEtcdctlCommands/etcdctl_role_revoke-permission SKIP: github.com/openshift/cluster-etcd-operator/test/e2e TestEtcdctlCommands/etcdctl_user_add SKIP: github.com/openshift/cluster-etcd-operator/test/e2e TestEtcdctlCommands/etcdctl_user_delete SKIP: github.com/openshift/cluster-etcd-operator/test/e2e TestEtcdctlCommands/etcdctl_user_get SKIP: github.com/openshift/cluster-etcd-operator/test/e2e TestEtcdctlCommands/etcdctl_user_grant-role SKIP: github.com/openshift/cluster-etcd-operator/test/e2e TestEtcdctlCommands/etcdctl_user_list SKIP: github.com/openshift/cluster-etcd-operator/test/e2e TestEtcdctlCommands/etcdctl_user_revoke-role SKIP: github.com/openshift/cluster-etcd-operator/test/e2e TestEtcdctlCommands/etcdctl_move-leader_8e9e05c52164694d SKIP: github.com/openshift/cluster-etcd-operator/test/e2e TestEtcdctlCommands/etcdctl_check_datascale SKIP: github.com/openshift/cluster-etcd-operator/test/e2e TestEtcdctlCommands/etcdctl_txn FAIL: github.com/openshift/cluster-etcd-operator/test/e2e TestEtcdctlCommands 41.53s make: *** [vendor/github.com/openshift/build-machinery-go/make/targets/golang/test-unit.mk:12: test-unit] Error 1 {"component":"entrypoint","error":"wrapped process failed: exit status 2","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:84","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.internalRun","level":"error","msg":"Error executing test process","severity":"error","time":"2025-01-31T15:03:52Z"} error: failed to execute wrapped command: exit status 2 }

I ran the test yesterday manually against a cluster and it passed from first run #1391 (comment)

overriding the failing test ...

Elbehery · 2025-01-31T15:24:45Z

The last failing prow https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-etcd-operator/1392/pull-ci-openshift-cluster-etcd-operator-master-e2e-operator/1885320815512129536

/override ci/prow/e2e-operator

openshift-ci · 2025-01-31T15:24:59Z

@Elbehery: Overrode contexts on behalf of Elbehery: ci/prow/e2e-operator

In response to this:

The last failing prow https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-etcd-operator/1392/pull-ci-openshift-cluster-etcd-operator-master-e2e-operator/1885320815512129536

/override ci/prow/e2e-operator

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci-robot · 2025-01-31T18:02:07Z

/retest-required

Remaining retests: 0 against base HEAD c956ac3 and 2 for PR HEAD 82d72cb in total

Elbehery · 2025-01-31T21:41:15Z

/retest-required

openshift-ci · 2025-02-01T01:18:31Z

@benluddy: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-operator-fips	`82d72cb`	link	false	`/test e2e-operator-fips`
ci/prow/e2e-aws-etcd-recovery	`82d72cb`	link	false	`/test e2e-aws-etcd-recovery`
ci/prow/okd-scos-e2e-aws-ovn	`82d72cb`	link	false	`/test okd-scos-e2e-aws-ovn`
ci/prow/e2e-aws-etcd-certrotation	`82d72cb`	link	false	`/test e2e-aws-etcd-certrotation`
ci/prow/e2e-metal-ovn-ha-cert-rotation-shutdown	`82d72cb`	link	false	`/test e2e-metal-ovn-ha-cert-rotation-shutdown`
ci/prow/e2e-metal-ovn-sno-cert-rotation-shutdown	`82d72cb`	link	false	`/test e2e-metal-ovn-sno-cert-rotation-shutdown`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Elbehery · 2025-02-01T08:00:24Z

/retest-required

Elbehery · 2025-02-01T20:51:32Z

/retest-required

openshift-ci-robot · 2025-02-01T23:37:04Z

@benluddy: Jira Issue OCPBUGS-48523: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-48523 has been moved to the MODIFIED state.

In response to this:

This is motivated by the installer controller fix in library-go, which has already moved to Kubernetes 1.32.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Elbehery · 2025-02-01T23:51:47Z

/retest-required

@benluddy Weekends are best to get things merged ;)

openshift-bot · 2025-02-02T01:39:29Z

[ART PR BUILD NOTIFIER]

Distgit: cluster-etcd-operator
This PR has been included in build cluster-etcd-operator-container-v4.19.0-202502020107.p0.g4c27d6f.assembly.stream.el9.
All builds following this will include this PR.

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 29, 2025

openshift-ci bot requested review from dusk125 and Elbehery January 29, 2025 15:59

benluddy mentioned this pull request Jan 29, 2025

OCPBUGS-45924: Fix "live get after write" in static pod installer controller. openshift/library-go#1929

Merged

benluddy force-pushed the bump-kube-1.32-and-library-go branch from a0f935e to 3f19a58 Compare January 29, 2025 21:07

openshift-ci bot requested a review from hasbro17 January 29, 2025 21:07

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 29, 2025

benluddy added 2 commits January 30, 2025 11:04

Bump to Kubernetes 1.32 and latest library-go.

a4c5f84

This is motivated by the installer controller fix in library-go, which has already moved to Kubernetes 1.32.

Update base image for 4.19.

82d72cb

benluddy force-pushed the bump-kube-1.32-and-library-go branch from 3f19a58 to 82d72cb Compare January 30, 2025 16:04

openshift-ci bot changed the title ~~Bump to Kubernetes 1.32 and latest library-go.~~ OCPBUGS-48523: Bump to Kubernetes 1.32 and latest library-go. Jan 30, 2025

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels Jan 30, 2025

openshift-ci bot requested a review from geliu2016 January 30, 2025 17:24

openshift-ci bot added the acknowledge-critical-fixes-only Indicates if the issuer of the label is OK with the policy. label Jan 30, 2025

openshift-ci bot assigned hasbro17 Jan 30, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jan 30, 2025

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 30, 2025

openshift-merge-bot bot merged commit 4c27d6f into openshift:master Feb 1, 2025
12 of 18 checks passed

OCPBUGS-48523: Bump to Kubernetes 1.32 and latest library-go. #1392

OCPBUGS-48523: Bump to Kubernetes 1.32 and latest library-go. #1392

Conversation

benluddy commented Jan 29, 2025

benluddy commented Jan 29, 2025

benluddy commented Jan 29, 2025

benluddy commented Jan 29, 2025

benluddy commented Jan 29, 2025

benluddy commented Jan 29, 2025

benluddy commented Jan 29, 2025

benluddy commented Jan 29, 2025

Elbehery commented Jan 29, 2025

hasbro17 commented Jan 29, 2025

Elbehery commented Jan 29, 2025

Elbehery commented Jan 29, 2025

hasbro17 commented Jan 29, 2025

benluddy commented Jan 30, 2025

Elbehery commented Jan 30, 2025

Elbehery commented Jan 30, 2025

benluddy commented Jan 30, 2025

openshift-ci-robot commented Jan 30, 2025

hasbro17 commented Jan 30, 2025

openshift-ci bot commented Jan 30, 2025

openshift-ci-robot commented Jan 30, 2025

benluddy commented Jan 30, 2025 • edited Loading

Elbehery commented Jan 31, 2025

Elbehery commented Jan 31, 2025

openshift-ci-robot commented Jan 31, 2025

openshift-ci-robot commented Jan 31, 2025

Elbehery commented Jan 31, 2025

Elbehery commented Jan 31, 2025

openshift-ci bot commented Jan 31, 2025

openshift-ci-robot commented Jan 31, 2025

Elbehery commented Jan 31, 2025

openshift-ci bot commented Feb 1, 2025 • edited Loading

Elbehery commented Feb 1, 2025

Elbehery commented Feb 1, 2025

openshift-ci-robot commented Feb 1, 2025

Elbehery commented Feb 1, 2025

openshift-bot commented Feb 2, 2025

benluddy commented Jan 30, 2025 •

edited

Loading

openshift-ci bot commented Feb 1, 2025 •

edited

Loading