clear pod's .status.nominatedNodeName when necessary #106816

Huang-Wei · 2021-12-04T20:13:16Z

What type of PR is this?

/kind bug
/sig scheduling

What this PR does / why we need it:

Clear a pod's .status.nominatedNodeName when needed.

Which issue(s) this PR fixes:

Fixes #106780 #85677

Special notes for your reviewer:

We may consider back-porting this as it's a bug fix.

Does this PR introduce a user-facing change?

Fixed a bug that a pod's .status.nominatedNodeName is not cleared properly, and thus over-occupied system resources.

k8s-ci-robot · 2021-12-04T20:13:34Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Huang-Wei

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/scheduler/OWNERS~~ [Huang-Wei]
~~test/integration/scheduler/OWNERS~~ [Huang-Wei]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Huang-Wei · 2021-12-04T20:13:55Z

cc @alculquicondor @mateuszlitwin

Huang-Wei · 2021-12-04T20:26:28Z

test/integration/scheduler/preemption_test.go

+			},
+			// Delete the node to simulate an ErrNoNodesAvailable error.
+			deleteNode:       true,
+			podNamesToDelete: []string{"low"},


In a real cluster, this is actually optional.

However, in an integration test it may flake without this - a node deletion event wouldn't trigger moving the preemptor back to activeQ, so if the preemptor happens to be backoffed and the 30 seconds timeout may not be enough. So add a pod deletion here to actively trigger retrying the preemptor.

alculquicondor · 2021-12-07T21:07:04Z

pkg/scheduler/scheduler.go

 			// No nodes available is counted as unschedulable rather than an error.
 			metrics.PodUnschedulable(fwk.ProfileName(), metrics.SinceInSeconds(start))
 		} else {
+			nominatedNode = pointer.StringPtr("")


in theory, this is the case of a transient error. Perhaps we shouldn't clear the nominated node here?

It's a bit debatable depends how we interpret this kind of error. If we believe it can be auto-healed, we shouldn't clear it; otherwise clear it to prevent the pod from occupying resources.

I don't have a strong option as personally, I don't have real-world data points to determine if it's transient and how transient it is... I'm giving a pessimistic judgment here :(

I also don't have a strong opinion. But I think that if the pod really needs preemption, it would get a nominated node in the next scheduling attempt. So it seems safer for utilization to clear.

pkg/scheduler/framework/preemption/preemption.go

alculquicondor · 2021-12-07T21:14:37Z

pkg/scheduler/scheduler.go

 	sched.Error(podInfo, err)

 	// Update the scheduling queue with the nominated pod information. Without
 	// this, there would be a race condition between the next scheduling cycle
 	// and the time the scheduler receives a Pod Update for the nominated pod.
 	// Here we check for nil only for tests.
 	if sched.SchedulingQueue != nil {
-		sched.SchedulingQueue.AddNominatedPod(podInfo.PodInfo, nominatedNode)
+		var nnn string


It looks like nil and empty string have the same result here. Why use a pointer?

Ah, it's more about updatePod below.

However, it looks like AddNominatedPod looks at the existing nominatedNode in the status when you pass empty string. This means that the nominated node name is not actually cleared from the cache.

Ah, it's more about updatePod below.

Yes.

However, it looks like AddNominatedPod looks at the existing nominatedNode in the status when you pass empty string.

Not quite. The entry in internal nominator implementation will be cleared first in all cases:

kubernetes/pkg/scheduler/internal/queue/scheduling_queue.go

Lines 834 to 845 in 4cc1abf

func (npm *nominator) add(pi *framework.PodInfo, nodeName string) {

// always delete the pod if it already exist, to ensure we never store more than

// one instance of the pod.

npm.delete(pi.Pod)

nnn := nodeName

if len(nnn) == 0 {

nnn = NominatedNodeName(pi.Pod)

if len(nnn) == 0 {

return

}

}

I think the problem is this line:

kubernetes/pkg/scheduler/internal/queue/scheduling_queue.go

Line 841 in 4cc1abf

nnn = NominatedNodeName(pi.Pod)

pkg/scheduler/scheduler.go

alculquicondor · 2021-12-07T21:41:21Z

pkg/scheduler/scheduler.go

 	sched.Error(podInfo, err)

 	// Update the scheduling queue with the nominated pod information. Without
 	// this, there would be a race condition between the next scheduling cycle
 	// and the time the scheduler receives a Pod Update for the nominated pod.
 	// Here we check for nil only for tests.
 	if sched.SchedulingQueue != nil {
-		sched.SchedulingQueue.AddNominatedPod(podInfo.PodInfo, nominatedNode)
+		var nnn string


Ah, it's more about updatePod below.

However, it looks like AddNominatedPod looks at the existing nominatedNode in the status when you pass empty string. This means that the nominated node name is not actually cleared from the cache.

pkg/scheduler/framework/preemption/preemption.go

alculquicondor · 2021-12-09T20:39:16Z

pkg/scheduler/internal/queue/scheduling_queue.go

@@ -692,7 +693,7 @@ func (npm *nominator) DeleteNominatedPodIfExists(pod *v1.Pod) {
 // This is called during the preemption process after a node is nominated to run
 // the pod. We update the structure before sending a request to update the pod
 // object to avoid races with the following scheduling cycles.
-func (npm *nominator) AddNominatedPod(pi *framework.PodInfo, nodeName string) {
+func (npm *nominator) AddNominatedPod(pi *framework.PodInfo, nodeName *string) {


Document the semantics of nil vs empty string.

I updated to compare nil vs. non-nil. PTAL.

alculquicondor · 2021-12-09T20:39:42Z

pkg/scheduler/internal/queue/scheduling_queue.go

-		if len(nnn) == 0 {
-			return
-		}
+	if nodeName == nil {


so nil removes the nominated node name?
Wasn't that supposed to be empty string?

In the before, if an empty string is passed in, we always look at and honor the nominatedNodeName if it carries. Now the logic gets tweaked:

if you want to follow the old logic, on the caller side, use adaptiveNominatedNodeName() to always honor nominatedNodeName

if you want to enforce overriding, pass in a non-nil string pointer (no matter it's empty or a concrete string).

I'm not liking adaptiveNominatedNodeName, as it could be forgotten. I think it should be handled within this function: if passing nil, leave; otherwise override

if passing nil, leave; otherwise override

Passing the string pointer still needs to be decided at the caller side - whether it's a pod with empty nominatedNodeName, so we interpreted it as nil; or, the caller have baked the string pointer properly (like recordSchedulingFailure()).

If we go handle the adaptiveNominatedNodeName logic in the function, if a user just pass in a pod.status.nominatedNodeName, inside the function, we don't have enough clues to know the exact intents.

Here is my proposal:

nil means do not change anything. Then, we would query the nominatedNodeName from the pod status.

"" means clear

"foo" is override

I think that logic can be handled here.

It makes sense to me and can save the efforts on the caller side.

One question on "" means clear: to keep the behavior with before, "" and "regular string" are treated the same. If we really want to clear the entry, we can do it in a follow-up.

I think distinguishing between nil and empty is asking for trouble. I also didn't like the old code that implicitly treated an empty nodeName as a fallback to the value in the pod. Lots of implicit behavior that one can't figure out without carefully parsing the code.

Not sure if this was the old code, but why not simply have the following:

// Call the following in the places were you are currently passing in nil. add(pi) { addToNode(pi, pi.Pod.Status.NominatedNodeName) }

// Alway adds the pod to the passed nodeName addToNode(pi, nodeName) { ... }

Not sure if this was the old code, but why not simply have the following

This is basically the previous version except for that some places needs to inject NominatedNodeName adapatively.

ahg-g · 2021-12-10T21:37:29Z

pkg/scheduler/framework/preemption/preemption.go

+//
+// - <PostFilterResult{}, Unschedulable>. It indicates the pod cannot be scheduled even with preemption.
+//   In this case, an empty PostFilterResult is returned to clear the pod's nominatedNodeName (if it carries).
+// - <PostFilterResult{non-empty nominatedNodeName}, Unschedulable>. It's the regular happy path


this should be success, not unschedulable.

Typo.. Will update along with further changes.

ahg-g · 2021-12-10T21:39:51Z

pkg/scheduler/framework/preemption/preemption.go

+// - <nil, Unschedulable>. This status is mostly as expected like the preemptor is waiting for the
+//   victims to be fully terminated.
+// - In both cases above, a nil PostFilterResult is returned to keep the pod's nominatedNodeName unchanged.
+//
+// - <PostFilterResult{}, Unschedulable>. It indicates the pod cannot be scheduled even with preemption.


The distinction between nil and empty is an anti-pattern and confusing. I suggest two options:

Lets add a field to PostFilterResult to explicitly express the distinction between nil and empty you are having now.

instead of returning Unschedulable, return UnschedulableUnresolvable in the case where we are returning an empty result

I thought of option 1, but it's sort of non-trivial work. Let me think about it.

ahg-g · 2021-12-10T22:03:29Z

pkg/scheduler/internal/queue/scheduling_queue.go

-		if len(nnn) == 0 {
-			return
-		}
+	if nodeName == nil {


I think distinguishing between nil and empty is asking for trouble. I also didn't like the old code that implicitly treated an empty nodeName as a fallback to the value in the pod. Lots of implicit behavior that one can't figure out without carefully parsing the code.

Not sure if this was the old code, but why not simply have the following:

// Call the following in the places were you are currently passing in nil. add(pi) { addToNode(pi, pi.Pod.Status.NominatedNodeName) }

// Alway adds the pod to the passed nodeName addToNode(pi, nodeName) { ... }

ahg-g · 2021-12-10T22:09:39Z

pkg/scheduler/framework/interface.go

+	// Notes for the semantics of `nodeName`:
+	// - if it's nil, the backing nominator aborts if the pod doesn't carry a nominatedNodeName; otherwise
+	//   sets the carried nominatedNodeName.
+	// - if it's non-nil, the internal nominating bits will be updated with the given `nodeName`.


Suggested change

// - if it's non-nil, the internal nominating bits will be updated with the given `nodeName`.

// - if it's non-nil, the internal cache will be updated with the given `nodeName`.

ahg-g · 2021-12-10T22:23:05Z

pkg/scheduler/framework/interface.go

+	// - if it's nil, the backing nominator aborts if the pod doesn't carry a nominatedNodeName; otherwise
+	//   sets the carried nominatedNodeName.
+	// - if it's non-nil, the internal nominating bits will be updated with the given `nodeName`.
+	AddNominatedPod(pod *PodInfo, nodeName *string)


Overloading the parameter nodeName to signal more than simply the nodeName is problematic, I suggest making the interface explicit. Here are some options:

Is there really a legitimate case where the nodeName is different from the one set in the passed pod if one is set? If so, I think we need to revisit that because I don't think there should be one.

If we fail with option 1, I think this function should have consistent semantics and always adds pod to the passed nodeName, no exceptions and fail on an empty nodeName. The caller of this function can do
use a wrapper function as follows (not part of this interface):

func AddNominatedPodIfSet(pod) { if pi.Pod.Status.NominatedNodeName != "" { AddNominatedPod(pod, pi.Pod.Status.NominatedNodeName) } }

Re 1: yes, it can be empty in recordSchedulingFailure() to clear the entry, or UpdateNominatedPod() which reveals a special case that right after a preemptor is marked as nominated in memory, and at the same time a podUpdate event (doesn't carry nominatedNodeName) comes in.

Re 2: the wrapper can help refactor the code for better readability. Thanks.

Huang-Wei · 2021-12-12T05:46:09Z

I appended a commit to use NominatingInfo to simply the manipulation on nominatedNodeName. @ahg-g @alculquicondor PTAL.

alculquicondor · 2021-12-15T16:47:21Z

pkg/scheduler/framework/interface.go

 // PostFilterResult wraps needed info for scheduler framework to act upon PostFilter phase.
 type PostFilterResult struct {
-	NominatedNodeName string
+	*NominatingInfo


Does it need to be a pointer?

Not mandatory. But it can simplify the code on caller side. (just pass in a nil to indicate its mode is Noop)

I think that's good when passing NominatingInfo to a function.

But when returning a Noop result, we are just using PostFilterResult=nil, no?

But when returning a Noop result, we are just using PostFilterResult=nil, no?

A Noop result equals a nil NominatingInfo, but not quite equivalent to PostFilterResult as NominatingInfo is a sub-field of PostFilterResult.

What I'm saying is that you are already using PostFilterResult=nil to signal Noop, so I don't see much point on having a pointer here. However, it's such a minor detail that I don't feel strongly about it.

I see. Given a pointer NominatingInfo can be carried over directly to other places (like recordSchedulingFailure()) that need a pointer, I'm inclined to keep it as-is.

pkg/scheduler/framework/interface.go

pkg/scheduler/framework/plugins/defaultpreemption/default_preemption_test.go

pkg/scheduler/framework/interface.go

pkg/scheduler/framework/runtime/framework.go

alculquicondor · 2021-12-15T17:12:04Z

pkg/scheduler/internal/queue/scheduling_queue.go

 	// We won't fall into below `if` block if the Update event represents:
 	// (1) NominatedNode info is added
 	// (2) NominatedNode info is updated
 	// (3) NominatedNode info is removed
 	if NominatedNodeName(oldPod) == "" && NominatedNodeName(newPodInfo.Pod) == "" {
 		if nnn, ok := npm.nominatedPodToNode[oldPod.UID]; ok {
 			// This is the only case we should continue reserving the NominatedNode
-			nodeName = nnn
+			nominatingInfo = &framework.NominatingInfo{


Use the New function

I didn't add a New functin for NominatingInfo. The NewPostFilterResultWithNominatedNode is for simplifying instantiation of PostFilterResult.

alculquicondor · 2021-12-15T17:23:41Z

test/integration/scheduler/preemption_test.go


-	cs := testCtx.ClientSet
+type alwaysFail struct{}


Add a comment for the functionality of this plugin

alculquicondor · 2021-12-15T17:27:13Z

test/integration/scheduler/preemption_test.go

+		podNamesToDelete []string
+
+		// Register dummy plugin to simulate particular scheduling failures. Optional.
+		outOfTreePlugins  *v1beta3.Plugins


customPlugins

alculquicondor · 2021-12-15T17:32:53Z

test/integration/scheduler/preemption_test.go

+				if err != nil {
+					t.Errorf("Error getting the medium pod: %v", err)
+				}
+				if len(pod.Status.NominatedNodeName) == 0 {


should we have test cases for when the nnn is not cleared?

Alternatively, instead of special casing "medium", is it worth having a map of the expected nnn or assigned nodes for all the pods not removed? Then you can do a cmp.Diff.

probably, but I won't bother here until we have a valid requirement in the future.

should we have test cases for when the nnn is not cleared?

We have e2e test covers this, like PreemptionExecutionPath. But I'm going to add an integration test for this specifically.

I did add a new integration test, but it needs to consistently verify if the pod's nomiantedNode is NOT cleared. It adds up the test time by 15 Seconds * 2 (the first check is to ensure an upcoming low-priority doesn't get scheduled, the second check is to ensure the preemptor's nnn is not cleared).

Uhm... sad that there is no way to know if there was a scheduling attempt that includes all the pod/node updates.

Maybe we shouldn't commit a test that is expected to stay idle for 15 seconds?

Which leave us with an untested path, but we still have the e2e and unit tests. WDYT? Sorry for the back and forth.

Maybe we shouldn't commit a test that is expected to stay idle for 15 seconds?

That's my concern as well.

Which leave us with an untested path, but we still have the e2e and unit tests. WDYT?

I think we can remove this integration test. The e2e test PreemptionExecutionPath has ensured the same path coverage.

alculquicondor · 2021-12-16T14:11:20Z

test/integration/scheduler/preemption_test.go

+			err := wait.Poll(100*time.Millisecond, 15*time.Second, func() (bool, error) {
+				pod, err := cs.CoreV1().Pods(ns).Get(context.TODO(), "medium", metav1.GetOptions{})
+				if err != nil {
+					// t.Errorf("Error getting the medium pod: %v", err)


reverted the most recent integration test and this should be gone.

alculquicondor · 2021-12-16T14:57:51Z

test/integration/scheduler/preemption_test.go

+				if err != nil {
+					t.Errorf("Error getting the medium pod: %v", err)
+				}
+				if len(pod.Status.NominatedNodeName) == 0 {


Uhm... sad that there is no way to know if there was a scheduling attempt that includes all the pod/node updates.

Maybe we shouldn't commit a test that is expected to stay idle for 15 seconds?

Which leave us with an untested path, but we still have the e2e and unit tests. WDYT? Sorry for the back and forth.

alculquicondor · 2021-12-16T19:06:59Z

/lgtm

ahg-g · 2021-12-16T20:11:40Z

/lgtm

thanks Wei, the explicit api for how to deal with nominatedNodeName feels more readable and easier to reason about.

Huang-Wei · 2021-12-16T20:56:20Z

/retest

Thanks @alculquicondor and @ahg-g for pushing me on the readability refactoring :)

Given this is going to benefit CA (#85677) and scheduler plugins (#106780), I'm going to back port this fix if no objections.

alculquicondor · 2021-12-16T21:04:26Z

SGTM

Huang-Wei · 2021-12-16T21:17:33Z

Interesting, the log (https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/106816/pull-kubernetes-integration/1471554622542843904) says all tests passed but the CI job failed.

/retest

k8s-triage-robot · 2021-12-17T02:14:23Z

The Kubernetes project has merge-blocking tests that are currently too flaky to consistently pass.

This bot retests PRs for certain kubernetes repos according to the following rules:

The PR does have any do-not-merge/* labels
The PR does not have the needs-ok-to-test label
The PR is mergeable (does not have a needs-rebase label)
The PR is approved (has cncf-cla: yes, lgtm, approved labels)
The PR is failing tests required for merge

You can:

Review the full test history for this PR
Prevent this bot from retesting with /lgtm cancel or /hold
Help make our tests less flaky by following our Flaky Tests Guide

/retest

cpanato · 2022-01-07T09:40:15Z

/triage accepted
/priority important-soon

cpanato · 2022-01-07T09:41:44Z

@Huang-Wei @ahg-g this fix is only needed on the release-1.23 branch? not in 1.22/1.21/1.20? thanks

mateuszlitwin · 2022-01-07T17:46:30Z

I think 1.22 and 1.21 would benefit - https://github.com/kubernetes-sigs/scheduler-plugins/releasesis affected by this is on 1.21.

ahg-g · 2022-01-07T17:58:57Z

I find the fix too large to patch back.

btw, I think this issue existed for a long time, it is not a new bug.

alculquicondor · 2022-01-07T19:07:39Z

you mean, to patch back beyond 1.23, right?

+1 to just backporting to 1.23

ahg-g · 2022-01-07T19:22:24Z

yeah, the patch to 1.23 should probably be automatic, but beyond that it will likely face conflicts.

cpanato · 2022-01-08T10:56:40Z

thanks for the update and clarifications

Huang-Wei · 2022-01-08T18:50:08Z

@ahg-g's comment is valid, and that's why I hesitated to backport to 1.21/1.22, instead only raised #107109 for 1.23.

…06816-upstream-release-1.23 Automated cherry pick of #106816: clear pod's .status.nominatedNodeName when necessary

k8s-ci-robot requested review from chendave and damemi December 4, 2021 20:13

k8s-ci-robot added sig/testing Categorizes an issue or PR as relevant to SIG Testing. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Dec 4, 2021

Huang-Wei commented Dec 4, 2021

View reviewed changes

Huang-Wei force-pushed the fix-nnn-not-cleared branch from 374db93 to 612b69b Compare December 5, 2021 00:33

alculquicondor reviewed Dec 7, 2021

View reviewed changes

Huang-Wei force-pushed the fix-nnn-not-cleared branch from 83201fb to ba9c8c7 Compare December 8, 2021 19:06

alculquicondor reviewed Dec 9, 2021

View reviewed changes

Huang-Wei force-pushed the fix-nnn-not-cleared branch from 7d89863 to 6f80da5 Compare December 10, 2021 00:27

ahg-g reviewed Dec 10, 2021

View reviewed changes

ahg-g mentioned this pull request Dec 10, 2021

refactor recordSchedulingFailure function name to more appropriate #106578

Merged

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 12, 2021

Huang-Wei force-pushed the fix-nnn-not-cleared branch from 692943e to 004df0e Compare December 12, 2021 06:16

alculquicondor reviewed Dec 15, 2021

View reviewed changes

Huang-Wei force-pushed the fix-nnn-not-cleared branch from 004df0e to 7677515 Compare December 15, 2021 19:36

alculquicondor reviewed Dec 16, 2021

View reviewed changes

clear pod's .status.nominatedNodeName when necessary

2433b08

Huang-Wei force-pushed the fix-nnn-not-cleared branch from d59ebfb to 2433b08 Compare December 16, 2021 18:55

k8s-ci-robot assigned alculquicondor Dec 16, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 16, 2021

k8s-ci-robot assigned ahg-g Dec 16, 2021

k8s-ci-robot merged commit 712745c into kubernetes:master Dec 17, 2021

k8s-ci-robot added this to the v1.24 milestone Dec 17, 2021

Huang-Wei deleted the fix-nnn-not-cleared branch December 17, 2021 04:55

Huang-Wei mentioned this pull request Dec 17, 2021

Automated cherry pick of #106816: clear pod's .status.nominatedNodeName when necessary #107109

Merged

k8s-ci-robot added a commit that referenced this pull request Jan 10, 2022

Merge pull request #107109 from Huang-Wei/automated-cherry-pick-of-#1…

94f7e18

…06816-upstream-release-1.23 Automated cherry pick of #106816: clear pod's .status.nominatedNodeName when necessary

Huang-Wei mentioned this pull request Apr 1, 2022

Nominated node in cache didn't match Pod status #109236

Closed

Timer mentioned this pull request Feb 20, 2023

STS pod stuck pending until deleted aws/karpenter-provider-aws#3420

Closed

	func (npm nominator) add(pi framework.PodInfo, nodeName string) {
	// always delete the pod if it already exist, to ensure we never store more than
	// one instance of the pod.
	npm.delete(pi.Pod)

	nnn := nodeName
	if len(nnn) == 0 {
	nnn = NominatedNodeName(pi.Pod)
	if len(nnn) == 0 {
	return
	}
	}

	// - if it's non-nil, the internal nominating bits will be updated with the given `nodeName`.
	// - if it's non-nil, the internal cache will be updated with the given `nodeName`.

clear pod's .status.nominatedNodeName when necessary #106816

clear pod's .status.nominatedNodeName when necessary #106816

Conversation

Huang-Wei commented Dec 4, 2021 • edited Loading

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

k8s-ci-robot commented Dec 4, 2021

Huang-Wei commented Dec 4, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Huang-Wei Dec 10, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Huang-Wei commented Dec 12, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alculquicondor commented Dec 16, 2021

ahg-g commented Dec 16, 2021

Huang-Wei commented Dec 16, 2021

alculquicondor commented Dec 16, 2021

Huang-Wei commented Dec 16, 2021

k8s-triage-robot commented Dec 17, 2021

cpanato commented Jan 7, 2022

cpanato commented Jan 7, 2022

mateuszlitwin commented Jan 7, 2022 • edited Loading

ahg-g commented Jan 7, 2022

alculquicondor commented Jan 7, 2022

ahg-g commented Jan 7, 2022

cpanato commented Jan 8, 2022

Huang-Wei commented Jan 8, 2022

Huang-Wei commented Dec 4, 2021 •

edited

Loading

Huang-Wei Dec 10, 2021 •

edited

Loading

mateuszlitwin commented Jan 7, 2022 •

edited

Loading