Delay removal of flow-restore-wait #6342
Conversation
3d6baba to
7ea44b5
Compare
| @@ -80,6 +83,12 @@ type Controller struct { | |||
| // or not when IPsec is enabled with "cert" mode. The NodeRouteController must wait for the certificate | |||
| // to be configured before installing routes/flows to peer Nodes to prevent unencrypted traffic across Nodes. | |||
| ipsecCertificateManager ipseccertificate.Manager | |||
| // flowRestoreCompleteWait is to be decremented after installing flows for initial Nodes. | |||
| flowRestoreCompleteWait *utilwait.Group | |||
| // hasProcessedInitialList keeps track of whether the initial informer list as been | |||
There was a problem hiding this comment.
| // hasProcessedInitialList keeps track of whether the initial informer list as been | |
| // hasProcessedInitialList keeps track of whether the initial informer list has been |
7ea44b5 to
5b169d1
Compare
|
/test-flexible-ipam-e2e |
| @@ -844,6 +847,21 @@ func run(o *Options) error { | |||
| go mcStrechedNetworkPolicyController.Run(stopCh) | |||
| } | |||
|
|
|||
| klog.InfoS("Waiting for flow restoration to complete") | |||
There was a problem hiding this comment.
@tnqn I wasn't sure what would be the best place for this code. I put it here because I think by that point all the controllers that need to install flows have been instantiated, initialized and "started" (even though at the moment, flowRestoreCompleteWait only waits for the NetworkPolicyController + NodeRouteController + CNIServer).
There was a problem hiding this comment.
It makes sense to me. I tried to delay FlowRestoreComplete but was unsure whether moving ConnectUplinkToOVSBridge has any impact. cc @gran-vmv
There was a problem hiding this comment.
We can run flexible-ipam-e2e multiple times to check if this change can work.
|
/test-flexible-ipam-e2e |
| @@ -844,6 +847,21 @@ func run(o *Options) error { | |||
| go mcStrechedNetworkPolicyController.Run(stopCh) | |||
| } | |||
|
|
|||
| klog.InfoS("Waiting for flow restoration to complete") | |||
There was a problem hiding this comment.
It makes sense to me. I tried to delay FlowRestoreComplete but was unsure whether moving ConnectUplinkToOVSBridge has any impact. cc @gran-vmv
|
/test-flexible-ipam-e2e |
1 similar comment
|
/test-flexible-ipam-e2e |
|
I go the following errors for However, I have also observed the same test failures for another PR unrelated to this one (#6321) |
|
/test-all |
|
|
So the issue seems to be that until flow-restore-wait is removed, the Agent may not be able to reach the So there is a chicken and egg problem here. flowRestoreCompleteWait.WaitWithTimeout(10*time.Second)
if err := agentInitializer.FlowRestoreComplete(); err != nil {
return err
}Because it blocks, it would still delay a few key operations, such as starting the apiserver, which answers liveness probes. BTW, I thought that if the watch couldn't fail in a reasonable amount of time, we would fallback to data stored in files. Do you know the timeout because I didn't see it happen during that 30s time window: |
It reminds me that I encountered this problem as well. Since we have a separate antreaClientProvider, do you think we could break the chicken-egg problem by not relying on the proxy, resolving the service to endpoint IP directly via something like https://github.com/kubernetes/kubernetes/blob/11f37d757fc0b710245446c80a8c9578ce2c02f1/staging/src/k8s.io/kube-aggregator/pkg/apiserver/resolvers.go#L33? |
Thanks for the idea @tnqn, I will try it. |
5b169d1 to
e98ae09
Compare
Until a set of "essential" flows has been installed. At the moment, we include NetworkPolicy flows (using podNetworkWait as the signal), Pod forwarding flows (reconciled by the CNIServer), and Node routing flows (installed by the NodeRouteController). This set can be extended in the future if desired. We leverage the wrapper around sync.WaitGroup which was introduced previously in antrea-io#5777. It simplifies unit testing, and we can achieve some symmetry with podNetworkWait. We can also start leveraging this new wait group (flowRestoreCompleteWait) as the signal to delete flows from previous rounds. However, at the moment this is incomplete, as we don't wait for all controllers to signal that they have installed initial flows. Because the NodeRouteController does not have an initial "reconcile" operation (like the CNIServer) to install flows for the initial Node list, we instead rely on a different mechanims provided by upstream K8s for controllers. When registering event handlers, we can request for the ADD handler to include a boolean flag indicating whether the object is part of the initial list retrieved by the informer. Using this mechanism, we can reliably signal through flowRestoreCompleteWait when this initial list of Nodes has been synced at least once. This change is possible because of antrea-io#6361, which removed the dependency on the proxy (kube-proxy or AntreaProxy) to access the Antrea Controller. Prior to antrea-io#6361, there would have been a circular dependency in the case where kube-proxy was removed: flow-restore-wait will not be removed until the Pod network is "ready", which will not happen until the NetworkPolicy controller has started its watchers, and that depends on antrea Service reachability which depends on flow-restore-wait being removed. Fixes antrea-io#6338 Signed-off-by: Antonin Bas <antonin.bas@broadcom.com>
e98ae09 to
665973c
Compare
Signed-off-by: Antonin Bas <antonin.bas@broadcom.com>
665973c to
313eb8f
Compare
|
/test-all |
|
/test-flexible-ipam-e2e |
1 similar comment
|
/test-flexible-ipam-e2e |
Until a set of "essential" flows has been installed. At the moment, we include NetworkPolicy flows (using podNetworkWait as the signal), Pod forwarding flows (reconciled by the CNIServer), and Node routing flows (installed by the NodeRouteController). This set can be extended in the future if desired.
We leverage the wrapper around sync.WaitGroup which was introduced previously in #5777. It simplifies unit testing, and we can achieve some symmetry with podNetworkWait.
We can also start leveraging this new wait group
(flowRestoreCompleteWait) as the signal to delete flows from previous rounds. However, at the moment this is incomplete, as we don't wait for all controllers to signal that they have installed initial flows.
Because the NodeRouteController does not have an initial "reconcile" operation (like the CNIServer) to install flows for the initial Node list, we instead rely on a different mechanims provided by upstream K8s for controllers. When registering event handlers, we can request for the ADD handler to include a boolean flag indicating whether the object is part of the initial list retrieved by the informer. Using this mechanism, we can reliably signal through flowRestoreCompleteWait when this initial list of Nodes has been synced at least once.
This change is possible because of #6361, which removed the dependency on the proxy (kube-proxy or AntreaProxy) to access the Antrea Controller. Prior to #6361, there would have been a circular dependency in the case where kube-proxy was removed: flow-restore-wait will not be removed until the Pod network is "ready", which will not happen until the NetworkPolicy controller has started its watchers, and that depends on antrea Service reachability which depends on flow-restore-wait being removed.
Fixes #6338