Multicluster dataplane change for Service access #3603

luolanzone · 2022-04-08T06:31:31Z

This PR is on top of #3463, it includes two commits, one is for data path change, another one is for changing Multi-cluster Service's Endpoint from Pod IP to Service ClusterIP.

Commit 1:
Add Multi-cluster feature in Agent

Add a new feature gate Multicluster and configs in antrea-agent.conf, and a few extra items in antrea-agent
cluster role including access to Gateway and ClusterInfoImport.
Rename the ServiceMarkTable to SNATMarkTable.

Add a controller for Gateway Nodes to watch Gateway and ClusterInfoImport's
events. It will set up a few openflow rules to forward cross-cluster traffic to remote Gateway Nodes.

Add a classification rule for cross-cluster traffic with global multicluster virtual
MAC aa:bb:cc:dd:ee:f0. A sample is like below:

 table=Classifier, priority=210,in_port="antrea-tun0",dl_dst=aa:bb:cc:dd:ee:f0 actions=load:0x1->NXM_NX_REG0[0..3],load:0x1->NXM_NX_REG0[9],resubmit(,SNATConntrackZone)

Add a rule in L3Forwarding table for cross-cluster request packets that modifies
the destination MAC to global multicluster virtual MAC. A sample is like below (the destination CIDR
is remote Service ClusterIP CIDR, the tunnel IP in NXM_NX_TUN_IPV4_DST is remote Gateway IP):

table=L3Forwarding, priority=200,ip,nw_dst=10.96.0.0/12 actions=mod_dl_src:ee:73:a5:81:09:c6,mod_dl_dst:aa:bb:cc:dd:ee:f0,load:0xab01b39->NXM_NX_TUN_IPV4_DST[],load:0x1->NXM_NX_REG0[4..7],resubmit(,L3DecTTL)

Add a rule in L3Forwarding table for cross-cluster reply packets. A sample is like below
(the destination IP is remote Gateway IP):

table=L3Forwarding, priority=200,ct_state=+rpl+trk,ip,nw_dst=10.176.27.57 actions=mod_dl_src:ee:73:a5:81:09:c6,mod_dl_dst:aa:bb:cc:dd:ee:f0,load:0xab01b39->NXM_NX_TUN_IPV4_DST[],load:0x1->NXM_NX_REG0[4..7],resubmit(,L3DecTTL)

Add a rule to SNATMark table to match the packets of multi-cluster Service connection and perform DNAT in DNAT zone.

table=SNATMark, priority=210,ip,nw_dst=10.96.0.0/12 actions=ct(commit,table=SNAT,zone=65520,exec(load:0x1->NXM_NX_CT_MARK[5]))

Add a rule to SNAT table to perform SNAT for any remote cluster traffic.

table=SNAT, priority=200,ct_state=+new+trk,ip,nw_dst=10.96.0.0/12 actions=ct(commit,table=L2ForwardingCalc,zone=65521,nat(src=10.176.27.224),exec(load:0x1->NXM_NX_CT_MARK[4]))

Add a rule to UnSNAT table to perform de-SNAT if destination IP is local GatewayIP.

table=UnSNAT, priority=200,ip,nw_dst=10.176.27.224 actions=ct(table=ConntrackZone,zone=65521,nat)

Add a rule in L2ForwardingCalc table to load the global virtual multi-cluster MAC's output to antrea-tun0

table=L2ForwardingCalc, priority=200,dl_dst=aa:bb:cc:dd:ee:f0 actions=load:0x1->NXM_NX_REG1[],load:0x1->NXM_NX_REG0[8],resubmit(,22)

Add a rule in Output table to match the multi-cluster traffic to forward the traffic from/to regular Node
through the same port.

table=Output, priority=210,reg0=0x200/0x200,reg1=0x1,in_port=1 actions=IN_PORT

Add a controller for regular Nodes to watch Gateway and ClusterInfoImport's events.
It will set up a few openflow rules to forward cross-cluster traffic to local Gateway Node.
- Add a rule in L3Forwarding table for cross-cluster request packets, and
  modify the destination MAC to global multicluster virtual MAC. A sample is like below
  (the destination CIDR is remote Service ClusterIP CIDR, the tunnel IP in NXM_NX_TUN_IPV4_DST is local Gateway's Internal IP.):
```
table=L3Forwarding, priority=200,ip,nw_dst=10.96.0.0/12 actions=mod_dl_src:f2:08:93:0c:82:bd,mod_dl_dst:aa:bb:cc:dd:ee:ff,load:0xab0193b->NXM_NX_TUN_IPV4_DST[],load:0x1->NXM_NX_REG0[4..7],resubmit(,L3DecTTL)
```
- Add a rule in L3Forwarding table for cross-cluster reply packets. A sample is like below
  (the destination IP is remote Gateway IP, the tunnel IP in NXM_NX_TUN_IPV4_DST is local Gateway's Internal IP):
```
table=L3Forwarding, priority=200,ct_state=+rpl+trk,ip,nw_dst=10.176.27.57 actions=mod_dl_src:f2:08:93:0c:82:bd,mod_dl_dst:aa:bb:cc:dd:ee:f0,load:0xab0193b->NXM_NX_TUN_IPV4_DST[],load:0x1->NXM_NX_REG0[4..7],resubmit(,L3DecTTL)
```
- Add a rule in L2ForwardingCalc table to load the global virtual multi-cluster MAC's output to antrea-tun0
```
table=L2ForwardingCalc, priority=200,dl_dst=aa:bb:cc:dd:ee:f0 actions=load:0x1->NXM_NX_REG1[],load:0x1->NXM_NX_REG0[8],resubmit(,22)
```
Add unit test cases
Refine e2e test for data plane change

Signed-off-by: Lan Luo luola@vmware.com
Co-authored-by: Hongliang Liu lhongliang@vmware.com

Commit 2:
Use Service ClusterIPs as MC Service's Endpoints

Use Service ClusterIPs instead of Pod IPs as MC Service's Endpoints.
The ServiceExport controller will only watch ServiceExport and
Service events, and wrap Service's ClusterIPs into a new Endpoint kind of
ResourceExport.
Includes local Serivce ClusterIP as multi-cluster Service's Endpoints as well.

Signed-off-by: Lan Luo luola@vmware.com

codecov-commenter · 2022-04-08T06:38:03Z

Codecov Report

Merging #3603 (f77ebbd) into main (90419f2) will decrease coverage by 7.16%.
The diff coverage is 51.59%.

@@            Coverage Diff             @@
##             main    #3603      +/-   ##
==========================================
- Coverage   63.96%   56.79%   -7.17%     
==========================================
  Files         288      406     +118     
  Lines       41252    57574   +16322     
==========================================
+ Hits        26386    32699    +6313     
- Misses      12733    22222    +9489     
- Partials     2133     2653     +520

Flag	Coverage Δ
integration-tests	`37.74% <ø> (?)`
kind-e2e-tests	`50.86% <3.31%> (+0.05%)`	⬆️
unit-tests	`44.37% <49.04%> (+0.05%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/agent/openflow/cookie/allocator.go	`76.74% <0.00%> (-3.75%)`	⬇️
pkg/apiserver/handlers/featuregates/handler.go	`75.40% <ø> (ø)`
pkg/features/antrea_features.go	`11.11% <ø> (ø)`
pkg/ovs/openflow/ofctrl_action.go	`69.45% <ø> (ø)`
pkg/ovs/openflow/ofctrl_group.go	`50.76% <ø> (ø)`
pkg/util/k8s/client.go	`40.00% <27.27%> (-1.08%)`	⬇️
pkg/agent/openflow/client.go	`66.14% <40.42%> (-1.73%)`	⬇️
pkg/agent/openflow/multicluster.go	`42.99% <42.99%> (ø)`
pkg/agent/multicluster/mc_route_controller.go	`52.27% <52.27%> (ø)`
...lticluster/commonarea/resourceimport_controller.go	`68.90% <66.66%> (-0.46%)`	⬇️
... and 133 more

luolanzone · 2022-04-14T08:45:08Z

/test-multicluster-dataplane-e2e

luolanzone · 2022-04-14T10:38:43Z

/test-multicluster-dataplane-e2e

luolanzone · 2022-04-14T12:47:59Z

/test-multicluster-dataplane-e2e

luolanzone · 2022-04-14T13:35:30Z

/test-multicluster-dataplane-e2e

luolanzone · 2022-04-14T14:15:09Z

/test-multicluster-dataplane-e2e

luolanzone · 2022-04-15T03:19:47Z

/test-multicluster-dataplane-e2e

multicluster/apis/multicluster/v1alpha1/tunnelendpoint_types.go

luolanzone · 2022-04-26T09:05:08Z

/test-multicluster-dataplane-e2e

luolanzone · 2022-04-26T09:35:50Z

/test-multicluster-dataplane-e2e

luolanzone · 2022-04-27T06:00:36Z

/test-multicluster-dataplane-e2e

luolanzone · 2022-04-27T07:26:43Z

/test-multicluster-dataplane-e2e

luolanzone · 2022-04-27T08:07:32Z

/test-multicluster-dataplane-e2e

luolanzone · 2022-04-27T09:41:08Z

/test-multicluster-dataplane-e2e

luolanzone · 2022-04-27T10:06:24Z

/test-multicluster-dataplane-e2e

luolanzone · 2022-04-28T05:38:35Z

/test-multicluster-dataplane-e2e

tnqn

LGTM except one question

pkg/agent/openflow/multicluster.go

luolanzone · 2022-06-08T03:41:17Z

/test-multicluster-dataplane-e2e

jianjuns · 2022-06-08T16:48:31Z

/test-all

tnqn

LGTM

jianjuns · 2022-06-08T23:20:39Z

/test-integration
/test-multicluster-e2e
/test-multicluster-dataplane-e2e
/test-ipv6-only-e2e

jianjuns · 2022-06-09T00:45:05Z

/test-integration

luolanzone · 2022-06-09T01:37:53Z

@jianjuns smee service is broken, the jenkins doesn't respond to the comment now, I will check with @XinShuYang.

luolanzone · 2022-06-09T01:44:08Z

Integration test failed due to the table rename, I will fix this.

luolanzone · 2022-06-09T02:32:35Z

/test-integration
/test-multicluster-e2e
/test-multicluster-dataplane-e2e
/test-ipv6-only-e2e

luolanzone · 2022-06-09T02:35:30Z

/test-conformance
/test-e2e
/test-networkpolicy

* Add a new feature gate `Multicluster` and configs in antrea-agent.conf, and a few extra items in antrea-agent cluster role including access to `Gateway` and `ClusterInfoImport`. * Rename the `ServiceMarkTable` to `SNATMarkTable`. * Add a controller for Gateway Nodes to watch Gateway and ClusterInfoImport's events. It will set up a few openflow rules to forward cross-cluster traffic to remote Gateway Nodes. * Add a classification rule for cross-cluster traffic with global multicluster virtual MAC `aa:bb:cc:dd:ee:f0`. A sample is like below: ``` table=Classifier, priority=210,in_port="antrea-tun0",dl_dst=aa:bb:cc:dd:ee:f0 actions=load:0x1->NXM_NX_REG0[0..3],load:0x1->NXM_NX_REG0[9],resubmit(,SNATConntrackZone) ``` * Add a rule in `L3Forwarding` table for cross-cluster request packets that modifies the destination MAC to global multicluster virtual MAC. A sample is like below (the destination CIDR is remote Service ClusterIP CIDR, the tunnel IP in NXM_NX_TUN_IPV4_DST is remote Gateway IP): ``` table=L3Forwarding, priority=200,ip,nw_dst=10.96.0.0/12 actions=mod_dl_src:ee:73:a5:81:09:c6,mod_dl_dst:aa:bb:cc:dd:ee:f0,load:0xab01b39->NXM_NX_TUN_IPV4_DST[],load:0x1->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) ``` * Add a rule in `L3Forwarding` table for cross-cluster reply packets. A sample is like below (the destination IP is remote Gateway IP): ``` table=L3Forwarding, priority=200,ct_state=+rpl+trk,ip,nw_dst=10.176.27.57 actions=mod_dl_src:ee:73:a5:81:09:c6,mod_dl_dst:aa:bb:cc:dd:ee:f0,load:0xab01b39->NXM_NX_TUN_IPV4_DST[],load:0x1->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) ``` * Add a rule to `SNATMark` table to match the packets of multi-cluster Service connection and perform DNAT in DNAT zone. ``` table=SNATMark, priority=210,ip,nw_dst=10.96.0.0/12 actions=ct(commit,table=SNAT,zone=65520,exec(load:0x1->NXM_NX_CT_MARK[5])) ``` * Add a rule to `SNAT` table to perform SNAT for any remote cluster traffic. ``` table=SNAT, priority=200,ct_state=+new+trk,ip,nw_dst=10.96.0.0/12 actions=ct(commit,table=L2ForwardingCalc,zone=65521,nat(src=10.176.27.224)) ``` * Add a rule to `UnSNAT` table to perform de-SNAT if destination IP is local GatewayIP. ``` table=UnSNAT, priority=200,ip,nw_dst=10.176.27.224 actions=ct(table=ConntrackZone,zone=65521,nat) ``` * Add a rule in `L2ForwardingCalc` table to load the global virtual multi-cluster MAC's output to `antrea-tun0` ``` table=L2ForwardingCalc, priority=200,dl_dst=aa:bb:cc:dd:ee:f0 actions=load:0x1->NXM_NX_REG1[],load:0x1->NXM_NX_REG0[8],resubmit(,22) ``` * Add a rule in `Output` table to match the multi-cluster traffic to forward the traffic from/to regular Node through the same port. ``` table=Output, priority=210,reg1=0x1,in_port=1 actions=IN_PORT ``` * Add a controller for regular Nodes to watch Gateway and ClusterInfoImport's events. It will set up a few openflow rules to forward cross-cluster traffic to local Gateway Node. * Add a rule in L3Forwarding table for cross-cluster request packets, and modify the destination MAC to global multicluster virtual MAC. A sample is like below (the destination CIDR is remote Service ClusterIP CIDR, the tunnel IP in NXM_NX_TUN_IPV4_DST is local Gateway's Internal IP.): ``` table=L3Forwarding, priority=200,ip,nw_dst=10.96.0.0/12 actions=mod_dl_src:f2:08:93:0c:82:bd,mod_dl_dst:aa:bb:cc:dd:ee:ff,load:0xab0193b->NXM_NX_TUN_IPV4_DST[],load:0x1->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) ``` * Add a rule in L3Forwarding table for cross-cluster reply packets. A sample is like below (the destination IP is remote Gateway IP, the tunnel IP in NXM_NX_TUN_IPV4_DST is local Gateway's Internal IP): ``` table=L3Forwarding, priority=200,ct_state=+rpl+trk,ip,nw_dst=10.176.27.57 actions=mod_dl_src:f2:08:93:0c:82:bd,mod_dl_dst:aa:bb:cc:dd:ee:f0,load:0xab0193b->NXM_NX_TUN_IPV4_DST[],load:0x1->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) ``` * Add a rule in L2ForwardingCalc table to load the global virtual multi-cluster MAC's output to `antrea-tun0` ``` table=L2ForwardingCalc, priority=200,dl_dst=aa:bb:cc:dd:ee:f0 actions=load:0x1->NXM_NX_REG1[],load:0x1->NXM_NX_REG0[8],resubmit(,22) ``` * Add unit test cases * Refine e2e test for data plane change Signed-off-by: Lan Luo <luola@vmware.com> Co-authored-by: Hongliang Liu <lhongliang@vmware.com>

1. Use Service ClusterIPs instead of Pod IPs as MC Service's Endpoints. The ServiceExport controller will only watch ServiceExport and Service events, and wrap Service's ClusterIPs into a new Endpoint kind of ResourceExport. 2. Includes local Serivce ClusterIP as multi-cluster Service's Endpoints as well. Signed-off-by: Lan Luo <luola@vmware.com>

luolanzone · 2022-06-09T03:04:10Z

/test-integration

jianjuns · 2022-06-09T04:08:16Z

Integration test failed due to the table rename, I will fix this.

Sure. Please take care of such issues earlier next time.

luolanzone · 2022-06-09T04:13:44Z

Integration test failed due to the table rename, I will fix this.

Sure. Please take care of such issues earlier next time.

Yeah, I will, I didn't notice integration test will be impacted, will check the result in time next time.
btw, do we need to rerun all tests again? or just those failed cases?

jianjuns · 2022-06-09T04:44:50Z

Ideally run all tests of test-all.

luolanzone · 2022-06-09T04:57:58Z

/test-all

luolanzone · 2022-06-09T06:04:16Z

/test-all

luolanzone · 2022-06-09T06:07:06Z

/test-multicluster-dataplane-e2e

luolanzone · 2022-06-09T07:46:51Z

Hi @tnqn all required tests are passed now, I skipped test-multicluster-e2e which is an old job. The new one jenkins-multicluster-dataplane-e2e is passed. Could you help to move forward? thanks.

tnqn · 2022-06-09T08:23:59Z

Since this change updates common flows, could you make sure ipv6-e2e and ipv6-only-e2e pass (except FQDNPolicyInCluster in ipv6-only-e2e)? We had issues with them several times and just fixed all of them except #3873, don't want to break them again when it's close to release.

luolanzone · 2022-06-09T08:50:02Z

/test-ipv6-only-e2e
/test-ipv6-e2e

luolanzone added the area/multi-cluster Issues or PRs related to multi cluster. label Apr 8, 2022

luolanzone marked this pull request as draft April 8, 2022 06:34

luolanzone force-pushed the mc-dataplane branch from 43ec8a1 to 14edcf7 Compare April 14, 2022 07:22

Dyanngg reviewed Apr 15, 2022

View reviewed changes

multicluster/apis/multicluster/v1alpha1/tunnelendpoint_types.go Outdated Show resolved Hide resolved

luolanzone force-pushed the mc-dataplane branch 2 times, most recently from f7ac8fd to 1e29016 Compare April 26, 2022 09:00

luolanzone force-pushed the mc-dataplane branch from 1e29016 to a8b5d8b Compare April 26, 2022 09:32

luolanzone changed the title ~~[WIP]Multicluster dataplane change for Service and Pod access~~ [WIP]Multicluster dataplane change for Service access Apr 27, 2022

luolanzone force-pushed the mc-dataplane branch 2 times, most recently from 079c4e7 to 7542ec3 Compare April 27, 2022 08:07

luolanzone force-pushed the mc-dataplane branch from 7542ec3 to ee0dfbb Compare April 27, 2022 09:40

luolanzone force-pushed the mc-dataplane branch 2 times, most recently from 552846a to be8750e Compare April 27, 2022 10:06

luolanzone added this to the Antrea v1.7 release milestone Apr 28, 2022

luolanzone force-pushed the mc-dataplane branch from be8750e to d918470 Compare April 28, 2022 05:38

tnqn reviewed Jun 7, 2022

View reviewed changes

pkg/agent/openflow/multicluster.go Show resolved Hide resolved

luolanzone force-pushed the mc-dataplane branch from 35ef532 to 2a7bd4a Compare June 8, 2022 03:39

luolanzone force-pushed the mc-dataplane branch from 2a7bd4a to e9877f3 Compare June 8, 2022 05:15

tnqn previously approved these changes Jun 8, 2022

View reviewed changes

luolanzone dismissed tnqn’s stale review via a53a3aa June 9, 2022 01:53

luolanzone force-pushed the mc-dataplane branch from e9877f3 to a53a3aa Compare June 9, 2022 01:53

luolanzone and others added 2 commits June 9, 2022 10:53

luolanzone force-pushed the mc-dataplane branch from a53a3aa to f77ebbd Compare June 9, 2022 03:03

tnqn approved these changes Jun 9, 2022

View reviewed changes

tnqn merged commit 968b330 into antrea-io:main Jun 9, 2022

luolanzone mentioned this pull request Jun 15, 2022

Release 1.7.0 #3895

Merged

Multicluster dataplane change for Service access #3603

Multicluster dataplane change for Service access #3603

Conversation

luolanzone commented Apr 8, 2022 • edited Loading

codecov-commenter commented Apr 8, 2022 • edited Loading

Codecov Report

luolanzone commented Apr 14, 2022

luolanzone commented Apr 14, 2022

luolanzone commented Apr 14, 2022

luolanzone commented Apr 14, 2022

luolanzone commented Apr 14, 2022

luolanzone commented Apr 15, 2022

luolanzone commented Apr 26, 2022

luolanzone commented Apr 26, 2022

luolanzone commented Apr 27, 2022

luolanzone commented Apr 27, 2022

luolanzone commented Apr 27, 2022

luolanzone commented Apr 27, 2022

luolanzone commented Apr 27, 2022

luolanzone commented Apr 28, 2022

tnqn left a comment

Choose a reason for hiding this comment

luolanzone commented Jun 8, 2022

jianjuns commented Jun 8, 2022

tnqn left a comment

Choose a reason for hiding this comment

jianjuns commented Jun 8, 2022

jianjuns commented Jun 9, 2022

luolanzone commented Jun 9, 2022

luolanzone commented Jun 9, 2022

luolanzone commented Jun 9, 2022

luolanzone commented Jun 9, 2022

luolanzone commented Jun 9, 2022

jianjuns commented Jun 9, 2022

luolanzone commented Jun 9, 2022

jianjuns commented Jun 9, 2022

luolanzone commented Jun 9, 2022

luolanzone commented Jun 9, 2022

luolanzone commented Jun 9, 2022

luolanzone commented Jun 9, 2022

tnqn commented Jun 9, 2022

luolanzone commented Jun 9, 2022

luolanzone commented Apr 8, 2022 •

edited

Loading

codecov-commenter commented Apr 8, 2022 •

edited

Loading