OVS flows in L3Forwarding table conflict #3806

gran-vmv · 2022-05-18T09:29:20Z

Describe the bug

To Reproduce
Run ipv6-ds-e2e test. This issue should affect ipv4/ipv6/ds, but actually only ds will fail.

Expected
All cases passed.

Actual behavior
Ingress drop policy may not take effect on some Node for intra-Node traffic.

=== RUN   TestAntreaPolicy/TestGroupDefaultDENY/Case=ACNPAllowXBtoA
I0518 08:19:05.627217    4918 antreapolicy_test.go:3005] running test case ACNP Allow X/B to A
I0518 08:19:05.627248    4918 antreapolicy_test.go:3007] running step Port 80 of test case ACNP Allow X/B to A
I0518 08:19:05.627264    4918 k8s_util.go:759] Creating/updating ClusterNetworkPolicy acnp-allow-xb-to-a
I0518 08:19:05.742255    4918 util.go:44] Confirming ready status costs 103.399129ms
    antreapolicy_test.go:3019: failure -- 19 wrong results
reachability: correct:62, incorrect:19, result=false

expected:

-	x/a	y/a	z/a	x/b	y/b	z/b	x/c	y/c	z/c
x/a	Con	Drp	Drp	Drp	Drp	Drp	Drp	Drp	Drp
y/a	Drp	Con	Drp	Drp	Drp	Drp	Drp	Drp	Drp
z/a	Drp	Drp	Con	Drp	Drp	Drp	Drp	Drp	Drp
x/b	Con	Con	Con	Con	Drp	Drp	Drp	Drp	Drp
y/b	Drp	Drp	Drp	Drp	Con	Drp	Drp	Drp	Drp
z/b	Drp	Drp	Drp	Drp	Drp	Con	Drp	Drp	Drp
x/c	Drp	Drp	Drp	Drp	Drp	Drp	Con	Drp	Drp
y/c	Drp	Drp	Drp	Drp	Drp	Drp	Drp	Con	Drp
z/c	Drp	Drp	Drp	Drp	Drp	Drp	Drp	Drp	Con


observed:

-	x/a	y/a	z/a	x/b	y/b	z/b	x/c	y/c	z/c
x/a	Con	Drp	Drp	Drp	Drp	Drp	Drp	Drp	Drp
y/a	Drp	Con	Drp	Drp	Drp	Drp	Drp	Drp	Drp
z/a	Drp	Drp	Con	Con	Con	Drp	Con	Drp	Con
x/b	Con	Con	Con	Con	Con	Drp	Con	Drp	Con
y/b	Drp	Drp	Con	Con	Con	Drp	Con	Drp	Con
z/b	Drp	Drp	Drp	Drp	Drp	Con	Drp	Drp	Drp
x/c	Drp	Drp	Con	Con	Con	Drp	Con	Drp	Con
y/c	Drp	Drp	Drp	Drp	Drp	Drp	Drp	Con	Drp
z/c	Drp	Drp	Con	Con	Con	Drp	Con	Drp	Con


comparison:

-	x/a	y/a	z/a	x/b	y/b	z/b	x/c	y/c	z/c
x/a	.	.	.	.	.	.	.	.	.
y/a	.	.	.	.	.	.	.	.	.
z/a	.	.	.	X	X	.	X	.	X
x/b	.	.	.	.	X	.	X	.	X
y/b	.	.	X	X	.	.	X	.	X
z/b	.	.	.	.	.	.	.	.	.
x/c	.	.	X	X	X	.	.	.	X
y/c	.	.	.	.	.	.	.	.	.
z/c	.	.	X	X	X	.	X	.	.


I0518 08:19:17.071650    4918 k8s_util.go:788] Deleting AntreaClusterNetworkPolicies acnp-allow-xb-to-a
I0518 08:19:17.180175    4918 util.go:44] Confirming deleted status costs 103.325475ms

Versions:
Antrea>=1.6

Additional context
Caused by priority conflict. Below flows are used by Egress feature which enabled by default since Antrea 1.6, and they conflict with other priority=190 flows.

 cookie=0x23040000000000, duration=6216.068s, table=L3Forwarding, n_packets=4113, n_bytes=303108, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf actions=goto_table:EgressMark
 cookie=0x23040000000000, duration=6216.068s, table=L3Forwarding, n_packets=0, n_bytes=0, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=set_field:f6:06:04:55:43:e7->eth_dst,goto_table:EgressMark
 cookie=0x23040000000000, duration=6216.068s, table=L3Forwarding, n_packets=1853, n_bytes=172530, priority=190,ct_state=-rpl+trk,ipv6,reg0=0x3/0xf actions=goto_table:EgressMark
 cookie=0x23040000000000, duration=6216.068s, table=L3Forwarding, n_packets=0, n_bytes=0, priority=190,ct_state=-rpl+trk,ipv6,reg0=0x1/0xf actions=set_field:f6:06:04:55:43:e7->eth_dst,goto_table:EgressMark

 cookie=0x23010000000000, duration=6216.069s, table=L3Forwarding, n_packets=22546, n_bytes=1799487, priority=190,ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=192.168.250.0/24 actions=goto_table:L2ForwardingCalc
 cookie=0x23010000000000, duration=6216.069s, table=L3Forwarding, n_packets=24251, n_bytes=2359207, priority=190,ipv6,reg0=0/0x200,reg8=0/0xfff,ipv6_dst=fd02:0:0:fa::/64 actions=goto_table:L2ForwardingCalc
 cookie=0x23030000000000, duration=6216.069s, table=L3Forwarding, n_packets=0, n_bytes=0, priority=190,ct_mark=0x10/0x10,reg0=0x200/0x200,reg4=0/0x100000 actions=set_field:f6:06:04:55:43:e7->eth_dst,load:0x2->NXM_NX_REG0[4..7],goto_table:L3DecTTL
 cookie=0x23030000000000, duration=6216.068s, table=L3Forwarding, n_packets=0, n_bytes=0, priority=190,ct_mark=0x10/0x10,reg4=0x100000/0x100000 actions=goto_table:L3DecTTL
 cookie=0x23010000000000, duration=6216.070s, table=L3Forwarding, n_packets=0, n_bytes=0, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],goto_table:L2ForwardingCalc

The text was updated successfully, but these errors were encountered:

gran-vmv · 2022-05-18T09:34:42Z

Wrong result trace:

// hit wrong flow in L3Forwarding table
17. ct_state=-rpl+trk,ip,reg0=0x3/0xf, priority 190, cookie 0x23040000000000
    goto_table:18
18. priority 0, cookie 0x23040000000000
    load:0x2->NXM_NX_REG0[4..7]
    goto_table:22
22. dl_dst=66:44:35:39:c2:19, priority 200, cookie 0x23010000000000
    load:0x18->NXM_NX_REG1[]
    load:0x1->NXM_NX_REG0[8]
    goto_table:23
// reg0 is set by above tables and hit wrong flow in IngressSecurityClassifier table
23. reg0=0x20/0xf0, priority 200, cookie 0x23020000000000
    goto_table:27
27. priority 0, cookie 0x23000000000000
    goto_table:28

Correct result trace:

17. ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=192.168.249.0/24, priority 190, cookie 0x57010000000000
    goto_table:22
22. dl_dst=ba:0d:67:fd:fe:16, priority 200, cookie 0x57010000000000
    load:0x3b->NXM_NX_REG1[]
    load:0x1->NXM_NX_REG0[8]
    goto_table:23
23. priority 0, cookie 0x57000000000000
    goto_table:24
24. priority 0, cookie 0x57000000000000
    goto_table:25
25. priority 0, cookie 0x57000000000000
    goto_table:26
26. reg1=0x3b, priority 200, cookie 0x57020000000000
    drop

gran-vmv · 2022-05-18T09:36:17Z

@hongliangl @wenyingd

Fix antrea-io#3806 Currently, in L3ForwardingTable, there are several flows like the follows: ``` 1. table=L3Forwarding, priority=190,ip,reg0=0/0x200,nw_dst=10.10.1.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x200/0x200 actions=mod_dl_dst:e6:d5:b1:dc:f3:1d,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf actions=resubmit(,EgressMark) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:e6:d5:b1:dc:f3:1d,resubmit(,EgressMark) 5. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` - Flow 1 is used to forward the packets of non-Service connections between local Pods. - Flow 2 is used to forward the packets of Service connections destined for external network. - Flow 3 is used to forward the packets sourced from local Pods and destined for external network, and the flow is for Egress. - Flow 4 is used to forward the packets sourced from tunnel and destined for external network, and the flow is also for Egress. - Flow 5 is the default flow. `load:0x2->NXM_NX_REG0[4..7]` means `to Gateway`. For request packets sourced from a local Pod and destined for another local Pod, they are expected to be matched by flow 1, however, they can be matched by flow 3, and this leads the result of antrea-io#3806. In addtion, when Egress is enabled, if a local Pod accesses to a Service whose Endpoint is on external Network, the request packets are expected to be matched by flow 3, but they can be also matched by flow 2. To resolve above issue, the new flows are like the follows: ``` 1. table=L3Forwarding, priority=200,ip,reg0=0/0x200,nw_dst=10.10.1.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=0 actions=resubmit(,L3ForwardingDefault) 3. table=L3ForwardingDefault, priority=210,ct_state=-rpl+trk,ip,reg0=0x3/0xf actions=resubmit(,EgressMark) 4. table=L3ForwardingDefault, priority=210,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:e6:d5:b1:dc:f3:1d,resubmit(,EgressMark) 5. table=L3ForwardingDefault, priority=200,ct_mark=0x10/0x10,reg0=0x200/0x200 actions=mod_dl_dst:e6:d5:b1:dc:f3:1d,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 6. table=L3ForwardingDefault, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` Signed-off-by: Hongliang Liu <lhongliang@vmware.com>

Fix antrea-io#3806 Currently, in L3ForwardingTable, there are several flows like the follows: ``` 1. table=L3Forwarding, priority=190,ip,reg0=0/0x200,nw_dst=10.10.1.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x200/0x200 actions=mod_dl_dst:e6:d5:b1:dc:f3:1d,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf actions=resubmit(,EgressMark) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:e6:d5:b1:dc:f3:1d,resubmit(,EgressMark) 5. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` - Flow 1 is used to forward the packets of non-Service connections between local Pods. - Flow 2 is used to forward the packets of Service connections destined for external network. - Flow 3 is used to forward the packets sourced from local Pods and destined for external network, and the flow is for Egress. - Flow 4 is used to forward the packets sourced from tunnel and destined for external network, and the flow is also for Egress. - Flow 5 is the default flow. `load:0x2->NXM_NX_REG0[4..7]` means `to Gateway`. For request packets sourced from a local Pod and destined for another local Pod, they are expected to be matched by flow 1, however, they can be matched by flow 3, and this leads the result of antrea-io#3806. In addtion, when Egress is enabled, if a local Pod accesses to a Service whose Endpoint is on external Network, the request packets are expected to be matched by flow 3, but they can be also matched by flow 2. To resolve above issue, one possible way is to add more priorities and rearrange the flows, but we don't have enough priorities. As a result, a new table L3ForwardingDefault is added. The new flows are like the follows: ``` 1. table=L3Forwarding, priority=200,ip,reg0=0/0x200,nw_dst=10.10.1.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=0 actions=resubmit(,L3ForwardingDefault) 3. table=L3ForwardingDefault, priority=210,ct_state=-rpl+trk,ip,reg0=0x3/0xf actions=resubmit(,EgressMark) 4. table=L3ForwardingDefault, priority=210,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:e6:d5:b1:dc:f3:1d,resubmit(,EgressMark) 5. table=L3ForwardingDefault, priority=200,ct_mark=0x10/0x10,reg0=0x200/0x200 actions=mod_dl_dst:e6:d5:b1:dc:f3:1d,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 6. table=L3ForwardingDefault, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` Signed-off-by: Hongliang Liu <lhongliang@vmware.com>

…n in L3Forwarding Fix antrea-io#3806 Currently, in L3ForwardingTable, there are several flows like the follows: ``` 1. table=L3Forwarding, priority=190,ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=10.10.0.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x200/0x200,reg4=0/0x100000 actions=mod_dl_dst:d2:35:24:7f:3a:f8,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg4=0x100000/0x100000 actions=resubmit(,L3DecTTL) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf actions=resubmit(,EgressMark) 5. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:d2:35:24:7f:3a:f8,resubmit(,EgressMark) 6. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` - Flow 1 is used to forward the packets of non-Service connections between local Pods. - Flow 2 is used to forward the packets of Service connections sourced from local Pods and destined for external network Endpoint. - Flow 3 is used to forward the packets of Service connections sourced from local AntreaIPAM Pods and destined for external network Endpoint. - Flow 4 is used to forward the packets sourced from local Pods and destined for external network, and the flow is for Egress. - Flow 5 is used to forward the packets sourced from tunnel and destined for external network, and the flow is also for Egress. - Flow 6 is the default flow. `load:0x2->NXM_NX_REG0[4..7]` means `to Gateway`. For request packets sourced from a local Pod and destined for another local Pod, they are expected to be matched by flow 1, however, they can be matched by flow 4, and this leads the unexpected result in antrea-io#3806. In addtion, when Egress is enabled, if a local Pod accesses to a Service whose Endpoint is on external Network, the request packets are expected to be matched by flow 4, but they can be also matched by flow 2. To resolve above issues, the new flows are like the follows: ``` 1. table=L3Forwarding, priority=200,ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=10.10.0.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x202/0x20f actions=mod_dl_dst:d2:35:24:7f:3a:f8,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf actions=resubmit(,EgressMark) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:d2:35:24:7f:3a:f8,resubmit(,EgressMark) 5. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` Issue antrea-io#3806 is fixed by promoting the priority of the flow matching packets between local Pods to normal (200). In legacy flow 2, packets of Service connections with unknown destionation can be either sourced from local Pods or Antrea gateway. In the new flow 2, only the packets of Service connections sourced Antrea gateway are matched. Packets of Service connections sourced from local non-AntreaIPAM Pods or AntreaIPAM Pods and destined for external network are matched by the new flow 3 (Egress is enabled) or the new flow 5 (Egress is disabled). Other modifications: fix some stale comments. Signed-off-by: Hongliang Liu <lhongliang@vmware.com>

…n in L3Forwarding Fix antrea-io#3806 Currently, when Egress and AntreaIPAM are enabled, there are several flows in L3ForwardingTable like the follows: ``` 1. table=L3Forwarding, priority=190,ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=10.10.0.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x200/0x200,reg4=0/0x100000 actions=mod_dl_dst:d2:35:24:7f:3a:f8,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg4=0x100000/0x100000 actions=resubmit(,L3DecTTL) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf actions=resubmit(,EgressMark) 5. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:d2:35:24:7f:3a:f8,resubmit(,EgressMark) 6. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` - Flow 1 is used to forward the packets of non-Service connections between local Pods. - Flow 2 is used to forward the packets of Service connections sourced from local non-AntreaIPAM Pods and destined for external network Endpoint. - Flow 3 is used to forward the packets of Service connections sourced from local AntreaIPAM Pods and destined for external network Endpoint. - Flow 4 is used to forward the packets sourced from local Pods and destined for external network, and the flow is for Egress. - Flow 5 is used to forward the packets sourced from tunnel and destined for external network, and the flow is also for Egress. - Flow 6 is the default flow. `load:0x2->NXM_NX_REG0[4..7]` means `to Gateway`. For request packets sourced from a local Pod and destined for another local Pod, they are expected to be matched by flow 1, however, they can be matched by flow 4, and this leads the unexpected result in antrea-io#3806. In addition, when Egress is enabled, if a local Pod accesses to a Service whose Endpoint is on external Network, the request packets are expected to be matched by flow 4, but they can be also matched by flow 2. To resolve above issues, the new flows are like the follows: ``` 1. table=L3Forwarding, priority=200,ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=10.10.0.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x202/0x20f actions=mod_dl_dst:d2:35:24:7f:3a:f8,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf,reg4=0/0x100000 actions=resubmit(,EgressMark) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:d2:35:24:7f:3a:f8,resubmit(,EgressMark) 5. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` Issue antrea-io#3806 is fixed by promoting the priority of the flow matching packets between local Pods to normal (200). In legacy flow 2, packets of Service connections with unknown destination can be either sourced from local Pods or Antrea gateway. In the new flow 2, only the packets of Service connections sourced Antrea gateway are matched. Packets of Service or non-Service connections sourced from local non-AntreaIPAM Pods are matched by the new flow 3 (Egress is enabled) or the new flow 5 (Egress is disabled). Packets of Service or non-Service connections sourced from local AntreaIPAM Pods and destined for external network are matched by the new flow 5. Other modifications: fix some stale comments. Signed-off-by: Hongliang Liu <lhongliang@vmware.com>

…n in L3ForwardingTable Fix antrea-io#3806 Currently, when Egress and AntreaIPAM are enabled, there are several flows in L3ForwardingTable like the follows: ``` 1. table=L3Forwarding, priority=190,ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=10.10.0.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x200/0x200,reg4=0/0x100000 actions=mod_dl_dst:d2:35:24:7f:3a:f8,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg4=0x100000/0x100000 actions=resubmit(,L3DecTTL) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf actions=resubmit(,EgressMark) 5. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:d2:35:24:7f:3a:f8,resubmit(,EgressMark) 6. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` - Flow 1 is used to forward the packets of non-Service connections between local Pods. - Flow 2 is used to forward the packets of Service connections sourced from local non-AntreaIPAM Pods and destined for external network Endpoint. - Flow 3 is used to forward the packets of Service connections sourced from local AntreaIPAM Pods and destined for external network Endpoint. - Flow 4 is used to forward the packets sourced from local Pods and destined for external network, and the flow is for Egress. - Flow 5 is used to forward the packets sourced from tunnel and destined for external network, and the flow is also for Egress. - Flow 6 is the default flow. `load:0x2->NXM_NX_REG0[4..7]` means `to Gateway`. For request packets sourced from a local Pod and destined for another local Pod, they are expected to be matched by flow 1, however, they can be matched by flow 4, and this leads the unexpected result in antrea-io#3806. In addition, when Egress is enabled, if a local Pod accesses to a Service whose Endpoint is on external Network, the request packets are expected to be matched by flow 4, but they can be also matched by flow 2. To resolve above issues, the new flows are like the follows: ``` 1. table=L3Forwarding, priority=200,ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=10.10.0.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x202/0x20f actions=mod_dl_dst:d2:35:24:7f:3a:f8,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf,reg4=0/0x100000 actions=resubmit(,EgressMark) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:d2:35:24:7f:3a:f8,resubmit(,EgressMark) 5. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` Issue antrea-io#3806 is fixed by promoting the priority of the flow matching packets between local Pods to normal (200). In legacy flow 2, packets of Service connections with unknown destination can be either sourced from local Pods or Antrea gateway. In the new flow 2, only the packets of Service connections sourced Antrea gateway are matched. Packets of Service or non-Service connections sourced from local non-AntreaIPAM Pods are matched by the new flow 3 (Egress is enabled) or the new flow 5 (Egress is disabled). Packets of Service or non-Service connections sourced from local AntreaIPAM Pods and destined for external network are matched by the new flow 5. Other modifications: fix some stale comments. Signed-off-by: Hongliang Liu <lhongliang@vmware.com>

Fix antrea-io#3806 Currently, in L3ForwardingTable, there are several flows like the follows: ``` 1. table=L3Forwarding, priority=190,ip,reg0=0/0x200,nw_dst=10.10.1.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x200/0x200 actions=mod_dl_dst:e6:d5:b1:dc:f3:1d,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf actions=resubmit(,EgressMark) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:e6:d5:b1:dc:f3:1d,resubmit(,EgressMark) 5. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` - Flow 1 is used to forward the packets of non-Service connections between local Pods. - Flow 2 is used to forward the packets of Service connections destined for external network. - Flow 3 is used to forward the packets sourced from local Pods and destined for external network, and the flow is for Egress. - Flow 4 is used to forward the packets sourced from tunnel and destined for external network, and the flow is also for Egress. - Flow 5 is the default flow. `load:0x2->NXM_NX_REG0[4..7]` means `to Gateway`. For request packets sourced from a local Pod and destined for another local Pod, they are expected to be matched by flow 1, however, they can be matched by flow 3, and this leads the result of antrea-io#3806. In addtion, when Egress is enabled, if a local Pod accesses to a Service whose Endpoint is on external Network, the request packets are expected to be matched by flow 3, but they can be also matched by flow 2. To resolve above issue, one possible way is to add more priorities and rearrange the flows, but we don't have enough priorities. As a result, a new table L3ForwardingDefault is added. The new flows are like the follows: ``` 1. table=L3Forwarding, priority=200,ip,reg0=0/0x200,nw_dst=10.10.1.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=0 actions=resubmit(,L3ForwardingDefault) 3. table=L3ForwardingDefault, priority=210,ct_state=-rpl+trk,ip,reg0=0x3/0xf actions=resubmit(,EgressMark) 4. table=L3ForwardingDefault, priority=210,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:e6:d5:b1:dc:f3:1d,resubmit(,EgressMark) 5. table=L3ForwardingDefault, priority=200,ct_mark=0x10/0x10,reg0=0x200/0x200 actions=mod_dl_dst:e6:d5:b1:dc:f3:1d,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 6. table=L3ForwardingDefault, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` Signed-off-by: Hongliang Liu <lhongliang@vmware.com>

…n in L3ForwardingTable Fix antrea-io#3806 Currently, when Egress and AntreaIPAM are enabled, there are several flows in L3ForwardingTable like the follows: ``` 1. table=L3Forwarding, priority=190,ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=10.10.0.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x200/0x200,reg4=0/0x100000 actions=mod_dl_dst:d2:35:24:7f:3a:f8,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg4=0x100000/0x100000 actions=resubmit(,L3DecTTL) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf actions=resubmit(,EgressMark) 5. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:d2:35:24:7f:3a:f8,resubmit(,EgressMark) 6. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` - Flow 1 is used to forward the packets of non-Service connections between local Pods. - Flow 2 is used to forward the packets of Service connections sourced from local non-AntreaIPAM Pods and destined for external network Endpoint. - Flow 3 is used to forward the packets of Service connections sourced from local AntreaIPAM Pods and destined for external network Endpoint. - Flow 4 is used to forward the packets sourced from local Pods and destined for external network, and the flow is for Egress. - Flow 5 is used to forward the packets sourced from tunnel and destined for external network, and the flow is also for Egress. - Flow 6 is the default flow. `load:0x2->NXM_NX_REG0[4..7]` means `to Gateway`. For request packets sourced from a local Pod and destined for another local Pod, they are expected to be matched by flow 1, however, they can be matched by flow 4, and this leads the unexpected result in antrea-io#3806. In addition, when Egress is enabled, if a local Pod accesses to a Service whose Endpoint is on external Network, the request packets are expected to be matched by flow 4, but they can be also matched by flow 2. To resolve above issues, the new flows are like the follows: ``` 1. table=L3Forwarding, priority=200,ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=10.10.0.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x202/0x20f actions=mod_dl_dst:d2:35:24:7f:3a:f8,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf,reg4=0/0x100000 actions=resubmit(,EgressMark) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:d2:35:24:7f:3a:f8,resubmit(,EgressMark) 5. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` Issue antrea-io#3806 is fixed by promoting the priority of the flow matching packets between local Pods to normal (200). In legacy flow 2, packets of Service connections with unknown destination can be either sourced from local Pods or Antrea gateway. In the new flow 2, only the packets of Service connections sourced Antrea gateway are matched. Packets of Service or non-Service connections sourced from local non-AntreaIPAM Pods are matched by the new flow 3 (Egress is enabled) or the new flow 5 (Egress is disabled). Packets of Service or non-Service connections sourced from local AntreaIPAM Pods and destined for external network are matched by the new flow 5. Other modifications: fix some stale comments. Signed-off-by: Hongliang Liu <lhongliang@vmware.com>

…n in L3ForwardingTable Fix antrea-io#3806 Currently, when Egress and AntreaIPAM are enabled, there are several flows in L3ForwardingTable like the follows: ``` 1. table=L3Forwarding, priority=190,ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=10.10.0.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x200/0x200,reg4=0/0x100000 actions=mod_dl_dst:d2:35:24:7f:3a:f8,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg4=0x100000/0x100000 actions=resubmit(,L3DecTTL) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf actions=resubmit(,EgressMark) 5. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:d2:35:24:7f:3a:f8,resubmit(,EgressMark) 6. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` - Flow 1 is used to forward the packets of non-Service connections between local Pods. - Flow 2 is used to forward the packets of Service connections sourced from local non-AntreaIPAM Pods and destined for external network Endpoint. - Flow 3 is used to forward the packets of Service connections sourced from local AntreaIPAM Pods and destined for external network Endpoint. - Flow 4 is used to forward the packets sourced from local Pods and destined for external network, and the flow is for Egress. - Flow 5 is used to forward the packets sourced from tunnel and destined for external network, and the flow is also for Egress. - Flow 6 is the default flow. `load:0x2->NXM_NX_REG0[4..7]` means `to Gateway`. For request packets sourced from a local Pod and destined for another local Pod, they are expected to be matched by flow 1, however, they can be matched by flow 4, and this leads the unexpected result in antrea-io#3806. In addition, when Egress is enabled, if a local Pod accesses to a Service whose Endpoint is on external Network, the request packets are expected to be matched by flow 4, but they can be also matched by flow 2. To resolve above issues, the new flows are like the follows: ``` 1. table=L3Forwarding, priority=200,ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=10.10.0.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x202/0x20f actions=mod_dl_dst:d2:35:24:7f:3a:f8,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf,reg4=0/0x100000 actions=resubmit(,EgressMark) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:d2:35:24:7f:3a:f8,resubmit(,EgressMark) 5. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` Issue antrea-io#3806 is fixed by promoting the priority of the flow matching packets between local Pods to normal (200). In legacy flow 2, packets of Service connections with unknown destination can be either sourced from local Pods or Antrea gateway. In the new flow 2, only the packets of Service connections sourced Antrea gateway are matched. Packets of connections sourced from local Pods are matched by the new flow 3 (Egress is enabled) or the new flow 5 (Egress is disabled). Packets of connections sourced from local AntreaIPAM Pods (AntreaIPAM is enabled) and destined for external network are matched by the new flow 5. Other modifications: fix some stale comments. Signed-off-by: Hongliang Liu <lhongliang@vmware.com>

…n in L3ForwardingTable Fix antrea-io#3806 Currently, when Egress and AntreaIPAM are enabled, there are several flows in L3ForwardingTable like the follows: ``` 1. table=L3Forwarding, priority=190,ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=10.10.0.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x200/0x200,reg4=0/0x100000 actions=mod_dl_dst:d2:35:24:7f:3a:f8,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg4=0x100000/0x100000 actions=resubmit(,L3DecTTL) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf actions=resubmit(,EgressMark) 5. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:d2:35:24:7f:3a:f8,resubmit(,EgressMark) 6. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` - Flow 1 is used to forward the packets of non-Service connections between local Pods. - Flow 2 is used to forward the packets of Service connections sourced from local non-AntreaIPAM Pods and destined for external network Endpoint. - Flow 3 is used to forward the packets of Service connections sourced from local AntreaIPAM Pods and destined for external network Endpoint. - Flow 4 is used to forward the packets sourced from local Pods and destined for external network, and the flow is for Egress. - Flow 5 is used to forward the packets sourced from tunnel and destined for external network, and the flow is also for Egress. - Flow 6 is the default flow. `load:0x2->NXM_NX_REG0[4..7]` means `to Gateway`. For request packets sourced from a local Pod and destined for another local Pod, they are expected to be matched by flow 1, however, they can be matched by flow 4, and this leads the unexpected result in antrea-io#3806. In addition, when Egress is enabled, if a local Pod accesses to a Service whose Endpoint is on external Network, the request packets are expected to be matched by flow 4, but they can be also matched by flow 2. To resolve above issues, the new flows are like the follows: ``` 1. table=L3Forwarding, priority=200,ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=10.10.0.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x202/0x20f actions=mod_dl_dst:d2:35:24:7f:3a:f8,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf,reg4=0/0x100000 actions=resubmit(,EgressMark) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:d2:35:24:7f:3a:f8,resubmit(,EgressMark) 5. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` Issue antrea-io#3806 is fixed by raising the priority of the legacy flow 1 to normal (200). It is the new flow 1 now. In legacy flow 2, packets of Service connections with unknown destination can be either sourced from local Pods or Antrea gateway. In the new flow 2, only the packets of Service connections sourced Antrea gateway are matched. Packets of Service connections sourced from local Pods and destined for external Packets of connections sourced from local Pods and destined for external network are matched by the new flow 3 (Egress is enabled) or the new flow 5 (Egress is disabled). Packets of connections sourced from local AntreaIPAM Pods (AntreaIPAM is enabled) and destined for external network are matched by the new flow 5. Other modifications: fix some stale comments. Signed-off-by: Hongliang Liu <lhongliang@vmware.com>

…n in L3ForwardingTable Fix antrea-io#3806 Currently, when Egress and AntreaIPAM are enabled, there are several flows in L3ForwardingTable like the follows: ``` 1. table=L3Forwarding, priority=190,ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=10.10.0.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x200/0x200,reg4=0/0x100000 actions=mod_dl_dst:d2:35:24:7f:3a:f8,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg4=0x100000/0x100000 actions=resubmit(,L3DecTTL) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf actions=resubmit(,EgressMark) 5. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:d2:35:24:7f:3a:f8,resubmit(,EgressMark) 6. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` - Flow 1 is used to forward the packets of non-Service connections between local Pods. - Flow 2 is used to forward the packets of Service connections sourced from local non-AntreaIPAM Pods and destined for external network Endpoint. - Flow 3 is used to forward the packets of Service connections sourced from local AntreaIPAM Pods and destined for external network Endpoint. - Flow 4 is used to forward the packets sourced from local Pods and destined for external network, and the flow is for Egress. - Flow 5 is used to forward the packets sourced from tunnel and destined for external network, and the flow is also for Egress. - Flow 6 is the default flow. `load:0x2->NXM_NX_REG0[4..7]` means `to Gateway`. For request packets sourced from a local Pod and destined for another local Pod, they are expected to be matched by flow 1, however, they can be matched by flow 4, and this leads the unexpected result in antrea-io#3806. In addition, when Egress is enabled, if a local Pod accesses to a Service whose Endpoint is on external Network, the request packets are expected to be matched by flow 4, but they can be also matched by flow 2. To resolve above issues, the new flows are like the follows: ``` 1. table=L3Forwarding, priority=200,ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=10.10.0.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x202/0x20f actions=mod_dl_dst:d2:35:24:7f:3a:f8,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf,reg4=0/0x100000 actions=resubmit(,EgressMark) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:d2:35:24:7f:3a:f8,resubmit(,EgressMark) 5. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` Issue antrea-io#3806 is fixed by raising the priority of the legacy flow 1 to normal (200). It is the new flow 1 now. In legacy flow 2, packets of Service connections with unknown destination can be either sourced from local Pods or Antrea gateway. In the new flow 2, only the packets of Service connections sourced Antrea gateway are matched. Packets of Service connections sourced from local Pods and destined for unknown destination (external network) are matched by the new flow 5. Packets of connections sourced from local Pods and destined for external network are matched by the new flow 3 (Egress is enabled) or the new flow 5 (Egress is disabled). Packets of connections sourced from local AntreaIPAM Pods (AntreaIPAM is enabled) and destined for external network are matched by the new flow 5. Other modifications: fix some stale comments. Signed-off-by: Hongliang Liu <lhongliang@vmware.com>

…n in L3ForwardingTable Fix antrea-io#3806 Currently, when Egress and AntreaIPAM are enabled, there are several flows in L3ForwardingTable like the follows: ``` 1. table=L3Forwarding, priority=190,ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=10.10.0.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x200/0x200,reg4=0/0x100000 actions=mod_dl_dst:d2:35:24:7f:3a:f8,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg4=0x100000/0x100000 actions=resubmit(,L3DecTTL) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf actions=resubmit(,EgressMark) 5. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:d2:35:24:7f:3a:f8,resubmit(,EgressMark) 6. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` - Flow 1 is used to forward the packets of non-Service connections between local Pods. - Flow 2 is used to forward the packets of Service connections sourced from local non-AntreaIPAM Pods and destined for external network Endpoint. - Flow 3 is used to forward the packets of Service connections sourced from local AntreaIPAM Pods and destined for external network Endpoint. - Flow 4 is used to forward the packets sourced from local Pods and destined for external network, and the flow is for Egress. - Flow 5 is used to forward the packets sourced from tunnel and destined for external network, and the flow is also for Egress. - Flow 6 is the default flow. `load:0x2->NXM_NX_REG0[4..7]` means `to Gateway`. For request packets sourced from a local Pod and destined for another local Pod, they are expected to be matched by flow 1, however, they can be matched by flow 4, and this leads the unexpected result in antrea-io#3806. In addition, when Egress is enabled, if a local Pod accesses to a Service whose Endpoint is on external Network, the request packets are expected to be matched by flow 4, but they can be also matched by flow 2. To resolve above issues, the new flows are like the follows: ``` 1. table=L3Forwarding, priority=200,ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=10.10.0.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x202/0x20f actions=mod_dl_dst:d2:35:24:7f:3a:f8,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf,reg4=0/0x100000 actions=resubmit(,EgressMark) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:d2:35:24:7f:3a:f8,resubmit(,EgressMark) 5. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` Issue antrea-io#3806 is fixed by promoting the priority of the flow matching packets between local Pods to normal (200). In legacy flow 2, packets of Service connections with unknown destination can be either sourced from local Pods or Antrea gateway. In the new flow 2, only the packets of Service connections sourced Antrea gateway are matched. Packets of Service or non-Service connections sourced from local non-AntreaIPAM Pods are matched by the new flow 3 (Egress is enabled) or the new flow 5 (Egress is disabled). Packets of Service or non-Service connections sourced from local AntreaIPAM Pods and destined for external network are matched by the new flow 5. Other modifications: fix some stale comments. Signed-off-by: Hongliang Liu <lhongliang@vmware.com>

…n in L3ForwardingTable Fix antrea-io#3806 Currently, when Egress and AntreaIPAM are enabled, there are several flows in L3ForwardingTable like the follows: ``` 1. table=L3Forwarding, priority=190,ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=10.10.0.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x200/0x200,reg4=0/0x100000 actions=mod_dl_dst:d2:35:24:7f:3a:f8,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg4=0x100000/0x100000 actions=resubmit(,L3DecTTL) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf actions=resubmit(,EgressMark) 5. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:d2:35:24:7f:3a:f8,resubmit(,EgressMark) 6. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` - Flow 1 is used to forward the packets of non-Service connections between local Pods. - Flow 2 is used to forward the packets of Service connections sourced from local non-AntreaIPAM Pods and destined for external network Endpoint. - Flow 3 is used to forward the packets of Service connections sourced from local AntreaIPAM Pods and destined for external network Endpoint. - Flow 4 is used to forward the packets sourced from local Pods and destined for external network, and the flow is for Egress. - Flow 5 is used to forward the packets sourced from tunnel and destined for external network, and the flow is also for Egress. - Flow 6 is the default flow. `load:0x2->NXM_NX_REG0[4..7]` means `to Gateway`. For request packets sourced from a local Pod and destined for another local Pod, they are expected to be matched by flow 1, however, they can be matched by flow 4, and this leads the unexpected result in antrea-io#3806. In addition, when Egress is enabled, if a local Pod accesses to a Service whose Endpoint is on external Network, the request packets are expected to be matched by flow 4, but they can be also matched by flow 2. To resolve above issues, the new flows are like the follows: ``` 1. table=L3Forwarding, priority=200,ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=10.10.0.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x202/0x20f actions=mod_dl_dst:d2:35:24:7f:3a:f8,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf,reg4=0/0x100000 actions=resubmit(,EgressMark) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:d2:35:24:7f:3a:f8,resubmit(,EgressMark) 5. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` Issue antrea-io#3806 is fixed by raising the priority of the legacy flow 1 to normal (200). It is the new flow 1 now. In legacy flow 2, packets of Service connections with unknown destination can be either sourced from local Pods or Antrea gateway. In the new flow 2, only the packets of Service connections sourced Antrea gateway are matched. Packets of Service connections sourced from local Pods and destined for unknown destination (external network) are matched by the new flow 5. Packets of connections sourced from local Pods and destined for external network are matched by the new flow 3 (Egress is enabled) or the new flow 5 (Egress is disabled). Packets of connections sourced from local AntreaIPAM Pods (AntreaIPAM is enabled) and destined for external network are matched by the new flow 5. Other modifications: fix some stale comments. Signed-off-by: Hongliang Liu <lhongliang@vmware.com>

…n in L3ForwardingTable (#3809) Fix #3806 Currently, when Egress and AntreaIPAM are enabled, there are several flows in L3ForwardingTable like the follows: ``` 1. table=L3Forwarding, priority=190,ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=10.10.0.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x200/0x200,reg4=0/0x100000 actions=mod_dl_dst:d2:35:24:7f:3a:f8,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg4=0x100000/0x100000 actions=resubmit(,L3DecTTL) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf actions=resubmit(,EgressMark) 5. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:d2:35:24:7f:3a:f8,resubmit(,EgressMark) 6. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` - Flow 1 is used to forward the packets of non-Service connections between local Pods. - Flow 2 is used to forward the packets of Service connections sourced from local non-AntreaIPAM Pods and destined for external network Endpoint. - Flow 3 is used to forward the packets of Service connections sourced from local AntreaIPAM Pods and destined for external network Endpoint. - Flow 4 is used to forward the packets sourced from local Pods and destined for external network, and the flow is for Egress. - Flow 5 is used to forward the packets sourced from tunnel and destined for external network, and the flow is also for Egress. - Flow 6 is the default flow. `load:0x2->NXM_NX_REG0[4..7]` means `to Gateway`. For request packets sourced from a local Pod and destined for another local Pod, they are expected to be matched by flow 1, however, they can be matched by flow 4, and this leads the unexpected result in #3806. In addition, when Egress is enabled, if a local Pod accesses to a Service whose Endpoint is on external Network, the request packets are expected to be matched by flow 4, but they can be also matched by flow 2. To resolve above issues, the new flows are like the follows: ``` 1. table=L3Forwarding, priority=200,ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=10.10.0.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x202/0x20f actions=mod_dl_dst:d2:35:24:7f:3a:f8,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf,reg4=0/0x100000 actions=resubmit(,EgressMark) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:d2:35:24:7f:3a:f8,resubmit(,EgressMark) 5. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` Issue #3806 is fixed by raising the priority of the legacy flow 1 to normal (200). It is the new flow 1 now. In legacy flow 2, packets of Service connections with unknown destination can be either sourced from local Pods or Antrea gateway. In the new flow 2, only the packets of Service connections sourced from Antrea gateway are matched. Packets of connections sourced from local Pods and destined for external network are matched by the new flow 3 (Egress is enabled) or the new flow 5 (Egress is disabled). Packets of connections sourced from local AntreaIPAM Pods (AntreaIPAM is enabled) and destined for external network are matched by the new flow 5. Other modifications: fix some stale comments. Signed-off-by: Hongliang Liu <lhongliang@vmware.com>

…n in L3ForwardingTable Fix antrea-io#3806 Currently, when Egress and AntreaIPAM are enabled, there are several flows in L3ForwardingTable like the follows: ``` 1. table=L3Forwarding, priority=190,ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=10.10.0.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x200/0x200,reg4=0/0x100000 actions=mod_dl_dst:d2:35:24:7f:3a:f8,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg4=0x100000/0x100000 actions=resubmit(,L3DecTTL) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf actions=resubmit(,EgressMark) 5. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:d2:35:24:7f:3a:f8,resubmit(,EgressMark) 6. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` - Flow 1 is used to forward the packets of non-Service connections between local Pods. - Flow 2 is used to forward the packets of Service connections sourced from local non-AntreaIPAM Pods and destined for external network Endpoint. - Flow 3 is used to forward the packets of Service connections sourced from local AntreaIPAM Pods and destined for external network Endpoint. - Flow 4 is used to forward the packets sourced from local Pods and destined for external network, and the flow is for Egress. - Flow 5 is used to forward the packets sourced from tunnel and destined for external network, and the flow is also for Egress. - Flow 6 is the default flow. `load:0x2->NXM_NX_REG0[4..7]` means `to Gateway`. For request packets sourced from a local Pod and destined for another local Pod, they are expected to be matched by flow 1, however, they can be matched by flow 4, and this leads the unexpected result in antrea-io#3806. In addition, when Egress is enabled, if a local Pod accesses to a Service whose Endpoint is on external Network, the request packets are expected to be matched by flow 4, but they can be also matched by flow 2. To resolve above issues, the new flows are like the follows: ``` 1. table=L3Forwarding, priority=200,ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=10.10.0.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x202/0x20f actions=mod_dl_dst:d2:35:24:7f:3a:f8,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf,reg4=0/0x100000 actions=resubmit(,EgressMark) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:d2:35:24:7f:3a:f8,resubmit(,EgressMark) 5. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` Issue antrea-io#3806 is fixed by raising the priority of the legacy flow 1 to normal (200). It is the new flow 1 now. In legacy flow 2, packets of Service connections with unknown destination can be either sourced from local Pods or Antrea gateway. In the new flow 2, only the packets of Service connections sourced Antrea gateway are matched. Packets of Service connections sourced from local Pods and destined for unknown destination (external network) are matched by the new flow 5. Packets of connections sourced from local Pods and destined for external network are matched by the new flow 3 (Egress is enabled) or the new flow 5 (Egress is disabled). Packets of connections sourced from local AntreaIPAM Pods (AntreaIPAM is enabled) and destined for external network are matched by the new flow 5. Other modifications: fix some stale comments. Signed-off-by: Hongliang Liu <lhongliang@vmware.com>

…n in L3ForwardingTable (#3878) Fix #3806 Currently, when Egress and AntreaIPAM are enabled, there are several flows in L3ForwardingTable like the follows: ``` 1. table=L3Forwarding, priority=190,ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=10.10.0.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x200/0x200,reg4=0/0x100000 actions=mod_dl_dst:d2:35:24:7f:3a:f8,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg4=0x100000/0x100000 actions=resubmit(,L3DecTTL) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf actions=resubmit(,EgressMark) 5. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:d2:35:24:7f:3a:f8,resubmit(,EgressMark) 6. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` - Flow 1 is used to forward the packets of non-Service connections between local Pods. - Flow 2 is used to forward the packets of Service connections sourced from local non-AntreaIPAM Pods and destined for external network Endpoint. - Flow 3 is used to forward the packets of Service connections sourced from local AntreaIPAM Pods and destined for external network Endpoint. - Flow 4 is used to forward the packets sourced from local Pods and destined for external network, and the flow is for Egress. - Flow 5 is used to forward the packets sourced from tunnel and destined for external network, and the flow is also for Egress. - Flow 6 is the default flow. `load:0x2->NXM_NX_REG0[4..7]` means `to Gateway`. For request packets sourced from a local Pod and destined for another local Pod, they are expected to be matched by flow 1, however, they can be matched by flow 4, and this leads the unexpected result in #3806. In addition, when Egress is enabled, if a local Pod accesses to a Service whose Endpoint is on external Network, the request packets are expected to be matched by flow 4, but they can be also matched by flow 2. To resolve above issues, the new flows are like the follows: ``` 1. table=L3Forwarding, priority=200,ip,reg0=0/0x200,reg8=0/0xfff,nw_dst=10.10.0.0/24 actions=resubmit(,L2ForwardingCalc) 2. table=L3Forwarding, priority=190,ct_mark=0x10/0x10,reg0=0x202/0x20f actions=mod_dl_dst:d2:35:24:7f:3a:f8,load:0x2->NXM_NX_REG0[4..7],resubmit(,L3DecTTL) 3. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x3/0xf,reg4=0/0x100000 actions=resubmit(,EgressMark) 4. table=L3Forwarding, priority=190,ct_state=-rpl+trk,ip,reg0=0x1/0xf actions=mod_dl_dst:d2:35:24:7f:3a:f8,resubmit(,EgressMark) 5. table=L3Forwarding, priority=0 actions=load:0x2->NXM_NX_REG0[4..7],resubmit(,L2ForwardingCalc) ``` Issue #3806 is fixed by raising the priority of the legacy flow 1 to normal (200). It is the new flow 1 now. In legacy flow 2, packets of Service connections with unknown destination can be either sourced from local Pods or Antrea gateway. In the new flow 2, only the packets of Service connections sourced Antrea gateway are matched. Packets of Service connections sourced from local Pods and destined for unknown destination (external network) are matched by the new flow 5. Packets of connections sourced from local Pods and destined for external network are matched by the new flow 3 (Egress is enabled) or the new flow 5 (Egress is disabled). Packets of connections sourced from local AntreaIPAM Pods (AntreaIPAM is enabled) and destined for external network are matched by the new flow 5. Other modifications: fix some stale comments. Signed-off-by: Hongliang Liu <lhongliang@vmware.com>

gran-vmv added the kind/bug Categorizes issue or PR as related to a bug. label May 18, 2022

gran-vmv assigned hongliangl May 18, 2022

hongliangl mentioned this issue May 18, 2022

Optimize the flows for forwarding the packets with unknown destination in L3ForwardingTable #3809

Merged

jianjuns added this to the Antrea v1.7 release milestone May 21, 2022

tnqn mentioned this issue Jun 9, 2022

ipv6 e2e tests keeps failing on TestAntreaPolicy #3842

Closed

tnqn closed this as completed in #3809 Jun 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OVS flows in L3Forwarding table conflict #3806

OVS flows in L3Forwarding table conflict #3806

gran-vmv commented May 18, 2022

gran-vmv commented May 18, 2022

gran-vmv commented May 18, 2022

OVS flows in L3Forwarding table conflict #3806

OVS flows in L3Forwarding table conflict #3806

Comments

gran-vmv commented May 18, 2022

gran-vmv commented May 18, 2022

gran-vmv commented May 18, 2022