Add user guide for Egress feature #2371

tnqn · 2021-07-09T14:17:50Z

Signed-off-by: Quan Tian qtian@vmware.com

codecov-commenter · 2021-07-09T14:24:17Z

Codecov Report

Merging #2371 (990c2bb) into main (5313dd7) will decrease coverage by 17.74%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             main    #2371       +/-   ##
===========================================
- Coverage   59.79%   42.05%   -17.75%     
===========================================
  Files         283      146      -137     
  Lines       22118    18136     -3982     
===========================================
- Hits        13226     7627     -5599     
- Misses       7479     9825     +2346     
+ Partials     1413      684      -729

Flag	Coverage Δ
kind-e2e-tests	`?`
unit-tests	`42.05% <ø> (+0.14%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/agent/agent_linux.go	`0.00% <0.00%> (-100.00%)`	⬇️
pkg/ovs/openflow/default.go	`0.00% <0.00%> (-100.00%)`	⬇️
pkg/agent/config/node_config.go	`0.00% <0.00%> (-100.00%)`	⬇️
pkg/agent/cniserver/pod_configuration_linux.go	`0.00% <0.00%> (-100.00%)`	⬇️
pkg/apis/controlplane/v1beta1/register.go	`0.00% <0.00%> (-92.86%)`	⬇️
pkg/ovs/openflow/logs.go	`9.52% <0.00%> (-90.48%)`	⬇️
pkg/apis/controlplane/register.go	`0.00% <0.00%> (-90.00%)`	⬇️
pkg/agent/nodeportlocal/k8s/annotations.go	`0.00% <0.00%> (-83.88%)`	⬇️
pkg/agent/client.go	`0.00% <0.00%> (-77.42%)`	⬇️
pkg/ovs/ovsconfig/ovs_client_linux.go	`0.00% <0.00%> (-76.93%)`	⬇️
... and 234 more

antoninbas

great job with the doc, added some suggestions

antoninbas · 2021-07-09T21:02:55Z

docs/egress.md

+  outside of the cluster, for source tracing in audit logs, or whitelisting
+  by source IP in external firewall, etc.
+
+- You want to limit outgoing external connections leave the cluster via certain


Suggested change

- You want to limit outgoing external connections leave the cluster via certain

- You want to force outgoing external connections to leave the cluster via certain

antoninbas · 2021-07-09T21:03:19Z

docs/egress.md

+  by source IP in external firewall, etc.
+
+- You want to limit outgoing external connections leave the cluster via certain
+  Nodes, for security controls, or due to the network topology restriction.


or due to network topology restrictions.

antoninbas · 2021-07-09T21:04:00Z

docs/egress.md

+IP if it's different from the Node that the Pod runs on and will be SNATed to
+the egress IP when leaving that Node.
+
+You may be interested in using this capability if any of the below are true:


Suggested change

You may be interested in using this capability if any of the below are true:

You may be interested in using this capability if any of the following apply:

antoninbas · 2021-07-09T21:05:30Z

docs/egress.md

+    podSelector:
+      matchLabels:
+        role: web
+  egressIP: 10.10.0.8


Suggested change

egressIP: 10.10.0.8

egressIP: 10.10.0.8 # can be populated by Antrea after assigning an IP from the pool below

antoninbas · 2021-07-09T21:07:55Z

docs/egress.md

+the `egressIP` is allocated from an `externalIPPool`**, i.e. when the
+`externalIPPool` is specified. If the Node hosting the `egressIP` fails, another
+Node will be elected as the new egress Node of this Egress. It will take over
+the IP and sends layer 2 advertisement (for example, Gratuitous ARP for IPv4) to


s/sends/send

antoninbas · 2021-07-09T21:16:16Z

docs/egress.md

+selected Pods to the external network should use. **The IP must be reachable
+from all Nodes.** The IP can be specified when creating the Egress. Starting


The IP must be reachable from all Nodes.

I wonder if this is really necessary. If I have a cluster with Nodes spread over multiple Node subnets, it should be ok to partition my egress IPs over the different subnets, and each egress IP only needs to be reachable from within the subnet in which it is used. I would configure the nodeSelector in my IP pools to only select Nodes within that subnet. But it's probably an unusual scenario and it's easier to frame the requirement as "The IP must be reachable from all Nodes".

The IP must be reachable from all Nodes because we use the egress IPs as tunnel destination (to "encap" the snat IP the traffic should use). As any Pod on any Node can use the egress IPs, the source Node must be able to reach the egress IP, otherwise the tunnel traffic would be dropped.
https://github.com/antrea-io/antrea/blob/main/docs/design/ovs-pipeline.md#snattable-71 has details about the SNAT IP as Tunnel destination IP.

antoninbas · 2021-07-09T21:16:57Z

docs/egress.md

+
+The `ipRanges` field contains a list of IP ranges representing the available IPs
+of this IP pool. Each IP range may consist of a `cidr` or a pair of `start` and
+`end` IPs inclusive.


Suggested change

`end` IPs inclusive.

`end` IPs (which are themselves included in the range).

antoninbas · 2021-07-09T21:18:59Z

docs/egress.md

+Now, the packets from the Pods with label `app=web` in `prod` Namespace to the
+external network will be redirected to `node-4` Node and SNATed to `10.10.0.11`
+while the packets from the Pods with label `app=web` in `staging` Namespace to
+the external network will be redirected to `node-6` Node and SNATed to
+`10.10.0.12`.


in the prod Namespaces
to the node-4 Node
in the staging Namespaces
to the node-6 Node

done. I assume you mean "in the prod Namespace", not "Namespaces"? I used namespace name label so it can only match one namespace.

antoninbas · 2021-07-09T21:20:24Z

docs/egress.md

+`10.10.0.12`.
+
+Finally, if `node-4` Node powers off, `10.10.0.11` will be re-assigned to
+another available Node in a short time and the packets from the Pods with label


s/in a short time/quickly

antoninbas · 2021-07-09T21:21:03Z

docs/egress.md

+Now, the packets from the Pods with in `prod` Namespace to the external network
+will be redirected to `node-4` Node and SNATed to `10.10.0.104` while the
+packets from the Pods in `staging` Namespace to the external network will be
+redirected to `node-5` Node and SNATed to `10.10.0.12`.


same comment as above

docs/egress.md

abhiraut · 2021-07-09T21:30:15Z

docs/egress.md

+The `nodeSelector` field specifies which Nodes the IPs in this pool can be
+assigned to. It's useful when you want to limit egress traffic to certain Nodes.
+The semantics of the selector is the same as those used elsewhere in Kubernetes,
+i.e. both `matchLabel` and `matchExpressions` are supported. It can be empty,


Suggested change

i.e. both `matchLabel` and `matchExpressions` are supported. It can be empty,

i.e. both `matchLabels` and `matchExpressions` are supported. It can be empty,

tnqn

@antoninbas @abhiraut thanks for the review.

tnqn · 2021-07-12T17:10:36Z

docs/egress.md

+IP if it's different from the Node that the Pod runs on and will be SNATed to
+the egress IP when leaving that Node.
+
+You may be interested in using this capability if any of the below are true:


tnqn · 2021-07-12T17:10:54Z

docs/egress.md

+  outside of the cluster, for source tracing in audit logs, or whitelisting
+  by source IP in external firewall, etc.
+
+- You want to limit outgoing external connections leave the cluster via certain


docs/egress.md

tnqn · 2021-07-12T17:17:56Z

docs/egress.md

+  by source IP in external firewall, etc.
+
+- You want to limit outgoing external connections leave the cluster via certain
+  Nodes, for security controls, or due to the network topology restriction.


tnqn · 2021-07-12T17:18:10Z

docs/egress.md

+    podSelector:
+      matchLabels:
+        role: web
+  egressIP: 10.10.0.8


tnqn · 2021-07-12T17:27:09Z

docs/egress.md

+
+The `ipRanges` field contains a list of IP ranges representing the available IPs
+of this IP pool. Each IP range may consist of a `cidr` or a pair of `start` and
+`end` IPs inclusive.


tnqn · 2021-07-12T17:27:14Z

docs/egress.md

+The `nodeSelector` field specifies which Nodes the IPs in this pool can be
+assigned to. It's useful when you want to limit egress traffic to certain Nodes.
+The semantics of the selector is the same as those used elsewhere in Kubernetes,
+i.e. both `matchLabel` and `matchExpressions` are supported. It can be empty,


tnqn · 2021-07-12T17:30:33Z

docs/egress.md

+Now, the packets from the Pods with label `app=web` in `prod` Namespace to the
+external network will be redirected to `node-4` Node and SNATed to `10.10.0.11`
+while the packets from the Pods with label `app=web` in `staging` Namespace to
+the external network will be redirected to `node-6` Node and SNATed to
+`10.10.0.12`.


done. I assume you mean "in the prod Namespace", not "Namespaces"? I used namespace name label so it can only match one namespace.

tnqn · 2021-07-12T17:30:40Z

docs/egress.md

+`10.10.0.12`.
+
+Finally, if `node-4` Node powers off, `10.10.0.11` will be re-assigned to
+another available Node in a short time and the packets from the Pods with label


tnqn · 2021-07-12T17:30:46Z

docs/egress.md

+Now, the packets from the Pods with in `prod` Namespace to the external network
+will be redirected to `node-4` Node and SNATed to `10.10.0.104` while the
+packets from the Pods in `staging` Namespace to the external network will be
+redirected to `node-5` Node and SNATed to `10.10.0.12`.


antoninbas

LGTM

jianjuns

LGTM. Just a minor comment/question.

jianjuns · 2021-07-12T18:52:05Z

docs/egress.md

+
+## Usage examples
+
+### Configuring high available Egress


I thought it should be "highly available", but after some researching I am not very certain now. @antoninbas : comments?

Personally I prefer "Egress with HA", or "auto-failover Egress".

yes, it should either be "highly-available Egress" or "high-availability Egress"

Thanks, updated to "High-Availability", following https://kubernetes.io/docs/tasks/administer-cluster/highly-available-control-plane

Signed-off-by: Quan Tian <qtian@vmware.com>

tnqn · 2021-07-13T06:23:24Z

/skip-all

tnqn requested review from jianjuns, abhiraut and antoninbas July 9, 2021 14:18

antoninbas reviewed Jul 9, 2021

View reviewed changes

abhiraut reviewed Jul 9, 2021

View reviewed changes

antoninbas added this to the Antrea v1.2 release milestone Jul 10, 2021

tnqn commented Jul 12, 2021

View reviewed changes

tnqn force-pushed the egress-doc branch from 28d3a75 to 837c84f Compare July 12, 2021 17:32

antoninbas previously approved these changes Jul 12, 2021

View reviewed changes

abhiraut previously approved these changes Jul 12, 2021

View reviewed changes

jianjuns reviewed Jul 12, 2021

View reviewed changes

tnqn dismissed stale reviews from abhiraut and antoninbas via c9ae608 July 13, 2021 02:42

tnqn force-pushed the egress-doc branch from 837c84f to c9ae608 Compare July 13, 2021 02:42

Add user guide for Egress feature

990c2bb

Signed-off-by: Quan Tian <qtian@vmware.com>

tnqn force-pushed the egress-doc branch from c9ae608 to 990c2bb Compare July 13, 2021 02:44

antoninbas approved these changes Jul 13, 2021

View reviewed changes

jianjuns approved these changes Jul 13, 2021

View reviewed changes

tnqn merged commit 744b853 into antrea-io:main Jul 13, 2021

tnqn deleted the egress-doc branch July 13, 2021 06:23

tnqn mentioned this pull request Aug 24, 2021

Support automatic failover for Egress #2128

Closed

6 tasks

	- You want to limit outgoing external connections leave the cluster via certain
	- You want to force outgoing external connections to leave the cluster via certain

	You may be interested in using this capability if any of the below are true:
	You may be interested in using this capability if any of the following apply:

	egressIP: 10.10.0.8
	egressIP: 10.10.0.8 # can be populated by Antrea after assigning an IP from the pool below

		selected Pods to the external network should use. **The IP must be reachable
		from all Nodes.** The IP can be specified when creating the Egress. Starting

	`end` IPs inclusive.
	`end` IPs (which are themselves included in the range).

	i.e. both `matchLabel` and `matchExpressions` are supported. It can be empty,
	i.e. both `matchLabels` and `matchExpressions` are supported. It can be empty,

Add user guide for Egress feature #2371

Add user guide for Egress feature #2371

Conversation

tnqn commented Jul 9, 2021

codecov-commenter commented Jul 9, 2021 • edited Loading

Codecov Report

antoninbas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tnqn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antoninbas left a comment

Choose a reason for hiding this comment

jianjuns left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tnqn commented Jul 13, 2021

codecov-commenter commented Jul 9, 2021 •

edited

Loading