Egress QoS support

Add `bandwidth` to Egress specifying the rate limit of north-south egress traffic of this Egress. All backend workloads selected by a rate-limited Egress share the same bandwidth while sending egress traffic via this Egress. Signed-off-by: graysonwu <wgrayson@vmware.com>
antrea-io · Oct 20, 2023 · 45a079e · 45a079e
1 parent 6771518
commit 45a079e
Show file tree

Hide file tree

Showing 45 changed files with 1,141 additions and 202 deletions.
diff --git a/build/charts/antrea/conf/antrea-agent.conf b/build/charts/antrea/conf/antrea-agent.conf
@@ -75,6 +75,9 @@ featureGates:
 # Allow users to specify the load balancer mode as DSR (Direct Server Return).
 {{- include "featureGate" (dict "featureGates" .Values.featureGates "name" "LoadBalancerModeDSR" "default" false) }}
 
+# Enable Egress traffic shaping.
+{{- include "featureGate" (dict "featureGates" .Values.featureGates "name" "EgressTrafficShaping" "default" false) }}
+
 # Name of the OpenVSwitch bridge antrea-agent will create and use.
 # Make sure it doesn't conflict with your existing OpenVSwitch bridges.
 ovsBridge: {{ .Values.ovs.bridgeName | quote }}

diff --git a/build/charts/antrea/crds/egress.yaml b/build/charts/antrea/crds/egress.yaml
@@ -224,6 +224,16 @@ spec:
                   type: array
                   items:
                     type: string
+                bandwidth:
+                  type: object
+                  required:
+                    - rate
+                    - burst
+                  properties:
+                    rate:
+                      type: string
+                    burst:
+                      type: string
             status:
               type: object
               properties:

diff --git a/build/yamls/antrea-aks.yml b/build/yamls/antrea-aks.yml
@@ -2360,6 +2360,16 @@ spec:
                   type: array
                   items:
                     type: string
+                bandwidth:
+                  type: object
+                  required:
+                    - rate
+                    - burst
+                  properties:
+                    rate:
+                      type: string
+                    burst:
+                      type: string
             status:
               type: object
               properties:
@@ -5530,6 +5540,9 @@ data:
     # Allow users to specify the load balancer mode as DSR (Direct Server Return).
     #  LoadBalancerModeDSR: false
 
+    # Enable Egress traffic shaping.
+    #  EgressTrafficShaping: false
+
     # Name of the OpenVSwitch bridge antrea-agent will create and use.
     # Make sure it doesn't conflict with your existing OpenVSwitch bridges.
     ovsBridge: "br-int"
@@ -6821,7 +6834,7 @@ spec:
         kubectl.kubernetes.io/default-container: antrea-agent
         # Automatically restart Pods with a RollingUpdate if the ConfigMap changes
         # See https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
-        checksum/config: d6e7f13a7366490d3b4fe8b7aa6183eef4477d1c7bef7337b24af502a50ca2da
+        checksum/config: 879cb25d5b25e7bb498d1c40d3b9b5171256a26e6d714e619723ae0f5d264688
       labels:
         app: antrea
         component: antrea-agent
@@ -7062,7 +7075,7 @@ spec:
       annotations:
         # Automatically restart Pod if the ConfigMap changes
         # See https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
-        checksum/config: d6e7f13a7366490d3b4fe8b7aa6183eef4477d1c7bef7337b24af502a50ca2da
+        checksum/config: 879cb25d5b25e7bb498d1c40d3b9b5171256a26e6d714e619723ae0f5d264688
       labels:
         app: antrea
         component: antrea-controller

diff --git a/build/yamls/antrea-crds.yml b/build/yamls/antrea-crds.yml
@@ -2351,6 +2351,16 @@ spec:
                   type: array
                   items:
                     type: string
+                bandwidth:
+                  type: object
+                  required:
+                    - rate
+                    - burst
+                  properties:
+                    rate:
+                      type: string
+                    burst:
+                      type: string
             status:
               type: object
               properties:

diff --git a/build/yamls/antrea-eks.yml b/build/yamls/antrea-eks.yml
@@ -2360,6 +2360,16 @@ spec:
                   type: array
                   items:
                     type: string
+                bandwidth:
+                  type: object
+                  required:
+                    - rate
+                    - burst
+                  properties:
+                    rate:
+                      type: string
+                    burst:
+                      type: string
             status:
               type: object
               properties:
@@ -5530,6 +5540,9 @@ data:
     # Allow users to specify the load balancer mode as DSR (Direct Server Return).
     #  LoadBalancerModeDSR: false
 
+    # Enable Egress traffic shaping.
+    #  EgressTrafficShaping: false
+
     # Name of the OpenVSwitch bridge antrea-agent will create and use.
     # Make sure it doesn't conflict with your existing OpenVSwitch bridges.
     ovsBridge: "br-int"
@@ -6821,7 +6834,7 @@ spec:
         kubectl.kubernetes.io/default-container: antrea-agent
         # Automatically restart Pods with a RollingUpdate if the ConfigMap changes
         # See https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
-        checksum/config: d6e7f13a7366490d3b4fe8b7aa6183eef4477d1c7bef7337b24af502a50ca2da
+        checksum/config: 879cb25d5b25e7bb498d1c40d3b9b5171256a26e6d714e619723ae0f5d264688
       labels:
         app: antrea
         component: antrea-agent
@@ -7063,7 +7076,7 @@ spec:
       annotations:
         # Automatically restart Pod if the ConfigMap changes
         # See https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
-        checksum/config: d6e7f13a7366490d3b4fe8b7aa6183eef4477d1c7bef7337b24af502a50ca2da
+        checksum/config: 879cb25d5b25e7bb498d1c40d3b9b5171256a26e6d714e619723ae0f5d264688
       labels:
         app: antrea
         component: antrea-controller

diff --git a/build/yamls/antrea-gke.yml b/build/yamls/antrea-gke.yml
@@ -2360,6 +2360,16 @@ spec:
                   type: array
                   items:
                     type: string
+                bandwidth:
+                  type: object
+                  required:
+                    - rate
+                    - burst
+                  properties:
+                    rate:
+                      type: string
+                    burst:
+                      type: string
             status:
               type: object
               properties:
@@ -5530,6 +5540,9 @@ data:
     # Allow users to specify the load balancer mode as DSR (Direct Server Return).
     #  LoadBalancerModeDSR: false
 
+    # Enable Egress traffic shaping.
+    #  EgressTrafficShaping: false
+
     # Name of the OpenVSwitch bridge antrea-agent will create and use.
     # Make sure it doesn't conflict with your existing OpenVSwitch bridges.
     ovsBridge: "br-int"
@@ -6821,7 +6834,7 @@ spec:
         kubectl.kubernetes.io/default-container: antrea-agent
         # Automatically restart Pods with a RollingUpdate if the ConfigMap changes
         # See https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
-        checksum/config: 4b6f93df4bcfc9e934772e87a653ce2b119700bf09a13792dda394470443f5aa
+        checksum/config: d1a6a70a1ce96d6a297ec9fe9aa6d9e39808b6f53867fec24c5950e12360290c
       labels:
         app: antrea
         component: antrea-agent
@@ -7060,7 +7073,7 @@ spec:
       annotations:
         # Automatically restart Pod if the ConfigMap changes
         # See https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
-        checksum/config: 4b6f93df4bcfc9e934772e87a653ce2b119700bf09a13792dda394470443f5aa
+        checksum/config: d1a6a70a1ce96d6a297ec9fe9aa6d9e39808b6f53867fec24c5950e12360290c
       labels:
         app: antrea
         component: antrea-controller

diff --git a/build/yamls/antrea-ipsec.yml b/build/yamls/antrea-ipsec.yml
@@ -2360,6 +2360,16 @@ spec:
                   type: array
                   items:
                     type: string
+                bandwidth:
+                  type: object
+                  required:
+                    - rate
+                    - burst
+                  properties:
+                    rate:
+                      type: string
+                    burst:
+                      type: string
             status:
               type: object
               properties:
@@ -5543,6 +5553,9 @@ data:
     # Allow users to specify the load balancer mode as DSR (Direct Server Return).
     #  LoadBalancerModeDSR: false
 
+    # Enable Egress traffic shaping.
+    #  EgressTrafficShaping: false
+
     # Name of the OpenVSwitch bridge antrea-agent will create and use.
     # Make sure it doesn't conflict with your existing OpenVSwitch bridges.
     ovsBridge: "br-int"
@@ -6834,7 +6847,7 @@ spec:
         kubectl.kubernetes.io/default-container: antrea-agent
         # Automatically restart Pods with a RollingUpdate if the ConfigMap changes
         # See https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
-        checksum/config: e62ec96721f017ce67609370b6d18c96425135721347e0d3af6c7712df0bf7ca
+        checksum/config: fc6bb433e21947d0651298d18e983e73a6f9d476c67ee77b1a0e87950af515f5
         checksum/ipsec-secret: d0eb9c52d0cd4311b6d252a951126bf9bea27ec05590bed8a394f0f792dcb2a4
       labels:
         app: antrea
@@ -7119,7 +7132,7 @@ spec:
       annotations:
         # Automatically restart Pod if the ConfigMap changes
         # See https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
-        checksum/config: e62ec96721f017ce67609370b6d18c96425135721347e0d3af6c7712df0bf7ca
+        checksum/config: fc6bb433e21947d0651298d18e983e73a6f9d476c67ee77b1a0e87950af515f5
       labels:
         app: antrea
         component: antrea-controller

diff --git a/build/yamls/antrea.yml b/build/yamls/antrea.yml
@@ -2360,6 +2360,16 @@ spec:
                   type: array
                   items:
                     type: string
+                bandwidth:
+                  type: object
+                  required:
+                    - rate
+                    - burst
+                  properties:
+                    rate:
+                      type: string
+                    burst:
+                      type: string
             status:
               type: object
               properties:
@@ -5530,6 +5540,9 @@ data:
     # Allow users to specify the load balancer mode as DSR (Direct Server Return).
     #  LoadBalancerModeDSR: false
 
+    # Enable Egress traffic shaping.
+    #  EgressTrafficShaping: false
+
     # Name of the OpenVSwitch bridge antrea-agent will create and use.
     # Make sure it doesn't conflict with your existing OpenVSwitch bridges.
     ovsBridge: "br-int"
@@ -6821,7 +6834,7 @@ spec:
         kubectl.kubernetes.io/default-container: antrea-agent
         # Automatically restart Pods with a RollingUpdate if the ConfigMap changes
         # See https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
-        checksum/config: 9c01298a6bca328841f5a2dbfda10dd531b76a0e1cda22588307e4cf97124a13
+        checksum/config: ea08c7827426718cc2019df3e71ab2b0524a3205461f1263208a9118baeedadc
       labels:
         app: antrea
         component: antrea-agent
@@ -7060,7 +7073,7 @@ spec:
       annotations:
         # Automatically restart Pod if the ConfigMap changes
         # See https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments
-        checksum/config: 9c01298a6bca328841f5a2dbfda10dd531b76a0e1cda22588307e4cf97124a13
+        checksum/config: ea08c7827426718cc2019df3e71ab2b0524a3205461f1263208a9118baeedadc
       labels:
         app: antrea
         component: antrea-controller

diff --git a/cmd/antrea-agent/agent.go b/cmd/antrea-agent/agent.go
@@ -161,6 +161,7 @@ func run(o *Options) error {
 		features.DefaultFeatureGate.Enabled(features.AntreaPolicy),
 		l7NetworkPolicyEnabled,
 		o.enableEgress,
+		features.DefaultFeatureGate.Enabled(features.EgressTrafficShaping),
 		enableFlowExporter,
 		o.config.AntreaProxy.ProxyAll,
 		features.DefaultFeatureGate.Enabled(features.LoadBalancerModeDSR),
@@ -515,6 +516,7 @@ func run(o *Options) error {
 		egressController, err = egress.NewEgressController(
 			ofClient, antreaClientProvider, crdClient, ifaceStore, routeClient, nodeConfig.Name, nodeConfig.NodeTransportInterfaceName,
 			memberlistCluster, egressInformer, nodeInformer, podUpdateChannel, serviceCIDRProvider, o.config.Egress.MaxEgressIPsPerNode,
+			features.DefaultFeatureGate.Enabled(features.EgressTrafficShaping),
 		)
 		if err != nil {
 			return fmt.Errorf("error creating new Egress controller: %v", err)

diff --git a/docs/egress.md b/docs/egress.md
@@ -9,6 +9,7 @@
   - [AppliedTo](#appliedto)
   - [EgressIP](#egressip)
   - [ExternalIPPool](#externalippool)
+  - [TrafficShaping](#trafficshaping)
 - [The ExternalIPPool resource](#the-externalippool-resource)
   - [IPRanges](#ipranges)
   - [NodeSelector](#nodeselector)
@@ -127,6 +128,44 @@ The `externalIPPool` field specifies the name of the `ExternalIPPool` that the
 be assigned to. It can be empty, which means users should assign the `egressIP`
 to one Node manually.
 
+### TrafficShaping
+
+The `bandwidth` field impose rate-limiting on egress traffic of the Egress. `rate`
+specifies the maximum transmission rate. `burst` specifies maximum burst for throttle.
+`rate` and `burst` should follow the k8s Quantity definition
+[here](https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/quantity/),
+e.g. 300k, 100M, 2G. All backend workloads selected by a rate-limited Egress share the
+same bandwidth while sending egress traffic via this Egress.
+
+**Note**: Traffic shaping is currently in alpha version. To use this feature, users should
+enable `EgressTrafficShaping` feature gate. Each Egress IP can be applied one bandwidth only.
+If multiple Egresses use the same IP but configure different bandwidths, the effective
+bandwidth will be selected randomly. The effective use of the `bandwidth`
+function hinges on the prerequisite of OVS meter support.
+
+An Egress with traffic shaping example:
+
+```yaml
+apiVersion: crd.antrea.io/v1beta1
+kind: Egress
+metadata:
+  name: egress-prod-web
+spec:
+  appliedTo:
+    namespaceSelector:
+      matchLabels:
+        env: prod
+    podSelector:
+      matchLabels:
+        role: web
+  egressIP: 10.10.0.8
+  bandwidth:
+    rate: 800M
+    burst: 2G
+status:
+  egressNode: node01
+```
+
 ## The ExternalIPPool resource
 
 ExternalIPPool defines one or multiple IP ranges that can be used in the