Support shared LoadBalancerIP for multiple Services

Users want to share LoadBalancerIP between multiple Services when they
face external IP shortage. It's possible to do it when the Services
sharing an IP meet the requirements:

* The Services use different ports
* The Services use the `Cluster` external traffic policy, or they have
  identical Endpoints.

However, the ability of using any IP that is already allocated to another
Service may incur a security risk that a Service can "steal" LoadBalancer
traffic intended for another Service.

To support the use case without introducing the security risk, we use
the annotation `service.antrea.io/allow-shared-ip: true` on Services to
restrict IPs that can be shared. Services without the annotation will
continue to have their LoadBalancerIPs exclusively used. Services with
the annotation can share an IP between themselves when requesting the
same IP.

Ideally, we should also check if the Services meet the first two
requirements before assigning the IP to them. However, it's difficult to
prevent Services from being changed to not meet the requirements after
they get the IP assigned. Therefore, we assume that users using the
feature know how to configure Services properly.

Signed-off-by: Quan Tian <quan.tian@broadcom.com>

docs/service-loadbalancer.md

@@ -165,6 +165,63 @@ spec:
   type: LoadBalancer
 ```
 
+By default, Antrea doesn't allocate a single IP to multiple Services. Before
+Antrea v2.1, if multiple Services requested the same IP, only one of them will
+get the IP assigned. Starting with Antrea v2.1, to share an IP between multiple
+Services, you can annotate the Services with
+`service.antrea.io/allow-shared-ip: "true"` when requesting a particular IP.
+Note that the IP will only be shared between Services having the annotation. If
+not all Services are annotated, the IP may either be allocated to one of the
+unannotated Services or shared between the annotated Services, depending on the
+order in which they are processed. For example, the following two Services will
+share an IP:
+
+```yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: my-service-1
+  annotations:
+    service.antrea.io/external-ip-pool: "service-external-ip-pool"
+    service.antrea.io/allow-shared-ip: "true"
+spec:
+  selector:
+    app: MyApp1
+  loadBalancerIP: "10.10.0.2"
+  ports:
+    - protocol: TCP
+      port: 80
+      targetPort: 80
+  type: LoadBalancer
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: my-service-2
+  annotations:
+    service.antrea.io/external-ip-pool: "service-external-ip-pool"
+    service.antrea.io/allow-shared-ip: "true"
+spec:
+  selector:
+    app: MyApp2
+  loadBalancerIP: "10.10.0.2"
+  ports:
+    - protocol: TCP
+      port: 8080
+      targetPort: 8080
+  type: LoadBalancer
+```
+
+Note that sharing LoadBalancerIP between multiple Services only makes sense
+under the following conditions:
+
+* The Services use different ports.
+* The Services use the `Cluster` external traffic policy, or they have
+  identical Endpoints.
+
+Otherwise, the datapath may not work even though the IP is allocated to the
+Services successfully.
+
 #### Validate Service external IP
 
 Once Antrea allocates an external IP for a Service of type LoadBalancer, it

pkg/agent/types/annotations.go

@@ -30,6 +30,9 @@ const (
 	// ServiceExternalIPPoolAnnotationKey is the key of the Service annotation that specifies the Service's desired external IP pool.
 	ServiceExternalIPPoolAnnotationKey string = "service.antrea.io/external-ip-pool"
 
+	// ServiceAllowSharedIPAnnotationKey is the key of the Service annotation that specifies whether the Service is allowed to use shared LoadBalancer IP.
+	ServiceAllowSharedIPAnnotationKey string = "service.antrea.io/allow-shared-ip"
+
 	// ServiceLoadBalancerModeAnnotationKey is the key of the Service annotation that specifies the Service's load balancer mode.
 	ServiceLoadBalancerModeAnnotationKey string = "service.antrea.io/load-balancer-mode"