Changes to make lighthouse MCS API complaint

[vthapar]

Currently Lighthouse implementation of MCS API differs on 2 key points:

1. There is no VIP allocated for a MultiCluster Service.
2. EndpointSlices don't carry PodIPs for ClusterIP services.

This Enhancement proposes design changes to support above two in a user configurable way without impacting existing deployments.

[submariner-bot]

🤖 Created branch: z_pr230/vthapar/mcs-compliance

[tpantelis]

Suggested change

	When creating `ServiceExport`, users will have option to add annotation for enabling VIP for that Service and another
	to use PodIPs in `EndpointSlices`. These annotations will also be added to the aggregated `ServiceImport`.
	When creating `ServiceExport`, users will have the option to add an annotation for enabling VIP for that Service. This annotation will also be added to the aggregated `ServiceImport`.

[vthapar]

Done

[tpantelis]

Suggested change

	Configuration flags will also be added to `Submariner` and `ServiceDiscovery` CRDs, and `subctl` to set these options at
	global level. This is for deployments where user only wants to use one specific option and doesn't want to set the
	annotation explicitly for each `ServiceExport`.
	A configuration flag will also be added to the `Submariner` and `ServiceDiscovery` CRDs, and `subctl` to set the options at
	the global level. This is for deployments where the user wants the option set for all exported Services and doesn't want to set the
	annotation explicitly for each `ServiceExport`.

[vthapar]

Done

[tpantelis]

Suggested change

	4. If `ServiceImport` exists, compare the combination of annotations on `ServiceExport` and global flags against
	annotations in `ServiceImport` for any conflicts.
	5. In case of conflict, set `ServiceExportConflict` condition on the `ServiceExport` and attempt conflict resolution.
	6. In case conflict can't be resolved, don't export the service.
	4. If a `ServiceImport` exists, compare the annotation on the `ServiceExport` and global flag against
	the annotation in `ServiceImport` for any conflicts.
	5. In case of conflict, set `ServiceExportConflict` condition on the `ServiceExport` and continue with export.

[vthapar]

Done

[yboaron]

Some corner case Qs:

Regarding
If Submariner is uninstalled on the cluster that allocated a given VIP, VIP is not changed on ServiceImport
Don't we delete all serviceimport CRs when submariner is uninstalled ?

Regarding
VIP allocation is done by the first cluster to export the Service.
do we need to implement some lock mechanism to resolve race condition in case two clusters export service at the same time ?

[tpantelis]

No - that’s handled by using K8s retry on conflict functionality.

[vthapar]

Some corner case Qs:

Regarding If Submariner is uninstalled on the cluster that allocated a given VIP, VIP is not changed on ServiceImport Don't we delete all serviceimport CRs when submariner is uninstalled ?
Aggregated ServiceImport will still exist if service is exported on other clusters. Deletion will only happen if this was the last cluster to export service, in which case it will be deleted. Point here is that if the cluster allocating VIP is no longer there [deleted or submariner removed] we don't try to change the VIP as it will impact existing clients.

Do we expect that cluster will get the same VIPs cidr subset after reinstall ?
No. We don't make any assumptions and it is also possible that some other cluster gets that CIDR. We can see this happen with Globalnet.

Regarding VIP allocation is done by the first cluster to export the Service. do we need to implement some lock mechanism to resolve race condition in case two clusters export service at the same time ?
Not needed. As both try to create Aggregated ServiceImport, only one will be able to do so. Other will get conflict with Resource already exists error.

[submariner-bot]

🤖 Closed branches: [z_pr230/vthapar/mcs-compliance]