EndpointSlice resource not getting created on the Secondary cluster

[yashpratap10]

I deployed Submariner in my multi-server redundant layout where one is a primary server where submariner-broker is installed and the other is a redundant secondary server. It is a globalnet deployment.
I've exported a headless service on the primary-cluster and I'm trying to access it from the secondary-cluster but couldn't. After checking the troubleshooting guide I observed that endpoint-slice for the service exists on the primary-cluster but not on the secondary-cluster. And while checking the logs of Lighthouse-agent in secondary, I found some errors like "error distributing resource: submariner-k8s-broker/-primary-cluster". I've attached the whole snippet of the logs below -

broker -> local for *v1beta1.EndpointSlice: Failed to process object with key "submariner-k8s-broker/<exported-service-name>-primary-cluster": error distributing resource "submariner-k8s-broker/<exported-service-name>-primary-cluster": error creating or updating resource: error creating &unstructured.Unstructured{Object:map[string]interface {}{"addressType":"IPv4", "apiVersion":"discovery.k8s.io/v1beta1", "endpoints":[]interface {}{map[string]interface {}{"addresses":[]interface {}{"<Global_IP>"}, "conditions":map[string]interface {}{"ready":true}, "hostname":"<exported-service-name>-78444c9d5f-2n6dq", "topology":map[string]interface {}{"kubernetes.io/hostname":"<primary-server-hostname>"}}}, "kind":"EndpointSlice", "metadata":map[string]interface {}{"labels":map[string]interface {}{"endpointslice.kubernetes.io/managed-by":"lighthouse-agent.submariner.io", "lighthouse.submariner.io/sourceNamespace":"a1-zone1-pri", "multicluster.kubernetes.io/service-name":"<exported-service-name>", "multicluster.kubernetes.io/source-cluster":"primary-cluster", "submariner-io/clusterID":"primary-cluster", "submariner-io/originatingNamespace":"a1-zone1-pri"}, "name":"supervised-event-correlator-primary-cluster", "namespace":"a1-zone1-pri"}, "ports":[]interface {}{map[string]interface {}{"name":"", "port":80, "protocol":"TCP"}}}}: namespaces "a1-zone1-pri" not found
It is being mentioned in the logs that "namespace not found". Is that a requirement for the service to be accessible that the same namespace should exist in both the clusters?

Environment config:
Rancher Kubernetes engine
Submariner version:v0.12.1
kubernetes version:v1.19.16

Please let me know how can I resolve this issue or any other details needed to look into the issue more closely.

[vthapar]

@yashpratap10 Yes, namespace sameness is a requirement of the spec. Ideally we should mark the Export with relevant error status, we don't do that yet. You can go into discussion behind this on kubernetes/enhancements#1645

[bhavin192]

Though the namespace has to exist for the service to work. The form with clusterid still works: cluster01.podinfo.ns.svc.clusterset.local, is it recommended to use / depend on this DNS to work? Or like the spec says it should be considered as implementation detail for Submariner?

[vthapar]

Though it can be considered an implementation detail by spec, we consider it a bug just a lower priority one. We already have an issue submariner-io/lighthouse#605 for this which has been reopened recently and will be fixed in one of upcoming releases [likely 0.15.0]. In meantime, you can use the clusterid workaround if it helps.