Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ServiceImport is not recreated if Service is deleted and recreated #693

Closed
davidohana opened this issue Feb 24, 2022 · 9 comments · Fixed by #1336
Closed

ServiceImport is not recreated if Service is deleted and recreated #693

davidohana opened this issue Feb 24, 2022 · 9 comments · Fixed by #1336
Assignees
@tpantelis
Labels
bug Something isn't working priority:medium

Comments

@davidohana
Copy link
Contributor

davidohana commented Feb 24, 2022 •
edited
Loading

What happened:
If a service is created after ServiceExport already exist, ServiceImport is not created.

What you expected to happen:
ServiceImport shall be created no matter the order of service & service export creation.

How to reproduce it (as minimally and precisely as possible):

  1. Deploy service monti-host and backing pods
  2. Create ServiceExport for monti-host with subctl
  3. (LH Agent creates service import)
  4. Delete service monti-host
  5. (LH Agent deletes service import)
  6. Re-create service monti-host
  7. Service import is not created

Anything else we need to know?:
Working on a fix for that in the Axon fork.

This is the status of the ServiceExport after step 5 is done:

kubectl get serviceexports -A -o yaml
apiVersion: v1
items:
- apiVersion: multicluster.x-k8s.io/v1alpha1
  kind: ServiceExport
  metadata:
    creationTimestamp: "2022-02-24T11:30:14Z"
    generation: 1
    name: monti-host
    namespace: monti
    resourceVersion: "14265"
    uid: df190782-ac4d-4e0b-abf7-d1e2f2060693
  status:
    conditions:
    - lastTransitionTime: "2022-02-24T11:30:14Z"
      message: Service doesn't have a global IP yet
      reason: ServiceGlobalIPUnavailable
      status: "False"
      type: Valid
    - lastTransitionTime: "2022-02-24T11:35:25Z"
      message: Awaiting sync of the ServiceImport to the broker
      reason: AwaitingSync
      status: "False"
      type: Valid
    - lastTransitionTime: "2022-02-24T11:35:25Z"
      message: Service was successfully synced to the broker
      reason: ""
      status: "True"
      type: Valid
    - lastTransitionTime: "2022-02-24T11:37:29Z"
      message: Service to be exported doesn't exist
      reason: ServiceUnavailable
      status: "False"
      type: Valid
    - lastTransitionTime: "2022-02-24T11:38:10Z"
      message: Service doesn't have a global IP yet
      reason: ServiceGlobalIPUnavailable
      status: "False"
      type: Valid
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
@davidohana davidohana added the bug Something isn't working label Feb 24, 2022
@vthapar
Copy link
Contributor

vthapar commented Feb 24, 2022

From the Export status it looks like GlobalIP is not allocated. @davidohana Can you check the globalingressIPs and check if globalIP is allocated for this service or not?

@davidohana davidohana changed the title ServiceImport is not created if Service is created ServiceImport is not recreated if Service is deleted and recreated Feb 24, 2022
@davidohana
Copy link
Contributor Author

davidohana commented Feb 24, 2022 •
edited
Loading

@vthapar no globalingressIPs. The recreated service monti-host ands its pods seems healthy.

% kubectl get globalingressips -A
No resources found

% kubectl get services -A
NAMESPACE             NAME                                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
default               kubernetes                              ClusterIP   100.0.0.1       <none>        443/TCP                  4h28m
kube-system           kube-dns                                ClusterIP   100.0.0.10      <none>        53/UDP,53/TCP,9153/TCP   4h28m
monti                 monti-host                              ClusterIP   100.0.17.18     <none>        8099/TCP                 88s
submariner-operator   submariner-gateway-metrics              ClusterIP   100.0.52.5      <none>        8080/TCP                 4m25s
submariner-operator   submariner-globalnet-metrics            ClusterIP   100.0.106.124   <none>        8081/TCP                 4m24s
submariner-operator   submariner-lighthouse-agent-metrics     ClusterIP   100.0.225.115   <none>        8082/TCP                 4m22s
submariner-operator   submariner-lighthouse-coredns           ClusterIP   100.0.156.171   <none>        53/UDP                   4m21s
submariner-operator   submariner-lighthouse-coredns-metrics   ClusterIP   100.0.186.61    <none>        9153/TCP                 4m21s
submariner-operator   submariner-operator-metrics             ClusterIP   100.0.116.214   <none>        8383/TCP,8686/TCP        4m34s

% kubectl get pods -A
NAMESPACE             NAME                                             READY   STATUS    RESTARTS   AGE
kube-system           coredns-74ff55c5b-rh7nw                          1/1     Running   0          4h31m
kube-system           coredns-74ff55c5b-wdsx6                          1/1     Running   0          4h31m
kube-system           etcd-cluster1-control-plane                      1/1     Running   0          4h31m
kube-system           kube-apiserver-cluster1-control-plane            1/1     Running   0          4h31m
kube-system           kube-controller-manager-cluster1-control-plane   1/1     Running   0          4h31m
kube-system           kube-proxy-jc957                                 1/1     Running   0          4h31m
kube-system           kube-proxy-qxcq8                                 1/1     Running   0          4h31m
kube-system           kube-scheduler-cluster1-control-plane            1/1     Running   0          4h31m
kube-system           weave-net-7lsxf                                  2/2     Running   1          4h31m
kube-system           weave-net-tpt9f                                  2/2     Running   1          4h31m
local-path-storage    local-path-provisioner-547f784dff-jk9f4          1/1     Running   0          4h31m
monti                 monti-deployment-server-6fb466fbfd-9xmrl         1/1     Running   0          6m25s
monti                 monti-deployment-server-6fb466fbfd-hdsqj         1/1     Running   0          6m25s
submariner-operator   submariner-gateway-sf4df                         1/1     Running   0          7m16s
submariner-operator   submariner-globalnet-6gwnq                       1/1     Running   0          7m15s
submariner-operator   submariner-lighthouse-agent-64c7cc8589-4m88m     1/1     Running   0          7m13s
submariner-operator   submariner-lighthouse-coredns-68766559c4-fzbs8   1/1     Running   0          7m12s
submariner-operator   submariner-lighthouse-coredns-68766559c4-g7rqw   1/1     Running   0          7m12s
submariner-operator   submariner-operator-5588b78fb5-blhtz             1/1     Running   1          7m31s
submariner-operator   submariner-routeagent-nvdr8                      1/1     Running   0          7m15s
submariner-operator   submariner-routeagent-vjvqb                      1/1     Running   0          7m15s

I0224 14:47:17.594283       1 agent.go:234] ServiceExport monti/monti-host created
I0224 14:47:17.594472       1 agent.go:294] Service to be exported (monti/monti-host) doesn't have a global IP yet
I0224 14:47:17.594495       1 agent.go:386] updateExportedServiceStatus for (monti/monti-host) - Type: "Valid", Status: "False", Reason: "ServiceGlobalIPUnavailable", Message: "Service doesn't have a global IP yet"
I0224 14:47:17.690615       1 agent.go:234] ServiceExport monti/monti-host created
I0224 14:47:17.690927       1 agent.go:294] Service to be exported (monti/monti-host) doesn't have a global IP yet
I0224 14:47:17.691918       1 agent.go:386] updateExportedServiceStatus for (monti/monti-host) - Type: "Valid", Status: "False", Reason: "ServiceGlobalIPUnavailable", Message: "Service doesn't have a global IP yet"
I0224 14:47:17.738332       1 agent.go:234] ServiceExport monti/monti-host created
I0224 14:47:17.738444       1 agent.go:294] Service to be exported (monti/monti-host) doesn't have a global IP yet
I0224 14:47:17.738449       1 agent.go:386] updateExportedServiceStatus for (monti/monti-host) - Type: "Valid", Status: "False", Reason: "ServiceGlobalIPUnavailable", Message: "Service doesn't have a global IP yet"
I0224 14:47:17.827441       1 agent.go:234] ServiceExport monti/monti-host created
I0224 14:47:17.827675       1 agent.go:386] updateExportedServiceStatus for (monti/monti-host) - Type: "Valid", Status: "False", Reason: "AwaitingSync", Message: "Awaiting sync of the ServiceImport to the broker"
I0224 14:47:17.837192       1 agent.go:316] Returning ServiceImport: &v1alpha1.ServiceImport{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"monti-host-monti-cluster1", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"lighthouse.submariner.io/sourceCluster":"cluster1", "lighthouse.submariner.io/sourceName":"monti-host", "lighthouse.submariner.io/sourceNamespace":"monti"}, Annotations:map[string]string{"cluster-ip":"242.254.1.253", "origin-name":"monti-host", "origin-namespace":"monti"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1alpha1.ServiceImportSpec{Ports:[]v1alpha1.ServicePort{v1alpha1.ServicePort{Name:"", Protocol:"TCP", AppProtocol:(*string)(nil), Port:8099}}, IPs:[]string{"242.254.1.253"}, Type:"ClusterSetIP", SessionAffinity:"", SessionAffinityConfig:(*v1.SessionAffinityConfig)(0xc00000e1b0)}, Status:v1alpha1.ServiceImportStatus{Clusters:[]v1alpha1.ClusterStatus{v1alpha1.ClusterStatus{Cluster:"cluster1"}}}}
I0224 14:47:17.849242       1 agent.go:386] updateExportedServiceStatus for (monti/monti-host) - Type: "Valid", Status: "True", Reason: "", Message: "Service was successfully synced to the broker"
I0224 14:47:17.854284       1 serviceimport.go:141] ServiceImport "submariner-operator/monti-host-monti-cluster1" created
I0224 14:47:17.854361       1 endpoint.go:47] Starting Endpoints controller for service monti/monti-host
I0224 14:47:17.869888       1 agent.go:234] ServiceExport monti/monti-host updated
I0224 14:47:17.955465       1 endpoint.go:144] Endpoints monti/monti-host created
I0224 14:47:17.955523       1 endpoint.go:197] Returning EndpointSlice: &v1beta1.EndpointSlice{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"monti-host-cluster1", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"endpointslice.kubernetes.io/managed-by":"lighthouse-agent.submariner.io", "lighthouse.submariner.io/sourceNamespace":"monti", "multicluster.kubernetes.io/service-name":"monti-host", "multicluster.kubernetes.io/source-cluster":"cluster1"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, AddressType:"IPv4", Endpoints:[]v1beta1.Endpoint{v1beta1.Endpoint{Addresses:[]string{"10.0.0.10"}, Conditions:v1beta1.EndpointConditions{Ready:(*bool)(0xc0005474b0)}, Hostname:(*string)(0xc0001f8480), TargetRef:(*v1.ObjectReference)(nil), Topology:map[string]string{"kubernetes.io/hostname":"cluster1-worker"}}, v1beta1.Endpoint{Addresses:[]string{"10.0.0.9"}, Conditions:v1beta1.EndpointConditions{Ready:(*bool)(0xc0005474b1)}, Hostname:(*string)(0xc0001f84f0), TargetRef:(*v1.ObjectReference)(nil), Topology:map[string]string{"kubernetes.io/hostname":"cluster1-worker"}}}, Ports:[]v1beta1.EndpointPort{v1beta1.EndpointPort{Name:(*string)(0xc00008c300), Protocol:(*v1.Protocol)(0xc00008c318), Port:(*int32)(0xc00008c310), AppProtocol:(*string)(nil)}}}
I0224 14:47:53.736412       1 agent.go:386] updateExportedServiceStatus for (monti/monti-host) - Type: "Valid", Status: "False", Reason: "ServiceUnavailable", Message: "Service to be exported doesn't exist"
I0224 14:47:53.761484       1 endpoint.go:133] Endpoints monti/monti-host deleted
I0224 14:47:53.800009       1 agent.go:234] ServiceExport monti/monti-host updated
I0224 14:47:53.800093       1 agent.go:251] Service to be exported (monti/monti-host) doesn't exist
I0224 14:47:53.800248       1 agent.go:386] updateExportedServiceStatus for (monti/monti-host) - Type: "Valid", Status: "False", Reason: "ServiceUnavailable", Message: "Service to be exported doesn't exist"
I0224 14:47:53.821333       1 agent.go:234] ServiceExport monti/monti-host updated
I0224 14:47:53.821385       1 agent.go:251] Service to be exported (monti/monti-host) doesn't exist
I0224 14:47:53.821399       1 agent.go:386] updateExportedServiceStatus for (monti/monti-host) - Type: "Valid", Status: "False", Reason: "ServiceUnavailable", Message: "Service to be exported doesn't exist"
I0224 14:47:53.874297       1 serviceimport.go:141] ServiceImport "submariner-operator/monti-host-monti-cluster1" deleted
I0224 14:47:53.887906       1 agent.go:234] ServiceExport monti/monti-host updated
I0224 14:47:53.887972       1 agent.go:251] Service to be exported (monti/monti-host) doesn't exist
I0224 14:47:53.887977       1 agent.go:386] updateExportedServiceStatus for (monti/monti-host) - Type: "Valid", Status: "False", Reason: "ServiceUnavailable", Message: "Service to be exported doesn't exist"
I0224 14:47:53.958739       1 agent.go:234] ServiceExport monti/monti-host updated
I0224 14:47:53.958796       1 agent.go:251] Service to be exported (monti/monti-host) doesn't exist
I0224 14:47:53.958810       1 agent.go:386] updateExportedServiceStatus for (monti/monti-host) - Type: "Valid", Status: "False", Reason: "ServiceUnavailable", Message: "Service to be exported doesn't exist"
I0224 14:47:54.055659       1 agent.go:234] ServiceExport monti/monti-host updated
I0224 14:47:54.055727       1 agent.go:251] Service to be exported (monti/monti-host) doesn't exist
I0224 14:47:54.055739       1 agent.go:386] updateExportedServiceStatus for (monti/monti-host) - Type: "Valid", Status: "False", Reason: "ServiceUnavailable", Message: "Service to be exported doesn't exist"
I0224 14:47:54.221205       1 agent.go:234] ServiceExport monti/monti-host updated
I0224 14:47:54.221261       1 agent.go:251] Service to be exported (monti/monti-host) doesn't exist
I0224 14:47:54.221275       1 agent.go:386] updateExportedServiceStatus for (monti/monti-host) - Type: "Valid", Status: "False", Reason: "ServiceUnavailable", Message: "Service to be exported doesn't exist"
I0224 14:47:54.547333       1 agent.go:234] ServiceExport monti/monti-host updated
I0224 14:47:54.547377       1 agent.go:251] Service to be exported (monti/monti-host) doesn't exist
I0224 14:47:54.547389       1 agent.go:386] updateExportedServiceStatus for (monti/monti-host) - Type: "Valid", Status: "False", Reason: "ServiceUnavailable", Message: "Service to be exported doesn't exist"
I0224 14:47:55.197554       1 agent.go:234] ServiceExport monti/monti-host updated
I0224 14:47:55.197813       1 agent.go:251] Service to be exported (monti/monti-host) doesn't exist
I0224 14:47:55.198060       1 agent.go:386] updateExportedServiceStatus for (monti/monti-host) - Type: "Valid", Status: "False", Reason: "ServiceUnavailable", Message: "Service to be exported doesn't exist"
I0224 14:47:56.484062       1 agent.go:234] ServiceExport monti/monti-host updated
I0224 14:47:56.484253       1 agent.go:251] Service to be exported (monti/monti-host) doesn't exist
I0224 14:47:56.484552       1 agent.go:386] updateExportedServiceStatus for (monti/monti-host) - Type: "Valid", Status: "False", Reason: "ServiceUnavailable", Message: "Service to be exported doesn't exist"
I0224 14:47:59.051496       1 agent.go:234] ServiceExport monti/monti-host updated
I0224 14:47:59.051569       1 agent.go:251] Service to be exported (monti/monti-host) doesn't exist
I0224 14:47:59.051574       1 agent.go:386] updateExportedServiceStatus for (monti/monti-host) - Type: "Valid", Status: "False", Reason: "ServiceUnavailable", Message: "Service to be exported doesn't exist"
I0224 14:48:04.176542       1 agent.go:234] ServiceExport monti/monti-host updated
I0224 14:48:04.176566       1 agent.go:251] Service to be exported (monti/monti-host) doesn't exist
I0224 14:48:04.176566       1 agent.go:386] updateExportedServiceStatus for (monti/monti-host) - Type: "Valid", Status: "False", Reason: "ServiceUnavailable", Message: "Service to be exported doesn't exist"
I0224 14:48:14.422746       1 agent.go:234] ServiceExport monti/monti-host updated
I0224 14:48:14.423006       1 agent.go:251] Service to be exported (monti/monti-host) doesn't exist
I0224 14:48:14.423165       1 agent.go:386] updateExportedServiceStatus for (monti/monti-host) - Type: "Valid", Status: "False", Reason: "ServiceUnavailable", Message: "Service to be exported doesn't exist"
I0224 14:48:34.908001       1 agent.go:234] ServiceExport monti/monti-host updated
I0224 14:48:34.908055       1 agent.go:251] Service to be exported (monti/monti-host) doesn't exist
I0224 14:48:34.908057       1 agent.go:386] updateExportedServiceStatus for (monti/monti-host) - Type: "Valid", Status: "False", Reason: "ServiceUnavailable", Message: "Service to be exported doesn't exist"
I0224 14:49:04.912090       1 agent.go:234] ServiceExport monti/monti-host updated
I0224 14:49:04.912207       1 agent.go:251] Service to be exported (monti/monti-host) doesn't exist
I0224 14:49:04.912257       1 agent.go:386] updateExportedServiceStatus for (monti/monti-host) - Type: "Valid", Status: "False", Reason: "ServiceUnavailable", Message: "Service to be exported doesn't exist"
I0224 14:49:34.917094       1 agent.go:234] ServiceExport monti/monti-host updated
I0224 14:49:34.917346       1 agent.go:294] Service to be exported (monti/monti-host) doesn't have a global IP yet
I0224 14:49:34.917383       1 agent.go:386] updateExportedServiceStatus for (monti/monti-host) - Type: "Valid", Status: "False", Reason: "ServiceGlobalIPUnavailable", Message: "Service doesn't have a global IP yet"
I0224 14:50:04.929462       1 agent.go:234] ServiceExport monti/monti-host updated

@vthapar
Copy link
Contributor

vthapar commented Feb 24, 2022

@davidohana Take a look at logs of submariner-globalnet pod if there are any errors in there. IngressIPs should've been allocated for every exported service.

@davidohana
Copy link
Contributor Author

davidohana commented Feb 27, 2022 •
edited
Loading

This is the submariner-globalnet pod output:

+ trap 'exit 1' SIGTERM SIGINT
+ SUBMARINER_VERBOSITY=2
+ '[' '' == true ']'
+ DEBUG=-v=2
+ exec submariner-globalnet -v=2 -alsologtostderr
I0224 14:46:14.542862       1 main.go:58] Starting submariner-globalnet{cluster1 submariner-operator []}
W0224 14:46:14.550414       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0224 14:46:20.849135       1 gateway_monitor.go:117] Starting GatewayMonitor to monitor the active Gateway node in the cluster.
I0224 14:46:20.949333       1 gateway_monitor.go:327] Install/ensure SUBMARINER-GN-MARK chain exists
I0224 14:46:20.949922       1 gateway_monitor.go:144] In processNextEndpoint, endpoint info: {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-21-0-3","namespace":"submariner-operator","uid":"07fd5d1c-5c0c-4d08-a646-03f0b7d92efe","resourceVersion":"44168","generation":2,"creationTimestamp":"2022-02-24T14:46:18Z","managedFields":[{"manager":"submariner-gateway","operation":"Update","apiVersion":"submariner.io/v1","time":"2022-02-24T14:46:18Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{".":{},"f:backend":{},"f:backend_config":{".":{},"f:natt-discovery-port":{},"f:preferred-server":{},"f:udp-port":{}},"f:cable_name":{},"f:cluster_id":{},"f:healthCheckIP":{},"f:hostname":{},"f:nat_enabled":{},"f:private_ip":{},"f:public_ip":{},"f:subnets":{}}}}]},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-21-0-3","healthCheckIP":"242.254.1.254","hostname":"cluster1-worker","subnets":["242.254.1.0/24"],"private_ip":"172.21.0.3","public_ip":"159.8.66.226","nat_enabled":false,"backend":"libreswan","backend_config":{"natt-discovery-port":"4490","preferred-server":"false","udp-port":"4500"}}}
I0224 14:46:20.950145       1 gateway_monitor.go:184] Transitioned to gateway node with endpoint private IP 172.21.0.3
I0224 14:46:20.950327       1 gateway_monitor.go:241] On Gateway node - starting controllers
I0224 14:46:20.950331       1 gateway_monitor.go:339] Install/ensure SUBMARINER-GN-INGRESS chain exists
I0224 14:46:20.988448       1 iptables.go:115] In nat table, iptables rule "-j SUBMARINER-GN-INGRESS", exists at index 1.
I0224 14:46:20.988482       1 iptables.go:136] In nat table, iptables rule "-j SUBMARINER-GN-INGRESS", already exists.
I0224 14:46:20.988498       1 gateway_monitor.go:350] Install/ensure SUBMARINER-GN-EGRESS chain exists
I0224 14:46:20.998789       1 gateway_monitor.go:356] Install/ensure SUBMARINER-POSTROUTING chain exists
I0224 14:46:21.013757       1 iptables.go:115] In nat table, iptables rule "-j SUBMARINER-GN-EGRESS", exists at index 1.
I0224 14:46:21.013790       1 iptables.go:136] In nat table, iptables rule "-j SUBMARINER-GN-EGRESS", already exists.
I0224 14:46:21.013798       1 gateway_monitor.go:327] Install/ensure SUBMARINER-GN-MARK chain exists
I0224 14:46:21.035130       1 gateway_monitor.go:376] Install/ensure SM-GN-EGRESS-PODS chain exists
I0224 14:46:21.047743       1 gateway_monitor.go:382] Install/ensure SM-GN-EGRESS-HDLS-PODS chain exists
I0224 14:46:21.057423       1 gateway_monitor.go:388] Install/ensure SM-GN-EGRESS-NS chain exists
I0224 14:46:21.068248       1 gateway_monitor.go:394] Install/ensure SM-GN-EGRESS-CLUSTER chain exists
I0224 14:46:21.204023       1 node_controller.go:44] Creating Node controller
I0224 14:46:21.225331       1 iface.go:204] Installing iptable ingress rules for Node: -p icmp -d 242.254.1.254 -j DNAT --to 10.0.0.1
I0224 14:46:21.244570       1 cluster_egressip_controller.go:47] Creating ClusterGlobalEgressIP controller
I0224 14:46:21.255678       1 cluster_egressip_controller.go:80] Creating ClusterGlobalEgressIP resource "cluster-egress.submariner.io"
I0224 14:46:21.261507       1 global_egressip_controller.go:49] Creating GlobalEgressIP controller
I0224 14:46:21.271290       1 global_ingressip_controller.go:43] Creating GlobalIngressIP controller
I0224 14:46:21.287701       1 service_export_controller.go:44] Creating ServiceExport controller
I0224 14:46:21.302398       1 service_controller.go:39] Creating Service controller
I0224 14:46:21.406951       1 node_controller.go:111] Processing created Node "cluster1-worker"
I0224 14:46:21.505519       1 cluster_egressip_controller.go:126] Processing created ClusterGlobalEgressIP "cluster-egress.submariner.io", Spec.NumberOfIPs: 8, Status: v1.GlobalEgressIPStatus{Conditions:[]v1.Condition(nil), AllocatedIPs:[]string(nil)}
I0224 14:46:21.505597       1 cluster_egressip_controller.go:235] Allocating 8 global IP(s) for "cluster-egress.submariner.io"
I0224 14:46:21.505713       1 iface.go:72] Installing iptable egress rules for Cluster: -p all -s 10.0.0.0/16 -m mark --mark 0xC0000/0xC0000 -j SNAT --to 242.254.1.1-242.254.1.8
I0224 14:46:21.513164       1 iface.go:72] Installing iptable egress rules for Cluster: -p all -s 100.0.0.0/16 -m mark --mark 0xC0000/0xC0000 -j SNAT --to 242.254.1.1-242.254.1.8
I0224 14:46:21.520140       1 base_controllers.go:195] Updated: &v1.GlobalEgressIPStatus{Conditions:[]v1.Condition{v1.Condition{Type:"Allocated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.February, 24, 14, 46, 21, 520114078, time.Local), Reason:"Success", Message:"Allocated 8 global IP(s)"}}, AllocatedIPs:[]string{"242.254.1.1", "242.254.1.2", "242.254.1.3", "242.254.1.4", "242.254.1.5", "242.254.1.6", "242.254.1.7", "242.254.1.8"}}
I0224 14:46:22.108240       1 gateway_monitor.go:311] Successfully started the controllers
I0224 14:46:31.229676       1 gateway_monitor.go:144] In processNextEndpoint, endpoint info: {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-21-0-4","namespace":"submariner-operator","uid":"f197aed4-5baf-4e3b-8a2b-8f2768f9a4cf","resourceVersion":"44205","generation":1,"creationTimestamp":"2022-02-24T14:46:31Z","labels":{"submariner-io/clusterID":"cluster2"},"managedFields":[{"manager":"submariner-gateway","operation":"Update","apiVersion":"submariner.io/v1","time":"2022-02-24T14:46:31Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:submariner-io/clusterID":{}}},"f:spec":{".":{},"f:backend":{},"f:backend_config":{".":{},"f:natt-discovery-port":{},"f:preferred-server":{},"f:udp-port":{}},"f:cable_name":{},"f:cluster_id":{},"f:hostname":{},"f:nat_enabled":{},"f:private_ip":{},"f:public_ip":{},"f:subnets":{}}}}]},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-21-0-4","hostname":"cluster2-worker","subnets":["242.254.2.0/24"],"private_ip":"172.21.0.4","public_ip":"159.8.66.226","nat_enabled":false,"backend":"libreswan","backend_config":{"natt-discovery-port":"4490","preferred-server":"false","udp-port":"4500"}}}
I0224 14:46:31.229772       1 gateway_monitor.go:147] Endpoint "cluster2", host: "cluster2-worker" belongs to a remote cluster
I0224 14:46:31.229812       1 gateway_monitor.go:443] Marking traffic destined to remote cluster: -d 242.254.2.0/24 -j MARK --set-mark 0xC0000/0xC0000
I0224 14:46:31.385685       1 gateway_monitor.go:144] In processNextEndpoint, endpoint info: {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-21-0-4","namespace":"submariner-operator","uid":"f197aed4-5baf-4e3b-8a2b-8f2768f9a4cf","resourceVersion":"44208","generation":2,"creationTimestamp":"2022-02-24T14:46:31Z","labels":{"submariner-io/clusterID":"cluster2"},"managedFields":[{"manager":"submariner-gateway","operation":"Update","apiVersion":"submariner.io/v1","time":"2022-02-24T14:46:31Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:submariner-io/clusterID":{}}},"f:spec":{".":{},"f:backend":{},"f:backend_config":{".":{},"f:natt-discovery-port":{},"f:preferred-server":{},"f:udp-port":{}},"f:cable_name":{},"f:cluster_id":{},"f:healthCheckIP":{},"f:hostname":{},"f:nat_enabled":{},"f:private_ip":{},"f:public_ip":{},"f:subnets":{}}}}]},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-21-0-4","healthCheckIP":"242.254.2.254","hostname":"cluster2-worker","subnets":["242.254.2.0/24"],"private_ip":"172.21.0.4","public_ip":"159.8.66.226","nat_enabled":false,"backend":"libreswan","backend_config":{"natt-discovery-port":"4490","preferred-server":"false","udp-port":"4500"}}}
I0224 14:46:31.385803       1 gateway_monitor.go:147] Endpoint "cluster2", host: "cluster2-worker" belongs to a remote cluster
I0224 14:46:31.385845       1 gateway_monitor.go:443] Marking traffic destined to remote cluster: -d 242.254.2.0/24 -j MARK --set-mark 0xC0000/0xC0000
I0224 14:46:38.673931       1 service_export_controller.go:147] Processing ServiceExport "default/nginx-demo"
I0224 14:46:38.789425       1 service_export_controller.go:161] Kubeproxy chain for service "default/nginx-demo" does not exist yet
I0224 14:46:38.796645       1 service_export_controller.go:147] Processing ServiceExport "default/nginx-demo"
I0224 14:46:38.824387       1 service_export_controller.go:161] Kubeproxy chain for service "default/nginx-demo" does not exist yet
I0224 14:46:38.830199       1 service_export_controller.go:147] Processing ServiceExport "default/nginx-demo"
I0224 14:46:38.878230       1 service_export_controller.go:161] Kubeproxy chain for service "default/nginx-demo" does not exist yet
I0224 14:46:38.900643       1 service_export_controller.go:147] Processing ServiceExport "default/nginx-demo"
I0224 14:46:39.946343       1 service_export_controller.go:161] Kubeproxy chain for service "default/nginx-demo" does not exist yet
I0224 14:46:39.951799       1 service_export_controller.go:147] Processing ServiceExport "default/nginx-demo"
I0224 14:46:39.969350       1 service_export_controller.go:161] Kubeproxy chain for service "default/nginx-demo" does not exist yet
I0224 14:46:40.110844       1 service_export_controller.go:147] Processing ServiceExport "default/nginx-demo"
I0224 14:46:40.132677       1 service_export_controller.go:161] Kubeproxy chain for service "default/nginx-demo" does not exist yet
I0224 14:46:40.776841       1 service_export_controller.go:147] Processing ServiceExport "default/nginx-demo"
I0224 14:46:40.845399       1 service_export_controller.go:161] Kubeproxy chain for service "default/nginx-demo" does not exist yet
I0224 14:46:42.136459       1 service_export_controller.go:147] Processing ServiceExport "default/nginx-demo"
I0224 14:46:42.168739       1 service_export_controller.go:179] Creating &v1.GlobalIngressIP{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"nginx-demo", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"submariner.io/kubeproxy-iptablechain":"KUBE-SVC-2ZZBI6IPY23MEJHC"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.GlobalIngressIPSpec{Target:"ClusterIPService", ServiceRef:(*v1.LocalObjectReference)(0xc0006f48d0), PodRef:(*v1.LocalObjectReference)(nil)}, Status:v1.GlobalIngressIPStatus{Conditions:[]v1.Condition(nil), AllocatedIP:""}}
I0224 14:46:42.187838       1 global_ingressip_controller.go:127] Processing created &v1.GlobalIngressIP{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"nginx-demo", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9b03809b-678f-480e-be67-91edfd0b049d", ResourceVersion:"44281", Generation:1, CreationTimestamp:time.Date(2022, time.February, 24, 14, 46, 42, 0, time.Local), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"submariner-io/originatingNamespace":"default"}, Annotations:map[string]string{"submariner.io/kubeproxy-iptablechain":"KUBE-SVC-2ZZBI6IPY23MEJHC"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"submariner-globalnet", Operation:"Update", APIVersion:"submariner.io/v1", Time:time.Date(2022, time.February, 24, 14, 46, 42, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00044a5e8)}}}, Spec:v1.GlobalIngressIPSpec{Target:"ClusterIPService", ServiceRef:(*v1.LocalObjectReference)(0xc00048ea00), PodRef:(*v1.LocalObjectReference)(nil)}, Status:v1.GlobalIngressIPStatus{Conditions:[]v1.Condition(nil), AllocatedIP:""}}
I0224 14:46:42.187981       1 global_ingressip_controller.go:151] Allocating global IP for "default/nginx-demo"
I0224 14:46:42.188012       1 iface.go:113] Installing iptables rule for Service -d 242.254.1.253 -j KUBE-SVC-2ZZBI6IPY23MEJHC
I0224 14:46:43.212342       1 base_controllers.go:195] Updated: &v1.GlobalIngressIPStatus{Conditions:[]v1.Condition{v1.Condition{Type:"Allocated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.February, 24, 14, 46, 43, 212316959, time.Local), Reason:"Success", Message:"Allocated global IP"}}, AllocatedIP:"242.254.1.253"}
I0224 14:46:44.921156       1 service_controller.go:114] Service "default/nginx-demo" deleted
I0224 14:46:44.959317       1 service_export_controller.go:187] ServiceExport "default/nginx-demo" deleted
I0224 14:46:44.983492       1 global_ingressip_controller.go:127] Processing deleted &v1.GlobalIngressIP{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"nginx-demo", GenerateName:"", Namespace:"default", SelfLink:"", UID:"9b03809b-678f-480e-be67-91edfd0b049d", ResourceVersion:"44302", Generation:2, CreationTimestamp:time.Date(2022, time.February, 24, 14, 46, 42, 0, time.Local), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"submariner-io/originatingNamespace":"default"}, Annotations:map[string]string{"submariner.io/kubeproxy-iptablechain":"KUBE-SVC-2ZZBI6IPY23MEJHC"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"submariner-globalnet", Operation:"Update", APIVersion:"submariner.io/v1", Time:time.Date(2022, time.February, 24, 14, 46, 43, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc000589c08)}}}, Spec:v1.GlobalIngressIPSpec{Target:"ClusterIPService", ServiceRef:(*v1.LocalObjectReference)(0xc00061c080), PodRef:(*v1.LocalObjectReference)(nil)}, Status:v1.GlobalIngressIPStatus{Conditions:[]v1.Condition{v1.Condition{Type:"Allocated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.February, 24, 14, 46, 43, 0, time.Local), Reason:"Success", Message:"Allocated global IP"}}, AllocatedIP:"242.254.1.253"}}
I0224 14:46:44.983642       1 base_controllers.go:154] Releasing previously allocated IPs [242.254.1.253] for "default/nginx-demo"
I0224 14:46:44.983672       1 iface.go:129] Deleting iptable ingress rule for Service: -d 242.254.1.253 -j KUBE-SVC-2ZZBI6IPY23MEJHC
I0224 14:47:17.638772       1 service_export_controller.go:147] Processing ServiceExport "monti/monti-host"
I0224 14:47:17.684908       1 service_export_controller.go:179] Creating &v1.GlobalIngressIP{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"monti-host", GenerateName:"", Namespace:"monti", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"submariner.io/kubeproxy-iptablechain":"KUBE-SVC-OFTNFUVKIH4CPA3C"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.GlobalIngressIPSpec{Target:"ClusterIPService", ServiceRef:(*v1.LocalObjectReference)(0xc000070070), PodRef:(*v1.LocalObjectReference)(nil)}, Status:v1.GlobalIngressIPStatus{Conditions:[]v1.Condition(nil), AllocatedIP:""}}
I0224 14:47:17.704710       1 global_ingressip_controller.go:127] Processing created &v1.GlobalIngressIP{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"monti-host", GenerateName:"", Namespace:"monti", SelfLink:"", UID:"479eb599-92cd-436a-b44c-b7ef4dd08690", ResourceVersion:"44454", Generation:1, CreationTimestamp:time.Date(2022, time.February, 24, 14, 47, 17, 0, time.Local), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"submariner-io/originatingNamespace":"monti"}, Annotations:map[string]string{"submariner.io/kubeproxy-iptablechain":"KUBE-SVC-OFTNFUVKIH4CPA3C"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"submariner-globalnet", Operation:"Update", APIVersion:"submariner.io/v1", Time:time.Date(2022, time.February, 24, 14, 47, 17, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00037abb8)}}}, Spec:v1.GlobalIngressIPSpec{Target:"ClusterIPService", ServiceRef:(*v1.LocalObjectReference)(0xc000587f20), PodRef:(*v1.LocalObjectReference)(nil)}, Status:v1.GlobalIngressIPStatus{Conditions:[]v1.Condition(nil), AllocatedIP:""}}
I0224 14:47:17.705057       1 global_ingressip_controller.go:151] Allocating global IP for "monti/monti-host"
I0224 14:47:17.705087       1 iface.go:113] Installing iptables rule for Service -d 242.254.1.253 -j KUBE-SVC-OFTNFUVKIH4CPA3C
I0224 14:47:17.722564       1 base_controllers.go:195] Updated: &v1.GlobalIngressIPStatus{Conditions:[]v1.Condition{v1.Condition{Type:"Allocated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.February, 24, 14, 47, 17, 722519612, time.Local), Reason:"Success", Message:"Allocated global IP"}}, AllocatedIP:"242.254.1.253"}
I0224 14:47:53.744403       1 service_controller.go:114] Service "monti/monti-host" deleted
I0224 14:47:53.830620       1 global_ingressip_controller.go:127] Processing deleted &v1.GlobalIngressIP{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"monti-host", GenerateName:"", Namespace:"monti", SelfLink:"", UID:"479eb599-92cd-436a-b44c-b7ef4dd08690", ResourceVersion:"44565", Generation:2, CreationTimestamp:time.Date(2022, time.February, 24, 14, 47, 17, 0, time.Local), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"submariner-io/originatingNamespace":"monti"}, Annotations:map[string]string{"submariner.io/kubeproxy-iptablechain":"KUBE-SVC-OFTNFUVKIH4CPA3C"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"submariner-globalnet", Operation:"Update", APIVersion:"submariner.io/v1", Time:time.Date(2022, time.February, 24, 14, 47, 17, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00000d770)}}}, Spec:v1.GlobalIngressIPSpec{Target:"ClusterIPService", ServiceRef:(*v1.LocalObjectReference)(0xc0005f0bc0), PodRef:(*v1.LocalObjectReference)(nil)}, Status:v1.GlobalIngressIPStatus{Conditions:[]v1.Condition{v1.Condition{Type:"Allocated", Status:"True", ObservedGeneration:0, LastTransitionTime:time.Date(2022, time.February, 24, 14, 47, 17, 0, time.Local), Reason:"Success", Message:"Allocated global IP"}}, AllocatedIP:"242.254.1.253"}}
I0224 14:47:53.831109       1 base_controllers.go:154] Releasing previously allocated IPs [242.254.1.253] for "monti/monti-host"
I0224 14:47:53.831266       1 iface.go:129] Deleting iptable ingress rule for Service: -d 242.254.1.253 -j KUBE-SVC-OFTNFUVKIH4CPA3C

@dfarrell07
Copy link
Member

We don't have a listener that watches for Service, just ServiceExport. I think this is fundamentally the same as submariner-io/submariner#1734.

@dfarrell07 dfarrell07 added the duplicate This issue or pull request already exists label Jun 7, 2022
@dfarrell07
Copy link
Member

This may also be fixed as a part of submariner-io/enhancements#111.

@stale
Copy link

stale bot commented Oct 15, 2022

This issue has been automatically marked as stale because it has not had activity for 60 days. It will be closed if no further activity occurs. Please make a comment if this issue/pr is still valid. Thank you for your contributions.

@stale stale bot added the wontfix This will not be worked on label Oct 15, 2022
@dfarrell07 dfarrell07 removed the duplicate This issue or pull request already exists label Oct 18, 2022
@stale stale bot removed the wontfix This will not be worked on label Oct 18, 2022
@dfarrell07
Copy link
Member

dfarrell07 commented Dec 13, 2022 •
edited
Loading

This may be solved as a part of submariner-io/submariner#2130, or more specifically #214.

@tpantelis tpantelis self-assigned this Apr 28, 2023
@dfarrell07
Copy link
Member

This may be solved as a part of submariner-io/submariner#2130, or more specifically #214.

Not fixed by this, but triaged for the next release.

@dfarrell07 dfarrell07 moved this to Todo in Submariner 0.16 May 9, 2023
@tpantelis tpantelis mentioned this issue Jul 26, 2023
Closed
@tpantelis tpantelis moved this from Todo to In Progress in Submariner 0.16 Jul 29, 2023
@tpantelis tpantelis mentioned this issue Aug 2, 2023
Merged
@tpantelis tpantelis moved this from In Progress to In Review in Submariner 0.16 Aug 2, 2023
@tpantelis tpantelis mentioned this issue Aug 2, 2023
Merged
@github-project-automation github-project-automation bot moved this from In Review to Done in Submariner 0.16 Aug 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tpantelis tpantelis

Labels
bug Something isn't working priority:medium
Projects
No open projects
Submariner 0.16
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add E2E test for service re-creation tpantelis/lighthouse
4 participants
@dfarrell07 @vthapar @tpantelis @davidohana