Using services for proxying traffic in the absense of Node.ExternalIP

[edwarnicke]

Currently, we use cmd-map-ip-k8s to generate a mapping of Node.InternalIPs to Node.ExternalIPs so that the nsmgr-proxy can translate between those IPs as traffic leaves/enters the cluster.

This works great for clusters setup with per-Node ExternalIPs. Sadly, this is not all Nodes.

For the case when Nodes do not have a Node.ExternalIPs, we do have the option of using Services.ExternalIP for the same purpose.

The lack of Node.ExternalIPs can be compensated for by creating a Service of type LoadBalancer for each Node and using Services without selectors to direct that Service to the Node.InternalIP.

This can be done in one of two ways:

1. PreCreated - cmd-map-ip-k8s could create a Service for each Node in the cluster, and simply write out the mapping ConfigMap in the same way it does now
2. OnDemand - cmd-map-ip-k8s could expose a simple service consumed by cmd-nsmgr-proxy that, on demand, would create a Service for each used IP (presumably a Node).

There are tradeoffs between these. PreCreated results in the lowest in-band latency. The downside of this is that it precreates a LoadBalancer Service per Node, which may incur costs for large clusters.

PreCreated is probably the easiest to start with, so we should probably start from there.

[denis-tingaikin]

@edwarnicke These solutions have a problem with multi nodes clusters. There are our vip could be mapped to a few internal nodes IP. So I see only one solution for multi-cluster it's creating a service per Node. Thoughts?

[edwarnicke]

I think we should go with a service per node.