External network connectivity for pod via the pod VM network

[bpradipt]

Currently, all pod traffic is routed via the worker node (WN). This is fine as all existing network policies that were applicable to pods running on the WN also apply to peer pods.
However, the peer pods approach has enabled some additional use cases that were earlier impossible or impractical. Concurrently, these new use cases have resulted in new requirements.

Consider a scenario where you are running a Kind K8s cluster on your laptop, experimenting with an AI model, and needing additional computing power or GPUs. With peer pods, you can spin up a pod using the appropriate instance type from the Kind K8s cluster running on your laptop.

From a networking standpoint, this means that there should be bidirectional connectivity between the cloud and your development setup.

One solution is to use a VPN, which complicates the setup. A relatively easier alternative is to use a public IP associated with the cloud instance for control-plane traffic and use the cloud instance network for external connectivity requirements (like accessing cloud object storage for data) instead of routing all traffic via your development setup.

A possible implementation of this idea is available in the draft PR - #2273
The basic premise is to use two NICs in the pod VM - primary NIC for all control-plane and data-plane (pod network) traffic. The secondary NIC is for external network connectivity for the pod, ie any traffic that is not sent to the pod network.

Would love to hear everyone's thoughts around this..

Thanks to @yoheiueda for helping me debug networking issues during my experiments with multi-NIC in pod VM.

Explicitly tagging @skaegi as he seems to have similar requirements.

[yoheiueda]

I think this is a good idea for resolving the network performance issue related to external connectivity.

One key consideration point for implementing this idea is how to distinguish between internal and external networks. If we use the pod VM network for all traffic, the solution is straightforward: we simply omit the tunnel between WN and peer pod VM and rely solely on the pod VM network. However, if we want to use the pod VM network selectively based on destination addresses, we need a way to determine whether a destination address is local or external.

I’m googling to find a reliable way to determine a Kubernetes cluster’s internal network address ranges. It appears that Kubernetes doesn’t provide a built-in mechanism to retrieve the Cluster CIDR and Service CIDR. I’ve found some related discussions on Stack Overflow, but they don’t offer a clear solution.

https://stackoverflow.com/questions/44190607/how-do-you-find-the-cluster-service-cidr-of-a-kubernetes-cluster
https://stackoverflow.com/questions/63520946/how-can-i-get-the-clusters-cidr-in-a-pod

I think we’ll need to add a command-line option to CAA that allows users to specify the address ranges that should be considered part of the local network.

Another consideration point is asymmetric routing for inbound traffics to the pod VM via ClusterIP and NodePort. Such inbound traffics come from the VXLAN tunnel. When Source IP is preserved in such inbound traffics, replies from the peer pod VM will go through the peer pod VM network instead of the VXLAN tunnel. If Source IP is not preserved and NAT'ed by Kubernetes, asymmetric routing will not happen.

https://kubernetes.io/docs/tutorials/services/source-ip/

Whether Source IP is preserved or not depends on Address type and configuration. I think we need to document this limitation.

[bpradipt]

I think we’ll need to add a command-line option to CAA that allows users to specify the address ranges that should be considered part of the local network.

Alternatively, we can add an option to specific external network ranges that needs to be routed via VM network and let the default route be the vxlan interface.

[bpradipt]

I think we’ll need to add a command-line option to CAA that allows users to specify the address ranges that should be considered part of the local network.

Alternatively, we can add an option to specific external network ranges that needs to be routed via VM network and let the default route be the vxlan interface.

This becomes impractical. As mentioned earlier by @yoheiueda having an option to specify the local podCIDR and service CIDR