Fix iptables/nftables issue #3

sridhargaddam · 2019-08-27T11:07:04Z

Both iptables and nftables use netfilter framework in the kernel for
packet filtering. Many distributions are moving in the direction of
using nftables over iptables. Although, nftables uses a new command
line utility (named nft), starting from iptables >=1.8, it uses
nftables under the hood while continuing to support the same iptables
syntax from the user.

Quoting from Dan's comment [#]

"In iptables 1.8, the maintainers have "deprecated" the classic ip_tables:
the iptables tool now does userspace translation from the legacy UI/UX,
and uses nf_tables under the hood. So, the commands look and feel the
same, but they're now programming a different kernel subsystem.

The problem arises when you mix and match invocations of iptables 1.6
(the previous stable) and 1.8 on the same machine, because although they
look identical, they're programming different kernel subsystems.

Empirically, this causes weird and wonderful things to happen - things
like if you trace a packet coming from a pod, you see it flowing through
both ip_tables and nf_tables, but even if both accept the packet, it then
vanishes entirely and never gets forwarded"

So, as long as we are programming either nf_tables or iptables, we would
not have any issues. Currently, there is no easy way to identify what type
of rules are programmed on the host. This patch follows the same approach
(as described here [*]) that is taken in OpenShift where the host file
system is mounted inside the docker container and iptables utility on the
host is exec'ed for programming any firewall rules.

[#] kubernetes/kubernetes#71305 (comment)
[*] kubernetes/kubernetes#71305 (comment)

Both iptables and nftables use netfilter framework in the kernel for packet filtering. Many distributions are moving in the direction of using nftables over iptables. Although, nftables uses a new command line utility (named nft), starting from iptables >=1.8, it uses nftables under the hood while continuing to support the same iptables syntax from the user. Quoting from Dan's comment [#] "In iptables 1.8, the maintainers have "deprecated" the classic ip_tables: the iptables tool now does userspace translation from the legacy UI/UX, and uses nf_tables under the hood. So, the commands look and feel the same, but they're now programming a different kernel subsystem. The problem arises when you mix and match invocations of iptables 1.6 (the previous stable) and 1.8 on the same machine, because although they look identical, they're programming different kernel subsystems. Empirically, this causes weird and wonderful things to happen - things like if you trace a packet coming from a pod, you see it flowing through both ip_tables and nf_tables, but even if both accept the packet, it then vanishes entirely and never gets forwarded" So, as long as we are programming either nf_tables or iptables, we would not have any issues. Currently, there is no easy way to identify what type of rules are programmed on the host. This patch follows the same approach (as described here [*]) that is taken in OpenShift where the host file system is mounted inside the docker container and iptables utility on the host is exec'ed for programming any firewall rules. [#] kubernetes/kubernetes#71305 (comment) [*] kubernetes/kubernetes#71305 (comment)

sridhargaddam · 2019-08-27T11:09:31Z

JFYI, I verified this change both on ubi-minimal as well as Ubuntu images.

Depends-On: submariner-io/submariner-charts#3 Both iptables and nftables use netfilter framework in the kernel for packet filtering. Many distributions are moving in the direction of using nftables over iptables. Although, nftables uses a new command line utility (named nft), starting from iptables >=1.8, it uses nftables under the hood while continuing to support the same iptables syntax from the user. Quoting from Dan's comment [#] "In iptables 1.8, the maintainers have "deprecated" the classic ip_tables: the iptables tool now does userspace translation from the legacy UI/UX, and uses nf_tables under the hood. So, the commands look and feel the same, but they're now programming a different kernel subsystem. The problem arises when you mix and match invocations of iptables 1.6 (the previous stable) and 1.8 on the same machine, because although they look identical, they're programming different kernel subsystems. Empirically, this causes weird and wonderful things to happen - things like if you trace a packet coming from a pod, you see it flowing through both ip_tables and nf_tables, but even if both accept the packet, it then vanishes entirely and never gets forwarded" So, as long as we are programming either nf_tables or iptables, we would not have any issues. Currently, there is no easy way to identify what type of rules are programmed on the host. This patch follows the same approach (as described here [*]) that is taken in OpenShift where the host file system is mounted inside the docker container and iptables utility on the host is exec'ed for programming any firewall rules. [#] kubernetes/kubernetes#71305 (comment) [*] kubernetes/kubernetes#71305 (comment)

…ffic As part of supporting Network policies and for ease of debugging, this patch implements the following. 1. Creates VxLAN tunnels in the local Cluster between the worker nodes and the Cluster Gateway Node. 2. Programms the necessary iptable rules on the Cluster nodes to allow inter-cluster traffic. 3. This patch also avoids SNAT/MASQ for inter-cluster traffic, thereby preserving the original source ip of the POD all the way until the destination POD. 4. Programs the routing rules on the workerNodes to forward the remoteCluster traffic over the VxLAN interface that is created between the worker node and Cluster GatewayNode. This patch depends on the following other patches Depends-On: submariner-io#135 Depends-On: submariner-io/submariner-charts#3 Depends-On: submariner-io/submariner-charts#4

mangelajo · 2019-09-03T14:45:41Z

Looks fine, but this has conflicts to resolve

mangelajo · 2019-09-03T14:47:02Z

Ok, fixed.

Oats87 · 2019-09-03T15:27:21Z

@sridhargaddam are there any SCC concerns we have due to this change? I'm pretty sure OCP's privileged allows for mounting hostPaths right?

sridhargaddam · 2019-09-03T15:44:47Z

@sridhargaddam are there any SCC concerns we have due to this change? I'm pretty sure OCP's privileged allows for mounting hostPaths right?

AFAICT, yes.

As part of supporting Network policies and for ease of debugging, this patch implements the following. 1. Creates VxLAN tunnels in the local Cluster between the worker nodes and the Cluster Gateway Node. 2. Programms the necessary iptable rules on the Cluster nodes to allow inter-cluster traffic. 3. This patch also avoids SNAT/MASQ for inter-cluster traffic, thereby preserving the original source ip of the POD all the way until the destination POD. 4. Programs the routing rules on the workerNodes to forward the remoteCluster traffic over the VxLAN interface that is created between the worker node and Cluster GatewayNode. This patch depends on the following other patches Depends-On: submariner-io#135 Depends-On: submariner-io/submariner-charts#3 Depends-On: submariner-io/submariner-charts#4

sridhargaddam · 2019-09-04T12:29:25Z

Please note that to have a functional solution, we also need the patch from submariner-io
submariner-io/submariner#135

Both the patches should go together and we also have to host the updated charts in a publicly accessible repo. Currently the charts are hosted at rancher url https://releases.rancher.com/submariner-charts/latest

In KIND based setup, its referenced here https://github.com/submariner-io/submariner/blob/master/scripts/kind-e2e/e2e.sh#L59

CC: @mangelajo , @Oats87 , @skitt, @tpantelis, @dimaunx

…ffic As part of supporting Network policies and for ease of debugging, this patch implements the following. 1. Creates VxLAN tunnels in the local Cluster between the worker nodes and the Cluster Gateway Node. 2. Programms the necessary iptable rules on the Cluster nodes to allow inter-cluster traffic. 3. This patch also avoids SNAT/MASQ for inter-cluster traffic, thereby preserving the original source ip of the POD all the way until the destination POD. 4. Programs the routing rules on the workerNodes to forward the remoteCluster traffic over the VxLAN interface that is created between the worker node and Cluster GatewayNode. This patch depends on the following other patches Depends-On: submariner-io#135 Depends-On: submariner-io/submariner-charts#3 Depends-On: submariner-io/submariner-charts#4

As part of supporting Network policies and for ease of debugging, this patch implements the following. 1. Creates VxLAN tunnels in the local Cluster between the worker nodes and the Cluster Gateway Node. 2. Programms the necessary iptable rules on the Cluster nodes to allow inter-cluster traffic. 3. This patch also avoids SNAT/MASQ for inter-cluster traffic, thereby preserving the original source ip of the POD all the way until the destination POD. 4. Programs the routing rules on the workerNodes to forward the remoteCluster traffic over the VxLAN interface that is created between the worker node and Cluster GatewayNode. This patch depends on the following other patches Depends-On: submariner-io#135 Depends-On: submariner-io/submariner-charts#3 Depends-On: submariner-io/submariner-charts#4

…ffic As part of supporting Network policies and for ease of debugging, this patch implements the following. 1. Creates VxLAN tunnels in the local Cluster between the worker nodes and the Cluster Gateway Node. 2. Programms the necessary iptable rules on the Cluster nodes to allow inter-cluster traffic. 3. This patch also avoids SNAT/MASQ for inter-cluster traffic, thereby preserving the original source ip of the POD all the way until the destination POD. 4. Programs the routing rules on the workerNodes to forward the remoteCluster traffic over the VxLAN interface that is created between the worker node and Cluster GatewayNode. This patch depends on the following other patches Depends-On: submariner-io#135 Depends-On: submariner-io/submariner-charts#3 Depends-On: submariner-io/submariner-charts#4

As part of supporting Network policies and for ease of debugging, this patch implements the following. 1. Creates VxLAN tunnels in the local Cluster between the worker nodes and the Cluster Gateway Node. 2. Programms the necessary iptable rules on the Cluster nodes to allow inter-cluster traffic. 3. This patch also avoids SNAT/MASQ for inter-cluster traffic, thereby preserving the original source ip of the POD all the way until the destination POD. 4. Programs the routing rules on the workerNodes to forward the remoteCluster traffic over the VxLAN interface that is created between the worker node and Cluster GatewayNode. This patch depends on the following other patches Depends-On: submariner-io#135 Depends-On: submariner-io/submariner-charts#3 Depends-On: submariner-io/submariner-charts#4

Depends-On: submariner-io/submariner-charts#3 Both iptables and nftables use netfilter framework in the kernel for packet filtering. Many distributions are moving in the direction of using nftables over iptables. Although, nftables uses a new command line utility (named nft), starting from iptables >=1.8, it uses nftables under the hood while continuing to support the same iptables syntax from the user. Quoting from Dan's comment [#] "In iptables 1.8, the maintainers have "deprecated" the classic ip_tables: the iptables tool now does userspace translation from the legacy UI/UX, and uses nf_tables under the hood. So, the commands look and feel the same, but they're now programming a different kernel subsystem. The problem arises when you mix and match invocations of iptables 1.6 (the previous stable) and 1.8 on the same machine, because although they look identical, they're programming different kernel subsystems. Empirically, this causes weird and wonderful things to happen - things like if you trace a packet coming from a pod, you see it flowing through both ip_tables and nf_tables, but even if both accept the packet, it then vanishes entirely and never gets forwarded" So, as long as we are programming either nf_tables or iptables, we would not have any issues. Currently, there is no easy way to identify what type of rules are programmed on the host. This patch follows the same approach (as described here [*]) that is taken in OpenShift where the host file system is mounted inside the docker container and iptables utility on the host is exec'ed for programming any firewall rules. [#] kubernetes/kubernetes#71305 (comment) [*] kubernetes/kubernetes#71305 (comment)

…ffic As part of supporting Network policies and for ease of debugging, this patch implements the following. 1. Creates VxLAN tunnels in the local Cluster between the worker nodes and the Cluster Gateway Node. 2. Programms the necessary iptable rules on the Cluster nodes to allow inter-cluster traffic. 3. This patch also avoids SNAT/MASQ for inter-cluster traffic, thereby preserving the original source ip of the POD all the way until the destination POD. 4. Programs the routing rules on the workerNodes to forward the remoteCluster traffic over the VxLAN interface that is created between the worker node and Cluster GatewayNode. This patch depends on the following other patches Depends-On: submariner-io/submariner#135 Depends-On: submariner-io/submariner-charts#3 Depends-On: submariner-io/submariner-charts#4

As part of supporting Network policies and for ease of debugging, this patch implements the following. 1. Creates VxLAN tunnels in the local Cluster between the worker nodes and the Cluster Gateway Node. 2. Programms the necessary iptable rules on the Cluster nodes to allow inter-cluster traffic. 3. This patch also avoids SNAT/MASQ for inter-cluster traffic, thereby preserving the original source ip of the POD all the way until the destination POD. 4. Programs the routing rules on the workerNodes to forward the remoteCluster traffic over the VxLAN interface that is created between the worker node and Cluster GatewayNode. This patch depends on the following other patches Depends-On: submariner-io/submariner#135 Depends-On: submariner-io/submariner-charts#3 Depends-On: submariner-io/submariner-charts#4

sridhargaddam mentioned this pull request Aug 27, 2019

Exec iptables from the host filesystem submariner-io/submariner#135

Merged

skitt approved these changes Aug 27, 2019

View reviewed changes

sridhargaddam mentioned this pull request Sep 2, 2019

Use VxLAN overlay tunnels for inter-cluster traffic submariner-io/submariner#140

Merged

Merge branch 'master' into use-iptables-from-host

a77d2ef

mangelajo approved these changes Sep 3, 2019

View reviewed changes

mangelajo approved these changes Sep 5, 2019

View reviewed changes

mangelajo merged commit 4629754 into submariner-io:master Sep 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix iptables/nftables issue #3

Fix iptables/nftables issue #3

sridhargaddam commented Aug 27, 2019

sridhargaddam commented Aug 27, 2019

mangelajo commented Sep 3, 2019

mangelajo commented Sep 3, 2019

Oats87 commented Sep 3, 2019

sridhargaddam commented Sep 3, 2019 •

edited

Loading

sridhargaddam commented Sep 4, 2019

Fix iptables/nftables issue #3

Fix iptables/nftables issue #3

Conversation

sridhargaddam commented Aug 27, 2019

sridhargaddam commented Aug 27, 2019

mangelajo commented Sep 3, 2019

mangelajo commented Sep 3, 2019

Oats87 commented Sep 3, 2019

sridhargaddam commented Sep 3, 2019 • edited Loading

sridhargaddam commented Sep 4, 2019

sridhargaddam commented Sep 3, 2019 •

edited

Loading