Fix Cilium ENI ipam

[olemarkus]

Fixes #14575

[olemarkus]

/test pull-kops-e2e-cni-cilium-eni

[olemarkus]

/cc @johngmyers @rifelpet
what do you think of this config?

[johngmyers]

Isn't a point of ENI to disable masquerading so that pods are directly addressable from outside the cluster? Or is this what is needed in order to get services to work?

[olemarkus]

The Pods are directly addressable from the VPC network. The masqing source NATs the Pod IPs behind the host IP (which is also a VPC address). I don't understand why the masqing is required, but with masqing disabled, I see packets going out the correct interface, but I don't see a single return packet. It's beyond me what drops the packets and why. The VPC flow logs suggests some NATing takes place even with masq disabled, but I couldn't make sense of it. I talked to Cilium as well and they didn't know what was happening here.

The default setup for EKS uses masq as well. The helm config looks like this:

  tunnel: disabled
  enable-endpoint-routes: "true"
  auto-create-cilium-node-resource: "true"
  ec2-api-endpoint: ""
  eni-tags: "{}"
  subnet-ids-filter: ""
  subnet-tags-filter: ""
  # Enables L7 proxy for L7 policy enforcement and visibility
  enable-l7-proxy: "true"

  enable-ipv4-masquerade: "true"
  enable-ipv6-masquerade: "true"
  egress-masquerade-interfaces: eth0

  enable-xt-socket-fallback: "true"
  install-iptables-rules: "true"
  install-no-conntrack-iptables-rules: "false"

[johngmyers]

Maybe the packets are going out the wrong ENI and are getting caught by the src/dst checks? I've had the reverse problem with AWS VPC CNI.

[olemarkus]

Nope. ens5 has the node IP and the EBS controllers have IPs also associated with ens5. So this is not the case here.

I am wondering if we should just merge this and then see if we can figure out a way of disabling masq later on. I'll also see if I can catch up to cilium 1.13 soon.

[johngmyers]

This is sub-optimal in that it makes it more difficult to trace network traffic back to the workload that generated it. Could you add to the documentation this quirk?

[olemarkus]

Yeah I'll update the docs on this.

Not sure how well IPs can be relied on as identity anyway though. The IPs change hands across Pods on the same host as they come and go.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: johngmyers

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [johngmyers]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[johngmyers]

IP plus timestamp can be useful for finding things in audit logs. With NAT, not so much.