Switch nginx ingress to use NLB

[bmonkman]

The current setup uses ELB for the nginx ingress controller, but there would be some benefits to switching over to NLB:

• With the current setup we don't get remote IPs in nginx logs
• We are already using L4 only on the ELB so why not, when NLB is better and more efficient at handling it.

The change should be straightforward, it will just require some testing.

These annotations need to be added to the ingress service:

    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '60'
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'

The nginx default keepalive timeout is 75s but let's also specify that explicitly in the code, maybe with a comment telling people that the lb timeout must be shorter than the nginx keepalive.

After this change, make sure that the nginx logs show the proper remote IP address, not an internal 10. address.
Make sure TLS still works properly.

[sshi100]

Nice to go this way.

Thinking: currently we have wireguard VPN service using a NLB, wondering that if we can merge it into the new NLB for less cost, although it could be tricky.

[bmonkman]

Unfortunately I'm not sure we'll be able to since the LB is created dynamically by the ingress controller. You could see if there's a configuration option to specify an existing LB though.

[sshi100]

Just a quick investigation: no configuration specific for such "reuse" yet, but one issue requested Create option to reuse an existing ALB instead of creating a new ALB per Ingress. Should be a long waiting, a guy created this interesting Ingress-Merge for solution.