For dual stack deployments, node addresses show only one IP if node-ip parameter is passed

[chandanD4]

What happened?

Case1:
With k8s 1.23.x and cloud-provider-openstack 1.23.x if we pass "--cloud-provider=external" along with "--node-ip=10.0.16.2" as KUBELET_EXTRA_ARGS to the kubelet.service file we observe that the Node.status.addresses was as follows
status
addresses:

address: 10.0.16.2
type: InternalIP
address: dead::5
type: InternalIP

Case:2
But with the current k8s 1.24.x and cloud-provider-openstack 1.24.x, with the same config we see only the node IP that was passed
status
addresses:

address: 10.0.16.5
type: InternalIP

Case3:
We tried with k8s 1.24.x and cloud-provider-openstack 1.23.x and the behavior is same as it was in the earlier release.
status
addresses:

address: 10.0.16.4
type: InternalIP
address: dead::2
type: InternalIP

What did you expect to happen?

The latest 1.24.x was expected to show the same behavior as in Case1 and 3.

How can we reproduce it (as minimally and precisely as possible)?

Case2 above.

Anything else we need to know?

No response

Kubernetes version

K8s version : 1.24.2

Cloud provider

cloud-provider-openstack

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[k8s-ci-robot]

@chandanD4: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[chandanD4]

/sig network
/sig node

[aojea]

are you not pointing out that the difference is with cloud-provider-openstack 1.24.x ?

[chandanD4]

are you not pointing out that the difference is with cloud-provider-openstack 1.24.x ?

@aojea : We are seeing this anomaly with cloud-provider-openstack 1.24.x version.

[aojea]

This seems related

#107750
kubernetes/cloud-provider#56

What I don't know if this is a problem on Kubernetes or https://github.com/kubernetes/cloud-provider-openstack

[chandanD4]

@mdbooth : It seems that as part of #107750 , this causing the reported failure.

Below code is skipping IPv6 or IPv4 address depending on the --node-ip argument. As we can have IPv6 and IPv4 address for the same address Type i.e. InternalIP and ExternalIP , but in the below code snippet we are filtering based on address Type and not considering the address Family , which causing node address to have only IP family address in dual stack environment. Please have look at this and let us know if we are missing something here.

https://github.com/kubernetes/cloud-provider/blob/bf769589bdb472d92d86b29eb4016a312d4b5ca8/node/helpers/address.go#L106

[mdbooth]

@chandanD4 out of curiosity, why are you using --node-ip in this case? The behaviour in 1.23 was that --node-ip was ignored when using external CCM. Can you remove it?

The behaviour you see now is consistent with the behaviour of --node-ip with the legacy cloud provider. There is one limitation, though: with the legacy cloud provider it was possible to pass 2 values in --node-ip, whereas with the Node annotation we only pass one. This means it is not possible to specify both an IPv4 and IPv6 --node-ip with external CCM.

However, if you don't need this parameter anyway (which I'm guessing you don't because it was working in 1.23), then presumably this isn't a limitation for you?

[chandanD4]

@mdbooth The notion behind using --node-ip param was indicating the default stack family in case of dual stack cluster. Kubelet/node as the node IP. As it picks by default as IPv4 if not specified ,to indicate IPv6 , we used this param.

One more observation, for a dual-stack default IPv6 deployment (k8s 1.24.2 and cloud-provider 1.24.1), if I pass "--node-ip=::" with "--cloud-provider=external"

But for PODs kube-apiserver-master-, kube-controller-manager- and kube-scheduler- if we describe the pods, under Status.IPs we see only one IP, the IPv6 address of the node. In 1.23.3 cloud-provider Status.IPs had both the IPs present.

[chandanD4]

The behaviour you see now is consistent with the behaviour of --node-ip with the legacy cloud provider. There is one limitation, though: with the legacy cloud provider it was possible to pass 2 values in --node-ip, whereas with the Node annotation we only pass one. This means it is not possible to specify both an IPv4 and IPv6 --node-ip with external CCM.

How do we indicate both families IP i.e. IPv4 and IPv6 for node ?

[mdbooth]

I don't think this was ever doing what you thought it was doing. To the best of my understanding (but there are quite a few interactions here and I'd be extremely interested to know if I missed something!), --node-ip was previously doing nothing for you. To retain this behaviour the simplest thing would be to remove it.

For context, we made these changes to support migration from the legacy cloud provider where --node-ip does have an effect. It was causing a regression moving from legacy cloud provider to external cloud provider to have the behaviour of --node-ip change. I confess I didn't give --node-ip with CCM prior to 1.24 much thought as, other than raise an error if it was invalid, it didn't actually use it at all.

Separately, I wonder if we need to add the ability to support a second node IP via the Node annotation to support dual stack.

Note that this all continues to feel like a stop-gap. I think the intention here was that CCMs would be the canonical source for network configuration information. Certainly OpenStack CCM has some config options for this. There are 3 problems with that, though:

• They're not really ready
• They're not actually canonical (the installer is)
• You can't write an automated upgrade if the metadata describing which networks are internal is lost because it was exclusively represented in --node-ip.

The latter problem was the fundamental issue we had in OpenShift. The second is the one is the real architectural problem, although the solution to it will inevitably be custom to each CCM.

[mdbooth]

The behaviour you see now is consistent with the behaviour of --node-ip with the legacy cloud provider. There is one limitation, though: with the legacy cloud provider it was possible to pass 2 values in --node-ip, whereas with the Node annotation we only pass one. This means it is not possible to specify both an IPv4 and IPv6 --node-ip with external CCM.

How do we indicate both families IP i.e. IPv4 and IPv6 for node ?

You were never able to do that. If it worked it was by happenstance. If the previous behaviour was working, you can get back to it by removing --node-ip.

If we wanted to implement this with --node-ip we would need to add support for a second address. However, ultimately I think we would want to add support to OpenStack CCM to order by address family. Longer term I think we want to move away from using --node-ip with CCMs.

[chandanD4]

One more observation, for a dual-stack default IPv6 deployment (k8s 1.24.2 and cloud-provider 1.24.1), if I pass "--node-ip=::" with "--cloud-provider=external"

But for PODs kube-apiserver-master-, kube-controller-manager- and kube-scheduler- if we describe the pods, under Status.IPs we see only one IP, the IPv6 address of the node. In 1.23.3 cloud-provider Status.IPs had both the IPs present.

Any insight on this ? Why these pod have only one family IP address i.e. address family specified using the --node-ip param ?

Based on your explanation , if we skip --node-ip param , Node.status.addresses will list both IPv4 and IPv6 addresses. However --node-ip used for
Make podIPs order match node IP family preference

[mdbooth]

That is indeed overloading --node-ip 😬

[mdbooth]

One more observation, for a dual-stack default IPv6 deployment (k8s 1.24.2 and cloud-provider 1.24.1), if I pass "--node-ip=::" with "--cloud-provider=external"

Did you forget to finish this sentence, btw? What was the behaviour you observed? It looks like this might do what you want: :: for IPv6, 0.0.0.0 for IPv4.

[mdbooth]

That is indeed overloading --node-ip 😬

We should probably revisit this too. Perhaps an alternative configuration option which applies only to pod IPs, not node IPs? Or perhaps this, too, should be configured separately by the source of the sandbox IPs instead of relying on kubelet to sort it?

[chandanD4]

But for PODs kube-apiserver-master-, kube-controller-manager- and kube-scheduler- if we describe the pods, under Status.IPs we see only one IP, the IPv6 address of the node. In 1.23.3 cloud-provider Status.IPs had both the IPs present.

One more observation, for a dual-stack default IPv6 deployment (k8s 1.24.2 and cloud-provider 1.24.1), if I pass "--node-ip=::" with "--cloud-provider=external"

Did you forget to finish this sentence, btw? What was the behaviour you observed? It looks like this might do what you want: :: for IPv6, 0.0.0.0 for IPv4.

But for PODs kube-apiserver-master-, kube-controller-manager- and kube-scheduler- if we describe the pods, under Status.IPs we see only one IP, the IPv6 address of the node. In 1.23.3 cloud-provider Status.IPs had both the IPs present.

[mdbooth]

But for PODs kube-apiserver-master-, kube-controller-manager- and kube-scheduler- if we describe the pods, under Status.IPs we see only one IP, the IPv6 address of the node. In 1.23.3 cloud-provider Status.IPs had both the IPs present.

One more observation, for a dual-stack default IPv6 deployment (k8s 1.24.2 and cloud-provider 1.24.1), if I pass "--node-ip=::" with "--cloud-provider=external"

Did you forget to finish this sentence, btw? What was the behaviour you observed? It looks like this might do what you want: :: for IPv6, 0.0.0.0 for IPv4.

But for PODs kube-apiserver-master-, kube-controller-manager- and kube-scheduler- if we describe the pods, under Status.IPs we see only one IP, the IPv6 address of the node. In 1.23.3 cloud-provider Status.IPs had both the IPs present.

@aojea I think the bug here is that --node-ip is overloaded for pod IPs. Thoughts?

[mdbooth]

But for PODs kube-apiserver-master-, kube-controller-manager- and kube-scheduler- if we describe the pods, under Status.IPs we see only one IP, the IPv6 address of the node. In 1.23.3 cloud-provider Status.IPs had both the IPs present.

One more observation, for a dual-stack default IPv6 deployment (k8s 1.24.2 and cloud-provider 1.24.1), if I pass "--node-ip=::" with "--cloud-provider=external"

Did you forget to finish this sentence, btw? What was the behaviour you observed? It looks like this might do what you want: :: for IPv6, 0.0.0.0 for IPv4.

But for PODs kube-apiserver-master-, kube-controller-manager- and kube-scheduler- if we describe the pods, under Status.IPs we see only one IP, the IPv6 address of the node. In 1.23.3 cloud-provider Status.IPs had both the IPs present.

@chandanD4 Please can you confirm what --node-ip was set to in these 2 cases? Is it set to ::, or was it set to an actual IP as in the original description?

[chandanD4]

Please can you confirm what --node-ip was set to in these 2 cases? Is it set to ::, or was it set to an actual IP as in the original description?

For k8s 1.24.2 and cloud provider openstack version : 1.24.1 , --node-ip was set to "::".
For k8s 1.24.2 and cloud provider openstack version : 1.23.3 , --node-ip was set to specific IPv6 address

[mdbooth]

Based on code inspection, my expection for these 2 cases is:

CPO 1.23.3 with specific IPv6 address:

• CPO is only validating --node-ip
• Node addresses are unfiltered
• sortPodIPs returns the first valid IPv6 pod IP followed by the first valid IPv4 pod IP.

CPO 1.24.1 with generic IPv6 address:

• CPO sorts node addresses to return all IPv6 addresses before all IPv4 addresses
• sortPodIPs returns the first valid IPv6 pod IP followed by the first valid IPv4 pod IP.

These 2 cases both seem like they should work to me.

I'd expect apiserver to use host networking so that could be affected by a change in host IPs. Are KCM and scheduler also using host networking?

However, I can't explain why kubelet would return different status IPs in this case, unless these are all host networking pods and those are handled by some other function I didn't find?

[mdbooth]

Please can you confirm what --node-ip was set to in these 2 cases? Is it set to ::, or was it set to an actual IP as in the original description?

For k8s 1.24.2 and cloud provider openstack version : 1.24.1 , --node-ip was set to "::". For k8s 1.24.2 and cloud provider openstack version : 1.23.3 , --node-ip was set to specific IPv6 address

Are you able to confirm what Node.Status.Addresses contains on a host running KCM for both cases, along with status.IPs for KCM for both cases?

[chandanD4]

I'd expect apiserver to use host networking so that could be affected by a change in host IPs. Are KCM and scheduler also using host networking?

Running in host networking : True.

[danwinship]

I think the bug here is that --node-ip is overloaded for pod IPs. Thoughts?

That behavior was there first. It has seniority.

The bug here is that we didn't consider that someone might take advantage of the fact that --node-ip was broken with external cloud providers and use it only for its side effects...

[mdbooth]

@danwinship Do we want to:

• Update kubelet to remove the annotation if --node-ip is not set
• Create a new annotation which allows both multiple node IPs and generic node IPs (and doesn't have Alpha in the name)

[chandanD4]

Please can you confirm what --node-ip was set to in these 2 cases? Is it set to ::, or was it set to an actual IP as in the original description?

For k8s 1.24.2 and cloud provider openstack version : 1.24.1 , --node-ip was set to "::". For k8s 1.24.2 and cloud provider openstack version : 1.23.3 , --node-ip was set to specific IPv6 address

Are you able to confirm what Node.Status.Addresses contains on a host running KCM for both cases, along with status.IPs for KCM for both cases?

Node.Status.Address has both IPv4 and IPv6 IP addresses.

[manuelbuil]

I'm experiencing the same problem. We have a simple cloud-provider, i.e. --cloud-provider=external in kubelet, through which we update the node object with the InternalAddress and ExternalAddress addresses of the node. As long as we don't pass the --node-ip flag to kubelet things work fine.

However, since 1.24, if we pass --node-ip to kubelet, e.g. because we use a secondary NIC for cluster communication, we can't do dual-stack anymore. The cloud-provider finds the correct InternalAddress addresses (ipv4 & ipv6) but they are ignored as kubelet's --node-ip has a higher priority. Note that it is impossible to pass a dual-stack parameter to --node-ip as you get an error: kubelet exited: failed to run Kubelet: dual-stack --node-ip \"....\" not supported when using a cloud provider".

Therefore, we are wondering how we could update the node object with the correct InternalAddress addresses ( both ipv4 & ipv6) when using dual-stack and an external cloud-provider in 1.24.

[danwinship]

@mdbooth we should definitely fix kubelet to remove any existing annotation in the cases where it is not setting the annotation, and backport that to at least 1.24. We probably also need some slightly better documentation to go along with that.

We should also eventually add the new-and-improved annotation (and make kubelet accept a dual-stack --node-ip value with --cloud-provider=external) but that would be a new feature and wouldn't be back-port-able.

[danwinship]

@manuelbuil

However, since 1.24, if we pass --node-ip to kubelet, e.g. because we use a secondary NIC for cluster communication

So note that this didn't work in 1.23 (passing --node-ip would not change what NIC was used for communication), so what you're describing isn't a new bug in 1.24, it's just the lack of a new feature, and the way to make 1.24 have the same behavior you were getting in 1.23 is to stop passing --node-ip.

[danwinship]

It's possible that we should declare the 1.24 change to be a (command-line) API break, and revert it, and then instead add --node-ips (with an "s") and say that you need to use that to get the new behavior, and --node-ip (with no "s") will continue to have the strange behavior it always used to have.

[manuelbuil]

@manuelbuil

However, since 1.24, if we pass --node-ip to kubelet, e.g. because we use a secondary NIC for cluster communication

So note that this didn't work in 1.23 (passing --node-ip would not change what NIC was used for communication), so what you're describing isn't a new bug in 1.24, it's just the lack of a new feature, and the way to make 1.24 have the same behavior you were getting in 1.23 is to stop passing --node-ip.

Perhaps we are not talking about the same feature but I must disagree: it worked. In 1.23, when deploying an env with multiple interfaces, I was able to decide what NIC to use for the internal cluster communication by specifying the --node-ip, so in my opinion it was working correctly.

Of course --node-ip does not support dual stack, so I had to decide whether I wanted internal cluster communication to use ipv4 or ipv6. However, the cloud provider was able to set both ipv4 and ipv6 node addresses correctly (from the previously picked NIC) in the node object. You had to be careful and the cloud provider and node-ip had to be in sync and point to the same NIC. I wonder if perhaps this logic is what was considered buggy and tried to be fixed by the PR that prioritized --node-ip. If this is the case (is it?), I'd agree that this is not a bug but the PR did not consider dual-stack and thus the API change reduced the "feature-set" accidentally, right? In any case, having both ipv4 and ipv6 node addresses is important, e.g. for network policies

[danwinship]

Perhaps we are not talking about the same feature but I must disagree: it worked. In 1.23, when deploying an env with multiple interfaces, I was able to decide what NIC to use for the internal cluster communication by specifying the --node-ip, so in my opinion it was working correctly.

In 1.23, with an external cloud provider that was built from the official external-cloud-provider sources, --node-ip should have exactly two effects:

• if you pass a --node-ip value that contains an IP that the external cloud provider does not know about, then the external cloud provider will log an error and exit.
• if your pods have dual-stack IPs, then kubelet will sort the pod IPs of non-hostNetwork pods so that the one that is the same IP family as the --node-ip comes first.

In particular, --node-ip will not affect the values that get written to node.Status.Addresses, and thus by extension will not effect the podIPs of hostNetwork pods.

If you mean that in 1.23 you were deciding between using an IPv4 IP on one NIC and an IPv6 IP on the other NIC, then you weren't really choosing between the NICs, you were just choosing whether to make pods be IPv4-primary or IPv6-primary, and you can still do that in 1.24 by passing --node-ip 0.0.0.0 / --node-ip ::.

[thockin]

What a roller-coaster!

I am sensitive to the behavioral difference between built-in (legacy) and external cloud-providers causing migration pain. But I think I agree that changing that behavior is an API break. Dan, that's what you are saying, right? Is that (breaking) change in 1.24 or only in (not yet released)1.25? That could make this decision easier or harder :)

I like the path of deprecating ambiguous flags and adding new, better defined ones. This whole area is sketchy beyond belief. We should try to express what we really mean to express and not rely on flags to mean other things (like a "node IP" flag -> pod IP sorting).

[danwinship]

I was saying we might want to call it an API break, yes. I wasn't ready to declare it that myself.

But thinking about it some more, we will definitely break some existing 1.24 users if we revert the change now, because there will be users who used to use a legacy cloud provider with --node-ip who were only able to migrate to an external cloud provider in 1.24 because of the fix to --node-ip. If we revert, they will have to switch back from external cloud provider to legacy cloud provider at the same time...

So, if we leave the fix in, some people will break when upgrading from 1.23 to 1.24. If we revert it, some people will break when upgrading from 1.24.4 to 1.24.5...

We could also try tweaking the behavior of --node-ip with external cloud providers again, making it sort node.status.addresses, but not filter it... This is still a change from 1.23 though, and might still break some people.

It's also possible that everyone currently breaking on 1.23 to 1.24 upgrade could be fixed by changing from --node-ip <EXPLICIT IPV6 ADDRESS> to --node-ip ::, since that has the same cloud-provider behavior as 1.23 (the new code wouldn't run, because kubelet doesn't pass the IP to the cloud provider if it's :: or 0.0.0.0), and it has the same kubelet behavior as 1.23 (pod IPs would be sorted IPv6-first because --node-ip is IPv6).

[manuelbuil]

If you mean that in 1.23 you were deciding between using an IPv4 IP on one NIC and an IPv6 IP on the other NIC, then you weren't really choosing between the NICs, you were just choosing whether to make pods be IPv4-primary or IPv6-primary, and you can still do that in 1.24 by passing --node-ip 0.0.0.0 / --node-ip ::.

Not exactly, my use case is two NICs in one node, both having ipv4 and ipv6 addresses. For my kubernetes traffic, I want to use the secondary NIC, not the one defined in the default route, and I would like both addresses of the secondary NIC to be under node.status.addresses

It's also possible that everyone currently breaking on 1.23 to 1.24 upgrade could be fixed by changing from --node-ip <EXPLICIT IPV6 ADDRESS> to --node-ip ::, since that has the same cloud-provider behavior as 1.23 (the new code wouldn't run, because kubelet doesn't pass the IP to the cloud provider if it's :: or 0.0.0.0), and it has the same kubelet behavior as 1.23 (pod IPs would be sorted IPv6-first because --node-ip is IPv6).

Thanks for the reply and the suggestion. This workaround works for me and I guess it is an acceptable solution for the time being :)

[danwinship]

OK, discussed in SIG Network. We are not going to revert the new behavior, but we should try to document the workaround better. (Can we belatedly add a 1.24 release note?)

We should also fix kubelet so that if you don't pass --node-ip, that it removes any existing node-ip annotation, rather than leaving the stale annotation there, and we should backport that fix to 1.24. #112184

[manuelbuil]

OK, discussed in SIG Network. We are not going to revert the new behavior, but we should try to document the workaround better. (Can we belatedly add a 1.24 release note?)

We should also fix kubelet so that if you don't pass --node-ip, that it removes any existing node-ip annotation, rather than leaving the stale annotation there, and we should backport that fix to 1.24. #112184

Was there any discussion around making node-ip dual-stack aware?

[danwinship]

We only talked about the immediate "are we going to call this a regression or not" issue.

We definitely eventually want node-ip to support dual-stack. (It already does for bare metal, but making it work for clouds too requires adding a new annotation or something.)

[SergeyKanzhelev]

/remove-sig node

looks like this discussion is network related, please feel free to re-add node here

[thockin]

ping to @danwinship

[lubronzhan]

Hi, our team has another use case. We are using kube-vip gratuitous arp to advertise control plane HA IP. So control plane VM could have two IPs, one is assigned by kube-vip, a floating IP that could be assigned randomly to 1 out of 3 control plane VM, and float around. Then we need to pass node-ip not to assign the floating ip to kubelet so that kubelet won't use that floating IP for pod with hostNetwork.
We used to pass the real machine IP through node-ip, but now for dual stack, out of tree cloud provider will only set single IP that's passed from node-ip.
In this case we can't come up with good workaround. @danwinship do you have any good idea?
Thank you

[danwinship]

There's no way to override/control both node IPs in a dual-stack external-cloud-provider cluster right now.

I'm working on a KEP for this:

### Goals

- Allow administrators of clusters using external cloud providers to
  override the cloud-selected node IPs in a more dual-stack-aware
  manner:

    - Allow administrators to override a single IP in a dual-stack
      cluster without causing the cluster to become single-stack.

    - Allow administrators to override both IPs in a dual-stack
      cluster.

    - Allow administrators to override the order of IPs in a
      dual-stack cluster (ie, requesting that nodes be IPv6-primary
      dual stack rather than IPv4-primary dual stack, or vice versa).

    - Allow administrators to configure nodes to be single-stack when
      the cloud provider suggests dual-stack, without needing to
      provide a specific IPv4 or IPv6 node IP.

- Define how kubelet will communicate these new intents to cloud
  providers.

- Update the code in `k8s.io/cloud-provider/node/helpers` to implement
  the needed algorithms for the new behaviors.

[thockin]

Dan, Should we accet this as triaged? Maybe retitle first?

[danwinship]

Dan, Should we accet this as triaged? Maybe retitle first?

I think we should close this; the original reported problem was fixed by #112184. The remaining problem that was discussed here (which is more of a feature request than a bug report) is now addressed by kubernetes/enhancements#3705.

[Burberrysmart]

Great